首页 文章

无法运行ALS.train,错误:java.lang.IllegalArgumentException

提问于
浏览
2

我正在尝试运行ALS of PySpark . 我复制并粘贴了链接中提供的示例代码 . 但是,错误 java.lang.IllegalArgumentException 出现在以下行:

model = ALS.train(ratings, rank, numIterations)

请问在这里我需要调查哪些可能的问题?

我的Spark版本是2.2.1,我的Java版本是9.0.4 . 但是,我不确定Spark是否正在使用正确版本的Java,尽管我设置了环境路径并且命令“java -version”确实返回“9.0.4” .

错误:

enter image description here

Py4JJavaError:调用z:org.apache.spark.mllib.api.python.SerDe.pythonToJava时发生错误 . :位于org.apache.xbean.asm5.ClassReader的org.apache.xbean.asm5.ClassReader . (未知来源)的java.lang.IllegalArgumentException . (未知来源)org.apache.xbean.asm5.ClassReader . (未知来源) )org.apache.spark.util.ClosureCleaner $ .getClassReader(ClosureCleaner.scala:46)at org.apache.spark.util.FieldAccessFinder $$ anon $ 3 $$ anonfun $ visitMethodInsn $ 2.apply(ClosureCleaner.scala:443) at org.apache.spark.util.FieldAccessFinder $$ anon $ 3 $$ anonfun $ visitMethodInsn $ 2.apply(ClosureCleaner.scala:426)at scala.collection.TraversableLike $ WithFilter $$ anonfun $ foreach $ 1.apply(TraversableLike.scala: 733)at scala.collection.mutable.HashMap $$ anon $ 1 $$ anonfun $ foreach $ 2.apply(HashMap.scala:103)at scala.collection.mutable.HashMap $$ anon $ 1 $$ anonfun $ foreach $ 2.apply( HashMap.scala:103)在scala.collection.mutable.mutable.mutable.mash.HashMap的scala.collection.mlection.mutable.HashMap.foreachEntry(HashMap.scala:40)scala.collection.mutable.HashTable $ class.foreachEntry(HashTable.scala:230)在scala上的$ anon $ 1.foreach(HashMap.scala:103) . collection.TraversableLike $ WithFilter.foreach(TraversableLike.scala:732)atg.apache.spark.util.FieldAccessFinder $$ anon $ 3.visitMethodInsn(ClosureCleaner.scala:426)at org.apache.xbean.asm5.ClassReader.a(位于org.apache.xbean.asm5.ClassReader.accept(未知来源)的org.apache.xbean.asm5.ClassReader.b(未知来源)org.apache.xbean.asm5.ClassReader.accept(未知来源) )org.apache.spark.util.ClosureCleaner $$ anonfun $ org $ apache $ spark $ util $ ClosureCleaner $$ clean $ 14.apply(ClosureCleaner.scala:257)at org.apache.spark.util.ClosureCleaner $$ anonfun $ org $ apache $ spark $ util $ ClosureCleaner $$ clean $ 14.apply(ClosureCleaner.scala:256)位于org.apache.spark.util.ClosureCleaner的scala.collection.immutable.List.foreach(List.scala:381) $ .org $ apache $ spark $ util $ cleosure(ClosureCleaner.scala:256)org.apache.spark.util.ClosureCleaner $ .clean(ClosureCleaner.scala:156)at org.apache.spark.SparkContext . 清理(SparkContext.scala:2294)org.apache.spark.rdd.RDD $$ anonfun $ mapP在org.apache.spark.rdd.RDD上的orgs.apache.spark.rdd.RDD $ anonfun $ mapPartitions $ 1.apply(RDD.scala:793)org.apache.spark.rdd.RDDOperationScope $ .withScope中的$ 1.apply(RDD.scala:794) RDDOperationScope.scala:151)位于org.apache的org.apache.spark.rdd.RDDOperationScope $ .withScope(RDDOperationScope.scala:112)atg.apache.spark.rdd.RDD.withScope(RDD.scala:362) . 位于org.apache.spark.mllib.api.python的org.apache.spark.mllib.api.python.SerDeBase.pythonToJava(PythonMLLibAPI.scala:1349)中的spark.rdd.RDD.mapPartitions(RDD.scala:793) . 来自java.base / java的java.base / jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)的jdk.internal.reflect.GeneratedMethodAccessor76.invoke(未知来源)的SerDe.pythonToJava(PythonMLLibAPI.scala) . 位于py4j.reflection的内容py4j.reflection.MethodInvoke(MethodInvoker.java:244)的py4j.reflection.invoke上的lang.reflect.Method.invoke(Method.java:564),反面是#4j.Gateway.invoke (Gateway.java:280)py4j.commands.AbstractCommand.inv okeMethod(AbstractCommand.java:132)py4j.commands.CallCommand.execute(CallCommand.java:79)at py4j.GatewayConnection.run(GatewayConnection.java:214)at java.base / java.lang.Thread.run(Thread的.java:844)

1 回答

  • 2

    我已经弄明白了这个问题 . Spark 2.2.1不适用于Java 9.0.4 . 也许我误解了教程中的“Java 8”部分 .

    如果有人遇到与我相同的错误,那就去Java 1.8.0吧!

相关问题