java.lang.NoClassDefFoundError：scala / collection / GenTraversableOnce $ class

我尝试实现Apache kafka和spark流集成这里是我的python代码：

from __future__ import print_function
import sys
from pyspark.streaming import StreamingContext
from pyspark import SparkContext,SparkConf
from pyspark.streaming.kafka import KafkaUtils

if __name__ == "__main__":
#conf = SparkConf().setAppName("Kafka-Spark").setMaster("spark://127.0.0.1:7077")
conf = SparkConf().setAppName("Kafka-Spark")
#sc = SparkContext(appName="KafkaSpark")
sc = SparkContext(conf=conf)
stream=StreamingContext(sc,1)
map1={'demo':1}
kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "test-consumer-group", map1)

# kafkaStream = KafkaUtils.createStream(stream, 'localhost:2181', "name", map1) #tried with localhost:2181 too
lines = kafkaStream.map(lambda x: x[1])
counts = lines.flatMap(lambda line: line.split(" ")) 
     .map(lambda word: (word, 1)) \
     .reduceByKey(lambda a, b: a+b)
counts.pprint()

stream.start()
stream.awaitTermination()

当我运行上面的程序然后它在终端上显示输出：

16/10/24 15:27:20错误执行程序：阶段0.0（TID 0）中任务0.0的异常java.lang.NoClassDefFoundError：kafka.utils.Pool中的scala / collection / GenTraversableOnce $ class . （Pool.scala：28 ）kafka.consumer.ZookeeperConsumerConnector . （ZookeeperConsumerConnector.scala：91）at kafka.consumer.ZookeeperConsumerConnector . （ZookeeperConsumerConnector.scala：143）位于org.apache的kafka.consumer.Consumer $ .create（ConsumerConnector.scala：94） . spark.streaming.kafka.KafkaReceiver.onStart（KafkaInputDStream.scala：100）在org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver（ReceiverSupervisor.scala：149）在org.apache.spark.streaming.receiver.ReceiverSupervisor . 启动（ReceiverSupervisor.scala：131）在org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $$ anonfun $ 9.apply（ReceiverTracker.scala：597）在org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $ $ anonfun $ 9.apply（ReceiverTracker.scala：587）at org.apache.spark.SparkContext $$ anonfun $ 33.apply（S parkContext.scala：1993）atg.apache.spark.SparkContext $$ anonfun $ 33.apply（SparkContext.scala：1993）atg.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：70）at org.apache .spark.scheduler.Task.run（Task.scala：86）在org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：274）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java ：1142）在java.util.concurrent.ThreadPoolExecutor中$ Worker.run（ThreadPoolExecutor.java:617）在java.lang.Thread.run（Thread.java:745）产生的原因：抛出java.lang.ClassNotFoundException：scala.collection . 在java.net.URLClassLoader.findClass（URLClassLoader.java:381）在java.lang.ClassLoader.loadClass（ClassLoader.java:424）在sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java:331）GenTraversableOnce $类at java.lang.ClassLoader.loadClass（ClassLoader.java:357）... 17更多16/10/24 15:27:20错误SparkUncaughtExceptionHandler：线程中未捕获的异常Thread [Executor task launch worker-0,5，main] java.lang.NoClassDefFoundEr ror：scaf / collection / GenTraversableOnce $ class at kafka.consumer.ZookeeperConsumerConnector . （ZookeeperConsumerConnector.scala：91）位于kafka.consumer.ZookeeperConsumerConnector的kafka.utils.Pool . （Pool.scala：28） . （ZookeeperConsumerConnector.scala：143 ）在kafka.consumer.Consumer $ .create（ConsumerConnector.scala：94）在org.apache.spark.streaming.kafka.KafkaReceiver.onStart（KafkaInputDStream.scala：100）在org.apache.spark.streaming.receiver.ReceiverSupervisor .startReceiver（ReceiverSupervisor.scala：149）在org.apache.spark.streaming.receiver.ReceiverSupervisor.start（ReceiverSupervisor.scala：131）在org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $$ anonfun $ 9.apply （ReceiverTracker.scala：597）在org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $$ anonfun $ 9.apply（ReceiverTracker.scala：587）在org.apache.spark.SparkContext $$ anonfun $ 33.apply（SparkContext .scala：1993）org.apache.spark.SparkContext $$ anonfun $ 33.apply（SparkContext.scala：1993）at o rg.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：70）在org.apache.spark.scheduler.Task.run（Task.scala：86）在org.apache.spark.executor.Executor $ TaskRunner . 运行（Executor.scala：274）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:617）在java.lang.Thread中.RUN（Thread.java:745）产生的原因：抛出java.lang.ClassNotFoundException：在java.lang.ClassLoader.loadClass在java.net.URLClassLoader.findClass（URLClassLoader.java:381）scala.collection.GenTraversableOnce $类（类加载器.java：424）at sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java:331）at java.lang.ClassLoader.loadClass（ClassLoader.java:357）... 17 more 16/10/24 15:27 ：20 INFO StreamingContext：从关闭钩子16/10/24 15:27:20调用stop（stopGracefully = false）WARN TaskSetManager：阶段0.0中丢失的任务0.0（TID 0，localhost）：java.lang.NoClassDefFoundError：scala / collection / GenTraversableOnce $ class at k afka.utils.Pool（Pool.scala：28） . 在kafka.consumer.ZookeeperConsumerConnector（ZookeeperConsumerConnector.scala：91） . 在kafka.consumer.ZookeeperConsumerConnector（ZookeeperConsumerConnector.scala：143） . 在kafka.consumer.Consumer $ .create（ConsumerConnector.scala：94）在org.apache.spark.streaming.kafka.KafkaReceiver.onStart（KafkaInputDStream.scala：100）在org.apache.spark.streaming.receiver.ReceiverSupervisor.startReceiver （ReceiverSupervisor.scala：149）atg.apache.spark.streaming.receiver.ReceiverSupervisor.start（ReceiverSupervisor.scala：131）at org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $$ anonfun $ 9.apply（ReceiverTracker） .scala：597）org.apache.spark.streaming.scheduler.ReceiverTracker $ ReceiverTrackerEndpoint $$ anonfun $ 9.apply（ReceiverTracker.scala：587）org.apache.spark.SparkContext $$ anonfun $ 33.apply（SparkContext.scala ：1993）org.apache.spark.SparkContext $$ anonfun $ 33.apply（SparkContext.scala：1993）atg.apache.spark.scheduler.ResultTask.runTask（ResultTask.scala：70）at org.apache.spark . scheduler.Task.run（Task.scala：86）at org.apache.spark.executor.Executor $ TaskRunner.run（Executor.scala：274）at java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadP） oolExecutor.java:1142）java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）at java.lang.Thread.run（Thread.java:745）引起：java.lang.ClassNotFoundException：scala在java.net.URLClassLoader.findClass（URLClassLoader.java:381）在java.lang.ClassLoader.loadClass（ClassLoader.java:424）在sun.misc.Launcher $ AppClassLoader.loadClass（Launcher.java .collection.GenTraversableOnce $类：331）在java.lang.ClassLoader.loadClass（ClassLoader.java:357）... 17更多

16/10/24 15:27:20错误TaskSetManager：阶段0.0中的任务0失败1次;