首页 文章

当Receiver失败并且WAL存储在s3中时,Spark流无法从写入提前日志记录中读取数据

提问于
浏览
1

这是错误日志 -

org.apache.spark.SparkException:无法在org.apache中读取写入提前日志记录FileBasedWriteAheadLogSegment(s3n:// ***** / checkpoint / receivedData / 20 / log-1439298698600-1439298758600,13678,5069)中的数据.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org $ apache $ spark $ streaming $ rdd $ WriteAheadLogBackedBlockRDD $$ getBlockFromWriteAheadLog $ 1(WriteAheadLogBackedBlockRDD.scala:144)at org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD $$ anonfun $ compute $ 1 . 应用(WriteAheadLogBackedBlockRDD.scala:168)org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD $$ anonfun $ compute $ 1.apply(WriteAheadLogBackedBlockRDD.scala:168)at the scla.Option.getOrElse(Option.scala:120)at org位于org.apache.spark.rdd.RDD.iterator的org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)的.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.compute(WriteAheadLogBackedBlockRDD.scala:168) (RDD.scala:244)org.apache.spark.rdd.UnionRDD.compute(UnionRDD.scala:87)at org.apache.spark.rdd.RDD.com位于org.apache的org.apache.spark.rdd.RDD.RD上的org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala:35)中的puteOrReadCheckpoint(RDD.scala:277)位于org.apache.spark.rdd.MapPartitionsRDD.compute(MapPartitionsRDD.scala)的org.apache.spark.rdd.RDD.iterator(RDD.scala:244)的.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277) :35)org.apache.spark.rdd.RDD.computeOrReadCheckpoint(RDD.scala:277)org.apache.spark.rdd.RDD.iterator(RDD.scala:244)atg.apache.spark.scheduler . ShuffleMapTask.runTask(ShuffleMapTask.scala:70)atg.apache.spark.scheduler.ShuffleMapTask.runTask(ShuffleMapTask.scala:41)atg.apache.spark.scheduler.Task.run(Task.scala:70)at org .apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:213)at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)at java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor) .java:615)at java.lang.Thread.run(Thread.java:745)引起:java.lang.NullPointer位于org.apache.spark.streaming.rdd.WriteAheadLogBackedBlockRDD.org的org.apache.spark.streaming.util.FileBasedWriteAheadLog.read(FileBasedWriteAheadLog.scala:106)中的异常$ apache $ spark $ streaming $ rdd $ WriteAheadLogBackedBlockRDD $$ getBlockFromWriteAheadLog $ 1 (WriteAheadLogBackedBlockRDD.scala:141)......还有22个

注意:如果HDFS用作存储,则从WAL读取工作正常 .

任何帮助非常感谢 .

1 回答

相关问题