首页 文章

Spark:不支持的类版本错误

提问于
浏览
3

我试图在所有节点都安装了java 1.7的集群上使用spark-submit运行java spark作业 .

作业失败,出现java.lang.UnsupportedClassVersionError:com / windlogics / dmf / wether / MyClass:不支持的major.minor版本51.0 .

此错误似乎是由使用较低版本的Java进行编译并使用更高版本运行引起的 . 但是,我已经验证代码是用1.7编译的 .

此外,当master设置为local时,作业可以正常工作 . 我该如何调试并修复此错误?

错误日志的一部分如下 .

15/01/21 15:14:57 WARN scheduler.TaskSetManager:阶段0.0中丢失的任务0.0(TID 0,sphddp07.zzz.local):java.lang.UnsupportedClassVersionError:com / zzz / dmf / wether / MyClass:Unsupported major.minor版本51.0 java.lang.ClassLoader.defineClass1(Native Method)java.lang.ClassLoader.defineClassCond(ClassLoader.java:631)java.lang.ClassLoader.defineClass(ClassLoader.java:615)java.security.SecureClassLoader . defineClass(SecureClassLoader.java:141)java.net.URLClassLoader.defineClass(URLClassLoader.java:283)java.net.URLClassLoader.access $ 000(URLClassLoader.java:58)java.net.URLClassLoader $ 1.run(URLClassLoader.java: 197)java.security.AccessController.doPrivileged(Native Method)java.net.URLClassLoader.findClass(URLClassLoader.java:190)java.lang.ClassLoader.loadClass(ClassLoader.java:306)java.lang.ClassLoader.loadClass(ClassLoader) .java:247)java.lang.Class.forName0(Native Method)java.lang.Class.forName(Class.java:247)org.apache.spark.serializer.JavaDeserializationStream $$ anon $ 1.resolve Class(JavaSerializer.scala:59)java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1574)java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1495)java.io.ObjectInputStream.readClass(ObjectInputStream.java:1461) java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1311)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java.io.ObjectInputStream.readOrdinaryObject( ObjectInputStream.java:1752)java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java . io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)scala.collection.immutable . $ colon $ colon .readObject(List.scala:3 62)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang.reflect.Method.invoke (Method.java:597)java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)java .io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream .java:1752)java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java.io . 宾语InputStream.readOrdinaryObject(ObjectInputStream.java:1752)java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)scala.collection.immutable . $ colon $ colon.readObject (List.scala:362)sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)java.lang . reflect.Method.invoke(Method.java:597)java.io.ObjectStreamClass.invokeReadObject(ObjectStreamClass.java:969)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1848)java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream . java:1752)java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)java.io.ObjectInputStream.readObject0( ObjectInputStream.java:1328)java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1946)java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1870)java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1752)java . io.ObjectInputStream.readObject0(ObjectInputStream.java:1328)java.io.ObjectInputStream.readObject(ObjectInputStream.java:350)org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)org.apache.spark . serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:87)org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:57)org.apache.spark.scheduler.Task.run(Task.scala:54)org . apache.spark.executor.Executor $ TaskRunner.run(Executor.scala:177)j ava.util.concurrent.ThreadPoolExecutor $ Worker.runTask(ThreadPoolExecutor.java:886)java.util.concurrent.ThreadPoolExecutor $ Worker.run(ThreadPoolExecutor.java:908)

2 回答

  • 0

    尝试在主机和从机上的bashrc文件中设置适当的JAVA_HOME,我遇到了同样的问题,但这没有用 .

    然后当我在整个集群级别设置JAVA_HOME时,major.minor错误消失了 . 我使用cloudera,所以我必须在cloudera manager中设置JAVA_HOME来解决这个错误 .

  • 1

    我遇到了同样的错误消息 . 我发现当我输入'java -version'时它是1.7 . 我需要Java 8.以下是更新方法:

    sudo yum install java-1.8.0
    sudo alternatives --config java
    

相关问题