Spark java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地类不兼容-Java 学习之路

我试图在远程集群上运行我的Spark应用程序，我收到序列化错误 . Scala和Spark版本是一样的 . 我陷入了困境 .

集群上的spark-shell -version：

root@a913008dd071:/usr/local/spark-2.1.1# ./bin/spark-submit --version
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /___/ .__/\_,_/_/ /_/\_\   version 2.1.1
      /_/

Using Scala version 2.11.8, OpenJDK 64-Bit Server VM, 1.8.0_131
Branch
Compiled by user jenkins on 2017-04-25T23:51:10Z
Revision
Url
Type --help for more information.

build.sbt

import sbt.ExclusionRule

name := "hxfa"
version := "1.0"
scalaVersion := "2.11.8"

val elasticVersion = "5.4.1"

resolvers += "Spark Packages" at "https://dl.bintray.com/spark-packages/maven/"
resolvers += "Additional spark packages" at "https://dl.bintray.com/sbcd90/org.apache.spark"
resolvers += "Apache HBase" at "https://repository.apache.org/content/repositories/releases"
resolvers += "Thrift" at "http://people.apache.org/~rawson/repo/"
resolvers += "Spring Plugins" at "http://repo.spring.io/plugins-release/"

/* Dependencies */
libraryDependencies ++= Seq(
  // Framework and configuration
  "org.springframework.boot" % "spring-boot-starter-web" % "1.5.4.RELEASE",
  "org.hibernate" % "hibernate-validator" % "5.2.4.Final",

  /* Serializations */
  "com.fasterxml.jackson.core" % "jackson-core" % "2.8.7",
  "com.fasterxml.jackson.core" % "jackson-databind" % "2.8.7",
  "com.fasterxml.jackson.module" % "jackson-module-scala_2.11" % "2.8.7",
  "com.esotericsoftware" % "kryo" % "4.0.0",


  // Spark and utilities
  "org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,
  "graphframes" % "graphframes" % "0.5.0-spark2.1-s_2.11",


  // Spark connectors
  "org.elasticsearch" % "elasticsearch-spark-20_2.11" % elasticVersion,
  "org.mongodb.spark" % "mongo-spark-connector_2.11" % "2.0.0",


  //JDBC
  "mysql" % "mysql-connector-java" % "5.1.35",

  // HBase
  "org.apache.hbase" % "hbase" % "1.2.4",
  "org.apache.hbase" % "hbase-client" % "1.2.4",
  "org.apache.hbase" % "hbase-common" % "1.2.4",

  // OrientDB
  "com.orientechnologies" % "orientdb-graphdb" % "2.2.20"

).map(_.excludeAll(ExclusionRule("org.slf4j", "slf4j-log4j12"), ExclusionRule("log4j", "log4j")))

libraryDependencies ++= Seq(
  "org.apache.hbase" % "hbase-server" % "1.2.4"
).map(_.excludeAll(
  ExclusionRule("com.sun.jersey", "jersey-server"),
  ExclusionRule("tomcat"),
  ExclusionRule("log4j", "log4j")
))


/* Assembly     */

mainClass in assembly := Some("com.x.x.hello.app.HX")
assemblyOption in assembly := (assemblyOption in assembly).value.copy(includeScala = false, includeDependency = false)

assemblyMergeStrategy in assembly := {
  case PathList("META-INF", xs@_*) => MergeStrategy.discard
  case x => MergeStrategy.first
}

堆栈跟踪：

java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地类不兼容：stream classdesc serialVersionUID = -2992553500466442037，本地类serialVersionUID = -5670082246090726217 at java.io.ObjectStreamClass.initNonProxy（ObjectStreamClass.java:616)~ [na：1.8.0_45] at java.io.ObjectInputStream.readNonProxyDesc（ObjectInputStream .java：1843）〜[na：1.8.0_45] at java.io.ObjectInputStream.readClassDesc（ObjectInputStream.java:1713）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2000 ）〜[na：1.8.0_45] java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245)~ [na ：1.8.0_45]在java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45 java.io.ObjectInputStream.readObject（ObjectInputStream.java：422）〜的java.io.ObjectInputStream.readObject0（ObjectInputStream.java：1535）〜[na：1.8.0_45]〜 [na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy.readObject（List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4 .invoke（未知来源）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45] at java.lang.reflect.Method.invoke（Method.java： 498）〜[na：1.8.0_45] at java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass.java:1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136)~ [ NA：1.8.0_45]在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[NA：1.8.0_45]在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[NA：1.8 . 0_45]在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] java .io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io .ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0 （ObjectInputStream.java:1535）〜[NA：1.8.0_45]在java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[NA：1.8.0_45]在scala.collection.immutable.List $ SerializationProxy.readObject （List.scala：479）〜[阶库-2.11.8.jar：1.0.0-M1]在sun.reflect.GeneratedMethodAccessor4.invoke（未知来源）〜[NA：NA]在sun.reflect.DelegatingMethodAccessorImpl . 调用（DelegatingMethodAccessorImpl.java:43）〜[NA：1.8.0_45]在java.lang.reflect.Method.invoke（Method.java:498）〜[NA：1.8.0_45]在java.io.ObjectStreamClass.invokeReadObject（ java.io.ObjectInputSt中的ObjectStreamClass.java:1058)~ [na：1.8.0_45] ream.readSerialData（ObjectInputStream.java:2136)~ [na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ java.io.ObjectInputStream.readSerialData（ObjectInputStream.java：）中java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45]的ObjectInputStream.java:1535)~ [na：1.8.0_45]： 2169）〜[NA：1.8.0_45]在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[NA：1.8.0_45]在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[ na：1.8.0_45]在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8 . 0_45]在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[NA：1.8.0_45]在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[NA：1.8.0_45]在java的.io.ObjectInputStream.defaultReadFields（ ObjectInputStream.java:2245)~ [na：1.8.0_45] atjava.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io. java.io.ObjectInputStream.readSerialData（java.io.ObjectInputStream.readSerialData） java.io.ObjectInputStream.readObject0（ObjectInputStream.java：）中的java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45]中的ObjectInputStream.java:2169)~ [na：1.8.0_45]： 1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [ na：1.8.0_45]在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8 . 0_45]在java.io.ObjectInputStre am.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169)~ [na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java）中java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45]的ObjectInputStream.java:2027)~ [na：1.8.0_45]：在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[ na：1.8.0_45]在java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8 . 0_45] java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8.0_45] java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] java .io.ObjectInputStream.readObject0（OB jectInputStream.java:1535)~ [na：1.8.0_45] at java.io.ObjectInputStream.readObject（ObjectInputStream.java:422）〜[na：1.8.0_45] at scala.collection.immutable.List $ SerializationProxy.readObject（ List.scala：479）〜[scala-library-2.11.8.jar：1.0.0-M1] at sun.reflect.GeneratedMethodAccessor4.invoke（Unknown Source）〜[na：na] at sun.reflect.DelegatingMethodAccessorImpl.invoke （DelegatingMethodAccessorImpl.java:43）〜[na：1.8.0_45]在java.io.ObjectStreamClass.invokeReadObject（ObjectStreamClass）java.lang.reflect.Method.invoke（Method.java:498）〜[na：1.8.0_45] .java：1058）〜[na：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2136)~ [na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027 ）〜[na：1.8.0_45] java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] at java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245)~ [na ：1.8.0_45] at java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2 169）〜[na：1.8.0_45] at java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] at java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535)~ [ na：1.8.0_45]在java.io.ObjectInputStream.defaultReadFields（ObjectInputStream.java:2245）〜[na：1.8.0_45] java.io.ObjectInputStream.readSerialData（ObjectInputStream.java:2169）〜[na：1.8 . 0_45]在java.io.ObjectInputStream.readOrdinaryObject（ObjectInputStream.java:2027）〜[na：1.8.0_45] java.io.ObjectInputStream.readObject0（ObjectInputStream.java:1535）〜[na：1.8.0_45] java orio.apache.spark.serializer.JavaDeserializationStream.readObject（JavaSerializer.scala：75）〜[spark-core_2.11-2.1 . 0.jar：2.1.0]在org.apache的org.apache.spark.serializer.JavaSerializerInstance.deserialize（JavaSerializer.scala：114）〜[spark-core_2.11-2.1.0.jar：2.1.0] . spark.scheduler.ResultTask.runTask（ResultTask.scala：80）〜[spark-core_2.11-2.1.0.jar：2.1.0] at o org.apache.spark.executor.Executor上的rg.apache.spark.scheduler.Task.run（Task.scala：99）〜[spark-core_2.11-2.1.0.jar：2.1.0] $ TaskRunner.run （Executor.scala：322）〜[spark-core_2.11-2.1.0.jar：2.1.0] java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1142）〜[na：1.8.0_45] at java.util.concurrent.ThreadPoolExecutor $ Worker.run（ThreadPoolExecutor.java:617）〜[na：1.8.0_45] at java.lang.Thread.run（Thread.java:748)~ [na：1.8.0_45]

1 回答

2
spark-submit --version显示火花和scala版本的软件包而不是你的系统，而你的sbt正在使用系统的scala版本 . 所以

请更换
```
"org.apache.spark" %% "spark-core" % "2.1.0",
  "org.apache.spark" %% "spark-sql" % "2.1.0" ,
  "org.apache.spark" %% "spark-mllib" % "2.1.0" ,
```
至
```
"org.apache.spark" % "spark-core_2.11" % "2.1.1",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.1" ,
  "org.apache.spark" % "spark-mllib_2.11" % "2.1.1" ,
```
如果它没有帮助，请使用您的系统scala版本更新您的问题，如何提交应用程序以及您的远程计算机的scala和spark版本 .
回复于 2024-05-15T12:22:42+08:00

Spark java.io.InvalidClassException：org.apache.spark.unsafe.types.UTF8String;本地类不兼容

1 回答

相关问题