首页 文章

java.lang.NoClassDefFoundError:更好/ files /文件在spark-sbumit中用于scala代码

提问于
浏览
0

当我将程序导出到jar文件并执行它时,我得到一个java.lang.NoClassDefFoundError:better / files / File错误 .

我正在使用的代码如下 .

在此先感谢您的任何帮助

SBT

name := "testFunctions"

version := "1.0"

scalaVersion := "2.11.8"

libraryDependencies += "com.github.pathikrit" %% "better-files" % "2.17.1"
libraryDependencies ++= Seq(
  "org.apache.spark" % "spark-core_2.11" % "2.1.0",
  "org.apache.spark" % "spark-sql_2.11" % "2.1.0",
  "com.github.pathikrit" %% "better-files" % "2.17.1"
)

initialize := {
  val _ = initialize.value
  if (sys.props("java.specification.version") != "1.8")
    sys.error("Java 8 is required for this project.")
}

Scala Code

/**
  * Created by cloudera on 7/23/17.
  */

import better.files.File._
import org.apache.spark.sql.SparkSession


object funcJM {


  val forDelete = (root/"/home/cloudera/Documents/fabo")
    .createIfNotExists()

  if (forDelete.exists)
    forDelete.delete()


  def main(args:Array[String]) : Unit = {

    val spark = SparkSession.builder
      .master("local")
      .appName("Get ForEx Data")
      .getOrCreate()



  }

}

Command line executing jar file

spark-submit --class funcJM --master local[*] /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar --driver-class-path /home/cloudera/testFunctions/target/scala-2.11/testfunctions_2.11-1.0.jar

Error

线程“main”java.lang.NoClassDefFoundError中的异常:java.lang上java.lang.Class.privateGetDeclaredMethods(Class.java:2701)中的java.lang.Class.getDeclaredMethods0(Native Method)中的better / files / File . Class.privateGetMethodRecursive(Class.java:3048)at java.lang.Class.getMethod0(Class.java:3018)at java.lang.Class.getMethod(Class.java:1784)org.apache.spark.deploy.SparkSubmit $ .org $ apache $ spark $ deploy $ org.apache.spark.deploy.SparkSubmit $ .doRunMain $ 1(SparkSubmit.scala:187)org.apache.spark.deploy上的$ SparkSubmit $$ runMain(SparkSubmit.scala:722) .SparkSubmit $ .submit(SparkSubmit.scala:212)org.apache.spark.deploy.SparkSubmit $ .main(SparkSubmit.scala:126)at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)引起by:java.lang.ClassNotFoundException:more.files.File at java.net.URLClassLoader.findClass(URLClassLoader.java:381)at java.lang.ClassLoader.loadClass(ClassLoader.java:424)at java.lang.ClassLoader . loadClass(ClassLoader.java:357)......还有10个

The Tree structure where the code live looks like below - 所以名为func的类存在于scala目录中

\main
    \java
    \resources
    \scala
        -funcJM(class)
    \scala-2.11

1 回答

  • 2

    你可以做三件事:

    • 使用像sbt assembly这样的东西编译一个胖 jar . 包含better.files.File的jar将打包在testfunctions_2.11-1.0.jar中 .

    • 将包含更好包的jar文件复制到spark jars目录中:../ spark2.1.0-bin-hadoop2.7/jars

    • 使用 --driver-class-path better-files-akka_2.10.jar 指定包含更好包的jar文件作为spark-submit调用中的参数

相关问题