Apache Spark多个上下文错误

3 回答

• 0

  我发现了这个问题 . 实际上Spark已经从其他用户启动了，所以当我和我的用户一起启动它时会出现多个上下文错误 . 从其他用户帐户停止后，其工作正常 .

  回复于 2024-04-29T08:38:45+08:00
• 0

  我有相同的情况，并将SparkContext转移到新对象帮助 .

  def spark = Action {
      val logFile = "README.md"
      val logData = SparkConnector.sc.textFile(logFile, 2).cache()
      val numSparks = logData.filter(line => line.contains("POST")).count()
      Ok(views.html.spark("Lines with Spark: " + numSparks))
    }
  

  SparkConnector.scala

  object SparkConnector {
    val sc = new SparkContext("local", "Application", "/opt/spark-1.5.2", List("target/scala-2.11/app_2.11-1.0-SNAPSHOT.jar"))
  }
  

  回复于 2024-04-29T08:38:45+08:00
• 0

  你可能会做这样的事情

  val conf = new SparkConf().setAppName(appName).setMaster(master)
  val sc1 = new SparkContext(conf)
  ...
  val sc2 = new SparkContext(conf) // <<<< HERE
  

  您应该只使用一个上下文，例如

  val conf = new SparkConf().setAppName(appName).setMaster(master)
  val sc = new SparkContext(conf)
  ...
  sc.DoSomething()
  sc.DoSomethingElse() // see, it is the same context
  

  回复于 2024-04-29T08:38:45+08:00