编译后执行Spark scala程序-Java 学习之路

-1

我在命令行上编译了Spark scala程序 . 但现在我想执行它 . 我不想使用Maven或sbt . 程序 . 我用这个命令来执行

scala -cp ".:sparkDIrector/jars/*" wordcount

但我收到此错误 java.lang.NoSuchMethodError: scala.Predef$.refArrayOps([Ljava/lang/Object;)Lscala/collection/mutable/ArrayOps;

import org.apache.spark._
import org.apache.spark.SparkConf

/** Create a RDD of lines from a text file, and keep count of
 *  how often each word appears.
 */
object wordcount1 {

  def main(args: Array[String]) {
      // Set up a SparkContext named WordCount that runs locally using
      // all available cores.

      println("before conf")
      val conf = new SparkConf().setAppName("WordCount")
      conf.setMaster("local[*]")
      val sc = new SparkContext(conf)
      println("after the textfile")

      // Create a RDD of lines of text in our book
      val input = sc.textFile("book.txt")

      println("after the textfile")
      // Use flatMap to convert this into an rdd of each word in each line
      val words = input.flatMap(line => line.split(' '))
      // Convert these words to lowercase
      val lowerCaseWords = words.map(word => word.toLowerCase())
      // Count up the occurence of each unique word

      println("before text file")
      val wordCounts = lowerCaseWords.countByValue()

      // Print the first 20 results
      val sample = wordCounts.take(20)

      for ((word, count) <- sample) {
        println(word + " " + count)
      }

      sc.stop()
    }
}

它显示错误在位置

val conf = new SparkConf（） . setAppName（“WordCount”） .

有帮助吗？

1 回答

从Spark 2.0开始，入口点是SparkSession：

import org.apache.spark.sql.SparkSession
val spark = SparkSession
  .builder
  .appName("App Name")
  .getOrCreate()

然后，您可以访问SparkContext并使用以下内容读取文件：

spark.sparkContext().textFile(yourFileOrURL)

记得最后停止你的会话：

spark.stop()

我建议你看看这些例子：https://github.com/apache/spark/blob/master/examples/src/main/scala/org/apache/spark/examples

然后，要启动您的应用程序，您必须使用 spark-submit ：

./bin/spark-submit \
  --class <main-class> \
  --master <master-url> \
  --deploy-mode <deploy-mode> \
  --conf <key>=<value> \
  ... # other options
  <application-jar> \
  [application-arguments]

在你的情况下，它将是这样的：

./bin/spark-submit \
  --class wordcount1 \
  --master local \
  /path/to/your.jar

回复于 2024-04-29T10:42:44+08:00

编译后执行Spark scala程序

1 回答

相关问题