首页 文章

Spark WordCount命令行编译中的错误

提问于
浏览
0

我正在尝试使用命令行编译和运行Scala的WordCOunt程序,而不需要任何Maven和sbt支持 . 我用来编译scala程序的命令是

scalac -classpath /spark-2.3.0-bin-hadoop2.7/jars/ Wordcount.scala

import org.apache.spark._
import org.apache.spark.SparkConf

/** Create a RDD of lines from a text file, and keep count of
 *  how often each word appears.
 */
object wordcount {

  def main(args: Array[String]) {
      // Set up a SparkContext named WordCount that runs locally using
      // all available cores.
      val conf = new SparkConf().setAppName("WordCount")
      conf.setMaster("local[*]")
      val sc = new SparkContext(conf)

MY RESEARCH: 我已经参考了源代码,发现import语句在他们所需的jar中 .
For example SparkConf存在于程序中提到的org.apache.spark包中 .

https://github.com/apache/spark/blob/v2.3.1/core/src/main/scala/org/apache/spark/SparkConf.scala

ERRORS I AM FACING :

Wordcount.scala:3:错误:**对象apache不是包org import的成员org.apache.spark._ ^ ** Wordcount.scala:4:错误:**对象apache不是包org的成员import org.apache.spark.SparkConf ** ^ Wordcount.scala:14:错误:找不到:类型SparkConf val conf = new SparkConf() . setAppName(“WordCount”) ^ Wordcount.scala:16:error :找不到:类型SparkContext val sc = new SparkContext(conf) ^

发现了四个错误

1 回答

  • 2

    试试这个:

    scalac -classpath "/spark-2.3.0-bin-hadoop2.7/jars/*" Wordcount.scala
    

    您的问题中提到的 scalac 命令存在问题 . 如果要从某个目录中选择所有jar并将其放在classpath中,则需要使用 *** wildcard** 字符并将路径包装在双引号内 .

    详情请参阅:Including all the jars in a directory within the Java classpath

相关问题