首页 文章

如何在Zeppelin上运行Spark来分析xml文件

提问于
浏览
1

我可以通过 bin/spark-shell --packages com.databricks:spark-xml_2.11:0.3.0 运行Spark shell来分析xml文件,例如:

import org.apache.spark.sql.SQLContext

val sqlContext = new SQLContext(sc)
val df = sqlContext.read
    .format("com.databricks.spark.xml")
    .option("rowTag", "book")
    .load("books.xml")

但我怎么能运行Zeppelin来做到这一点 . Zeppelin在开始导入 com.databricks.spark.xml 时是否需要一些参数?现在我得到:

java.lang.RuntimeException:无法加载数据源的类:com.databricks.spark.xml at scala.sys.package $ .error(package.scala:27)at org.apache.spark.sql.sources.ResolvedDataSource $ .lookupDataSource(ddl.scala:220)atg.apache.spark.sql.sources.ResolvedDataSource $ .apply(ddl.scala:233)at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) )atg.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:26)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:31)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:33)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:35)at $ iwC $$ iwC $$ iwC $$ iwC . (:37)at $ iwC $$ iwC $$ iwC . (:39)at $ iwC $$ iwC . (:41)at $ iwC . (:41)at(:45)at . (:49)at . ()at . (:7)at . ()at $ print()at sun .reflect.NativeMethodAccessorImpl.invoke0(本机方法)在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect中 . 方法.invoke(Method.java:497)org.apache.spark.repl.SparkIMain $ ReadEvalPrint.call(SparkIMain.scala:1065)at org.apache.spark.repl.SparkIMain $ Request.loadAndRun(SparkIMain.scala: 1338)org.apache.spark.repl.SparkIMain.loadAndRunReq $ 1(SparkIMain.scala:840)org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)org.apache.spark.repl . SparkIMain.interpret(SparkIMain.scala:819)在org.apache.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:709)在org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:674)在组织.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:667)org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter) .java:93)atg.apache.zeppelin.interpreter.remote.RemoteInterpreterServer $ InterpretJob.jobRun(RemoteInterpreterServer.java:300)at org.apache.zeppelin.scheduler.Jo b.run(Job.java:169)在org.apache.zeppelin.scheduler.FIFOScheduler $ 1.run(FIFOScheduler.java:134)在java.util.concurrent.Executors $ RunnableAdapter.call(Executors.java:511)在java.util.concurrent.FutureTask.run(FutureTask.java:266)在java.util.concurrent.ScheduledThreadPoolExecutor中$ $ ScheduledFutureTask.access 201(ScheduledThreadPoolExecutor.java:180)在java.util.concurrent.ScheduledThreadPoolExecutor中$ ScheduledFutureTask.run(的ScheduledThreadPoolExecutor的.java:293)在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run(ThreadPoolExecutor.java:617)在java.lang.Thread.run( Thread.java:745)

1 回答

  • 5

    在Zeppelin中,您需要在创建SparkContext之前调用这些依赖项 .

    在单独的单元格中,添加并运行以下内容

    %dep
    z.reset()
    z.addRepo("Spark Packages Repo").url("http://dl.bintray.com/spark-packages/maven")
    z.load("com.databricks:spark-xml_2.11:0.3.0")
    

    如果这给你一个类型的错误:“你必须在启动SparkContext之前添加依赖项”,只需重新启动解释器或Zeppelin .

相关问题