我可以通过 bin/spark-shell --packages com.databricks:spark-xml_2.11:0.3.0
运行Spark shell来分析xml文件,例如:
import org.apache.spark.sql.SQLContext
val sqlContext = new SQLContext(sc)
val df = sqlContext.read
.format("com.databricks.spark.xml")
.option("rowTag", "book")
.load("books.xml")
但我怎么能运行Zeppelin来做到这一点 . Zeppelin在开始导入 com.databricks.spark.xml
时是否需要一些参数?现在我得到:
java.lang.RuntimeException:无法加载数据源的类:com.databricks.spark.xml at scala.sys.package $ .error(package.scala:27)at org.apache.spark.sql.sources.ResolvedDataSource $ .lookupDataSource(ddl.scala:220)atg.apache.spark.sql.sources.ResolvedDataSource $ .apply(ddl.scala:233)at org.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:114) )atg.apache.spark.sql.DataFrameReader.load(DataFrameReader.scala:104)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:26)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:31)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:33)at $ iwC $$ iwC $$ iwC $$ iwC $$ iwC . (:35)at $ iwC $$ iwC $$ iwC $$ iwC . (:37)at $ iwC $$ iwC $$ iwC . (:39)at $ iwC $$ iwC . (:41)at $ iwC . (:41)at(:45)at . (:49)at . ()at . (:7)at . ()at $ print()at sun .reflect.NativeMethodAccessorImpl.invoke0(本机方法)在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect中 . 方法.invoke(Method.java:497)org.apache.spark.repl.SparkIMain $ ReadEvalPrint.call(SparkIMain.scala:1065)at org.apache.spark.repl.SparkIMain $ Request.loadAndRun(SparkIMain.scala: 1338)org.apache.spark.repl.SparkIMain.loadAndRunReq $ 1(SparkIMain.scala:840)org.apache.spark.repl.SparkIMain.interpret(SparkIMain.scala:871)org.apache.spark.repl . SparkIMain.interpret(SparkIMain.scala:819)在org.apache.zeppelin.spark.SparkInterpreter.interpretInput(SparkInterpreter.java:709)在org.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:674)在组织.apache.zeppelin.spark.SparkInterpreter.interpret(SparkInterpreter.java:667)org.apache.zeppelin.interpreter.ClassloaderInterpreter.interpret(ClassloaderInterpreter.java:57)at org.apache.zeppelin.interpreter.LazyOpenInterpreter.interpret(LazyOpenInterpreter) .java:93)atg.apache.zeppelin.interpreter.remote.RemoteInterpreterServer $ InterpretJob.jobRun(RemoteInterpreterServer.java:300)at org.apache.zeppelin.scheduler.Jo b.run(Job.java:169)在org.apache.zeppelin.scheduler.FIFOScheduler $ 1.run(FIFOScheduler.java:134)在java.util.concurrent.Executors $ RunnableAdapter.call(Executors.java:511)在java.util.concurrent.FutureTask.run(FutureTask.java:266)在java.util.concurrent.ScheduledThreadPoolExecutor中$ $ ScheduledFutureTask.access 201(ScheduledThreadPoolExecutor.java:180)在java.util.concurrent.ScheduledThreadPoolExecutor中$ ScheduledFutureTask.run(的ScheduledThreadPoolExecutor的.java:293)在java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run(ThreadPoolExecutor.java:617)在java.lang.Thread.run( Thread.java:745)
1 回答
在Zeppelin中,您需要在创建SparkContext之前调用这些依赖项 .
在单独的单元格中,添加并运行以下内容
如果这给你一个类型的错误:“你必须在启动SparkContext之前添加依赖项”,只需重新启动解释器或Zeppelin .