首页 文章

在Windows上使用Scala IDE和Apache Spark

提问于
浏览
0

我想开始研究在Windows 7上使用Spark with Scala的项目 . 我下载了针对hadoop 2.4(download page)的Apache Spark预构建,我可以从命令提示符(cmd)运行它 . 我可以在自包含应用程序部分之前快速启动spark页面上运行所有代码 .

然后我从其下载页面下载了Scala IDE 4.0.0(很抱歉,它不可能发布2个以上的链接) .

现在我创建了一个新的scala项目,并将spark程序集jar文件导入到项目中 . 当我想在快速启动spark页面的自包含应用程序部分中运行示例但是我遇到以下错误:

15/03/26 11:59:55 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 11:59:58 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 11:59:58 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:15 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:17 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:17 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:35 INFO AppClient$ClientActor: Connecting to master akka.tcp://sparkMaster@myhost:7077/user/Master...

15/03/26 12:00:37 WARN AppClient$ClientActor: Could not connect to akka.tcp://sparkMaster@myhost:7077: akka.remote.InvalidAssociation: Invalid address: akka.tcp://sparkMaster@myhost:7077

15/03/26 12:00:37 WARN Remoting: Tried to associate with unreachable remote address [akka.tcp://sparkMaster@myhost:7077]. Address is now gated for 5000 ms, all messages to this address will be delivered to dead letters. Reason: myhost

15/03/26 12:00:55 ERROR SparkDeploySchedulerBackend: Application has been killed. Reason: All masters are unresponsive! Giving up.

15/03/26 12:00:55 ERROR TaskSchedulerImpl: Exiting due to error from cluster scheduler: All masters are unresponsive! Giving up.

15/03/26 12:00:55 WARN SparkDeploySchedulerBackend: Application ID is not initialized yet.

我添加到示例中的唯一代码行是.setMaster(“spark:// myhost:7077”),用于SparkConf定义 . 我想我需要配置Scala IDE以在我的计算机上使用预构建火花,但实际上我不知道如何通过谷歌搜索找不到任何东西 .

你能帮助我让Scala IDE与Windows 7上的火花一起工作吗?

提前致谢

1 回答

  • 1

    我找到了答案:

    我应该在我的代码中更正主定义如下:

    更换:

    .setMaster("spark://myhost:7077")
    

    有:

    .setMaster("local[*]")
    

    希望它也能帮到你 .

相关问题