我试图冻结一个火花项目的依赖，以便能够脱机工作（sbt无法再下载依赖项） . 这是我遵循的过程：
-
创建sbt项目并使用Internet连接进行编译
-
暂停互联网连接
-
验证项目是否继续编译
-
复制SBT项目并删除TARGET文件夹
-
告诉Build.sbt文件解析/.ivy2/cache文件夹中的依赖关系
这是build.sbt：
name := "Test"
version := "1.0"
scalaVersion := "2.10.4"
libraryDependencies += "org.apache.spark" %% "spark-core" % "1.3.0"
resolvers += Resolver.file("Frozen IVY2 Cache Dependences", file("/home/luis/.ivy2/cache")) (Resolver.ivyStylePatterns) ivys "/home/luis/.ivy2/cache/[organisation]/[module]/ivy-[revision].xml" artifacts "/home/luis/.ivy2/cache/[organisation]/[module]/[type]s/[module]-[revision].[type]"
事实上，到达此Build.sbt的过程与此处描述的过程完全相同（并且没有回答）：
Troubles with sbt compiling offline using org.apache.hadoop/* dependencies
我将适当的常 Spring 藤样式模式包含在指向正确的常 Spring 藤[修订] .xml文件中 .
当我编译时，sbt正在为每个依赖找到.ivy2 /缓存“冻结”存储库的正确路径，但是我收到与解析文件“ivy- [revision] .xml.original”相关的警告和错误这四个依赖：
[warn] Note: Unresolved dependencies path:
[warn] org.apache.hadoop:hadoop-mapreduce-client-app:2.2.0
[warn] +- org.apache.hadoop:hadoop-client:2.2.0
[warn] +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn] +- Test:Test_2.10:1.0
[warn] org.apache.hadoop:hadoop-yarn-api:2.2.0
[warn] +- org.apache.hadoop:hadoop-client:2.2.0
[warn] +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn] +- Test:Test_2.10:1.0
[warn] org.apache.hadoop:hadoop-mapreduce-client-core:2.2.0
[warn] +- org.apache.hadoop:hadoop-client:2.2.0
[warn] +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn] +- Test:Test_2.10:1.0
[warn] org.apache.hadoop:hadoop-mapreduce-client-jobclient:2.2.0
[warn] +- org.apache.hadoop:hadoop-client:2.2.0
[warn] +- org.apache.spark:spark-core_2.10:1.3.0 (/home/luis/Test/build.sbt#L7-8)
[warn] +- Test:Test_2.10:1.0
让我们集中精力于其中一种依赖关系，因为警告和错误对所有这些都是相同的 . 假设org.apache.hadoop：hadoop-mapreduce-client-app：2.2.0
解析文件“ivy- [revision] .xml.original”的警告示例如下：
[warn] xml parsing: ivy-2.2.0.xml.original:18:69: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:19:11: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
[warn] xml parsing: ivy-2.2.0.xml.original:20:17: schema_reference.4: Failed to read schema document 'http://maven.apache.org/xsd/maven-4.0.0.xsd', because 1) could not find the document; 2) the document could not be read; 3) the root element of the document is not <xsd:schema>.
.......
.......
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: UNRESOLVED DEPENDENCIES ::
[warn] ::::::::::::::::::::::::::::::::::::::::::::::
[warn] :: org.apache.hadoop#hadoop-mapreduce-client-app;2.2.0: java.text.ParseException: [xml parsing: ivy-2.2.0.xml.original:18:69: cvc-elt.1: Cannot find the declaration of element 'project'. in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag project in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag parent in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag modelVersion in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag name in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag properties in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag applink.base in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[warn] , unknown tag mr.basedir in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
导致错误：
[error] (*:update) sbt.ResolveException: unresolved dependency: org.apache.hadoop#hadoop-mapreduce-client-app;2.2.0: java.text.ParseException: [xml parsing: ivy-2.2.0.xml.original:18:69: cvc-elt.1: Cannot find the declaration of element 'project'. in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag project in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag parent in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag modelVersion in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag groupId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag artifactId in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag version in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag name in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag properties in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag applink.base in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] , unknown tag mr.basedir in file:/home/luis/.ivy2/cache/org.apache.hadoop/hadoop-mapreduce-client-app/ivy-2.2.0.xml.original
[error] ]
只是为了澄清，文件ivy-2.2.0.xml.original的内容如下所示：
<?xml version="1.0"?>
<!--
Licensed under the Apache License, Version 2.0 (the "License");
you may not use this file except in compliance with the License.
You may obtain a copy of the License at
http://www.apache.org/licenses/LICENSE-2.0
Unless required by applicable law or agreed to in writing, software
distributed under the License is distributed on an "AS IS" BASIS,
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
See the License for the specific language governing permissions and
limitations under the License. See accompanying LICENSE file.
-->
<project xmlns="http://maven.apache.org/POM/4.0.0"
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:schemaLocation="http://maven.apache.org/POM/4.0.0
http://maven.apache.org/xsd/maven-4.0.0.xsd">
<parent>
<artifactId>hadoop-yarn</artifactId>
<groupId>org.apache.hadoop</groupId>
<version>2.2.0</version>
</parent>
<modelVersion>4.0.0</modelVersion>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-yarn-api</artifactId>
<version>2.2.0</version>
<name>hadoop-yarn-api</name>
<properties>
<!-- Needed for generating FindBugs warnings using parent pom -->
<yarn.basedir>${project.parent.basedir}</yarn.basedir>
</properties>
<build>
<plugins>
<plugin>
<groupId>org.apache.hadoop</groupId>
<artifactId>hadoop-maven-plugins</artifactId>
<executions>
<execution>
<id>compile-protoc</id>
<phase>generate-sources</phase>
<goals>
<goal>protoc</goal>
</goals>
<configuration>
<protocVersion>${protobuf.version}</protocVersion>
<protocCommand>${protoc.path}</protocCommand>
<imports>
<param>${basedir}/../../../hadoop-common-project/hadoop-common/src/main/proto</param>
<param>${basedir}/src/main/proto</param>
<param>${basedir}/src/main/proto/server</param>
</imports>
<source>
<directory>${basedir}/src/main/proto</directory>
<includes>
<include>yarn_protos.proto</include>
<include>yarn_service_protos.proto</include>
<include>applicationmaster_protocol.proto</include>
<include>applicationclient_protocol.proto</include>
<include>containermanagement_protocol.proto</include>
<include>server/yarn_server_resourcemanager_service_protos.proto</include>
<include>server/resourcemanager_administration_protocol.proto</include>
</includes>
</source>
<output>${project.build.directory}/generated-sources/java</output>
</configuration>
</execution>
</executions>
</plugin>
</plugins>
</build>
</project>
毕竟这个介绍......这些是我的问题：
-
文件"ivy-[revision].xml.original"的工作是什么，谁试图解析它 .
-
为什么无法识别xml标记 .
我会感激任何帮助！
SBT版本：0.13.8
谢谢 .
2 回答
我问你在帖子中提到的未回答的问题，我很高兴地宣布它已在几天前得到解答，并且建议的解决方案对我有用 .
尝试更新到sbt 0.13.9-RC3（按照http://www.scala-sbt.org/release/tutorial/Manual-Installation.html处的说明操作并获取https://dl.bintray.com/typesafe/ivy-releases/org.scala-sbt/sbt-launch/0.13.9-RC3/处的jar） .
最好的祝福，
/马丁
我终于成功地用sbt OFFLINE编译/打包/汇编了一个冻结库的子集 . 总结一下这个过程，我将重写一下上面的描述 .
这些是产生问题的步骤：
在名为ORIGIN的计算机中，让我们使用scala源创建一个sbt项目，并使用Internet连接进行编译
暂停互联网连接验证项目是否继续编译
复制SBT项目或复制到没有Internet连接的其他计算机（DESTINATION）
尝试编译 . 它不会起作用，因为sbt将尝试在线下载依赖项，而DESTINATION是一台OFFLINE计算机 .
这些是解决问题的步骤：
假设我们正在将sbt项目复制到没有Internet连接的新计算机（DESTINATION） . 我们必须确保DESTINATION的sbt版本和scala版本与ORIGIN中的版本相同 . 如果sbt或scala版本不同，那么当在DESTINATION中运行SBT时，sbt将尝试下载由于缺少连接而导致错误的正确版本 .
如果SBT和scala版本相同，那么我们必须从ORIGIN复制到DESTINATION以下文件夹：
ORIGIN：/home/userA/.ivy2
ORIGIN：/home/userA/.sbt/boot
确保正确配置了指向SBT和SCALA的环境变量
使用类似这样的build.sbt文件：
Build.sbt：
我不确定"provided"是否是强制性的，因为正确读取了joda依赖性
请注意，您在DESTINATION中可以使用的所有依赖项必须先前已在ORIGIN中下载并复制到DESTINATION .
我试图编译一个只使用Spark上下文（而不是spark-sql）的简单项目 . 所以它应该能够使用唯一依赖项进行编译：
libraryDependencies =“org.apache.spark”%%“spark-core”％“1.3.0”％“提供”
但是，我们测试它没有编译!! SBT抱怨“ Jackson ”包裹 . 也许“jackson”包部署在spark-sql依赖...无论如何，包括spark-sql使项目编译/打包/汇编 .
最后的评论：如果按照这个程序你没有成功编译，那么就有一个“手动”替代方案......我也成功脱机工作，没有SBT独立编译器......使用Eclipse进行Scala . 在eclipse中，您可以在图形界面中手动选择依赖项，并且可以选择所有spark，hadoop，mapreduce ...依赖项手动 . 一旦Eclipse识别出这些依赖关系，它就会在路径中编译你的类：“workspace / eclipse_project_name / bin”文件夹 . 然后你可以拿起它们并将它们手动打包到一个 jar 里（可能需要一个MANIFEST，但我想这没有必要） . 如果所有依赖关系已在集群中运行，则可以将此jar提交到集群 .