首页 文章

在oozie Java Action中传递HBase凭据

提问于
浏览
2

我需要安排一个与安全的hbase交互的oozie Java操作,所以我需要为Java操作提供hbase凭据 . 我使用的是安全的hortonworks 2.2环境,我的工作流XML如下所示

<workflow-app xmlns="uri:oozie:workflow:0.4" name="solr-wf">
    <credentials>
         <credential name="hbase" type="hbase">
         </credential>
      </credentials>

    <start to="java-node"/>
    <action name="java-node" cred="hbase">
        <java>  
             <job-tracker>${jobTracker}</job-tracker>
             <name-node>${nameNode}</name-node>
             <main-class>com.test.hbase.TestHBaseSecure</main-class>
            <arg>${arg1}</arg>
        </java>
        <ok to="end"/>
        <error to="fail"/>
    </action>
    <kill name="fail">
        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end"/>
</workflow-app>

我还修改了oozie属性以包含HbaseCredentials类

oozie.credentials.credentialclasses=hcat=org.apache.oozie.action.hadoop.HCatCredentials,hbase=org.apache.oozie.action.hadoop.HbaseCredentials

但我无法运行它抛出错误的工作,下面是堆栈跟踪

java.lang.NoClassDefFoundError: org/apache/hadoop/hbase/HBaseConfiguration
    at org.apache.oozie.action.hadoop.HbaseCredentials.copyHbaseConfToJobConf(HbaseCredentials.java:60)
    at org.apache.oozie.action.hadoop.HbaseCredentials.addtoJobConf(HbaseCredentials.java:49)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.setCredentialTokens(JavaActionExecutor.java:1054)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.submitLauncher(JavaActionExecutor.java:913)
    at org.apache.oozie.action.hadoop.JavaActionExecutor.start(JavaActionExecutor.java:1135)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:228)
    at org.apache.oozie.command.wf.ActionStartXCommand.execute(ActionStartXCommand.java:63)
    at org.apache.oozie.command.XCommand.call(XCommand.java:281)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:323)
    at org.apache.oozie.service.CallableQueueService$CompositeCallable.call(CallableQueueService.java:252)
    at org.apache.oozie.service.CallableQueueService$CallableWrapper.run(CallableQueueService.java:174)
    at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
    at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
    at java.lang.Thread.run(Thread.java:745)

其他工作运行良好,只有hbase交互失败的工作 . 我已经在我的lib目录中包含了所有的hbase jar,我无法弄清楚这个问题 .

Updated workflow.xml:

<workflow-app xmlns="uri:oozie:workflow:0.4" name="${appName}">
<credentials>
        <credential name="hbase-cred" type="hbase">
            <property>
                <name>hbase.master.kerberos.principal</name>
                <value>hbase/_HOST@ABC.COM</value>
            </property>

            <property>
                <name>hbase.master.keytab.file</name>
                <value>/etc/security/keytabs/hbase.service.keytab</value>
            </property>

            <property>
                <name>hbase.regionserver.kerberos.principal</name>
                <value>hbase/_HOST@ABC.COM</value>
            </property>

            <property>
                <name>hbase.regionserver.keytab.file</name>
                <value>/etc/security/keytabs/hbase.service.keytab</value>
            </property>

            <property>
                <name>hbase.security.authentication</name>
                <value>kerberos</value>
            </property>

            <property>
                <name>hbase.zookeeper.quorum</name>
                <value>dev1-dn2,dev1-dn3,dev1-dn1</value>
            </property>

            <property>
                <name>zookeeper.znode.parent</name>
                <value>/hbase-secure</value>
            </property>
        </credential>


    </credentials>
    <start to="java-node" />
    <action name="java-node" cred='hbase-cred'>
        <java>
            <job-tracker>${jobTracker}</job-tracker>
            <name-node>${nameNode}</name-node>
            <main-class>com.test.hbase.TestHBaseSecure</main-class>
        </java>
        <ok to="end" />
        <error to="fail" />
    </action>
    <kill name="fail">
        <message>Java failed, error message[${wf:errorMessage(wf:lastErrorNode())}]</message>
    </kill>
    <end name="end" />
</workflow-app>

2 回答

  • 2

    该解决方案在HDP2.2.8上进行了测试:

    • 复制到 /usr/hdp/current/oozie-server/oozie-server/webapps/oozie/WEB-INF/lib 以下 jar :

    • hbase-client - * - hadoop2.jar

    • hbase-common - * - hadoop2.jar

    • hbase-protocol - * - hadoop2.jar

    • hbase-server - * - hadoop2.jar

    • htrace-core-2.04.jar

    • 重启Oozie服务器 .

  • 5

    这些"credentials"由 Oozie service 管理,而不是由您的工作管理 .

    所以,如果HortonWorks在包装发行版方面做得不错......

    安装时_259_ hbase-common-*-hadoop2.jar 将部署在 /usr/hdp/current/oozie-client/libserver/

    • 该JAR不会因类org.apache.hadoop.conf.Configuration的定义而与其他JAR冲突

    • 最后您将能够在Oozie中管理HBase凭据

    我们的Prod集群上安装的HDP2.2.4不是这种情况 . Arghh . The damn thing is broken in that damn release. You've got to manage the Kerberos ticket all by yourself ,从HDFS下载密钥表 <file> 并在实际连接到HBase之前创建TGT . 我们去过那儿 .

    有关如何完成的一些见解,请查看that post .

相关问题