Hue上的Pig脚本错误

HUE版本:hue-3.9.0 cdh5.11.2 5098-1.cdh5.11.2.p0.7.el6.x86_64

CDH版本:2.6.0-cdh5.11.2

猪脚本:

A = LOAD'/ user / hdfs / test_data'; DUMP A;

在shell上运行此脚本时,它成功运行

但是当在Hue上运行它会挂起50%的检查日志

��h��׶9�A@���P  VERSIONAPPLICATION_ACLVIEW_APP 
MODIFY_APP APPLICATION_OWNERhdfs(&container_1513308088308_0001_02_000001�stderr240log4j:WARN No appenders could be found for logger (org.apache.hadoop.mapreduce.v2.app.MRAppMaster).
log4j:WARN Please initialize the log4j system properly.
log4j:WARN See http://logging.apache.org/log4j/1.2/faq.html#noconfig for more info.
stdout0syslog50982017-12-15 10:22:50,445 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Created MRAppMaster for application appattempt_1513308088308_0001_000002
2017-12-15 10:22:50,584 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Executing with tokens:
2017-12-15 10:22:50,584 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: YARN_AM_RM_TOKEN, Service: , Ident: (org.apache.hadoop.yarn.security.AMRMTokenIdentifier@1169afe1)
2017-12-15 10:22:50,713 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Kind: RM_DELEGATION_TOKEN, Service: 10.90.7.116:8032, Ident: (RM_DELEGATION_TOKEN owner=hdfs, renewer=oozie mr token, realUser=oozie, issueDate=1513308162650, maxDate=1513912962650, sequenceNumber=2, masterKeyId=2)
2017-12-15 10:22:51,202 INFO [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: OutputCommitter set in config org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter
2017-12-15 10:22:51,204 INFO [main] org.apache.hadoop.service.AbstractService: Service org.apache.hadoop.mapreduce.v2.app.MRAppMaster failed in state INITED; cause: java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2231)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:492)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:469)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMaster.java:1579)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1537)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1534)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1467)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2223)
    ... 12 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
    ... 13 more
2017-12-15 10:22:51,206 FATAL [main] org.apache.hadoop.mapreduce.v2.app.MRAppMaster: Error starting MRAppMaster
java.lang.RuntimeException: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2231)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:492)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$1.call(MRAppMaster.java:469)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.callWithJobClassLoader(MRAppMa
ster.java:1579)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.createOutputCommitter(MRAppMaster.java:469)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.serviceInit(MRAppMaster.java:391)
    at org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster$4.run(MRAppMaster.java:1537)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.Subject.doAs(Subject.java:422)
    at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.initAndStartAppMaster(MRAppMaster.java:1534)
    at org.apache.hadoop.mapreduce.v2.app.MRAppMaster.main(MRAppMaster.java:1467)
Caused by: java.lang.RuntimeException: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2199)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2223)
    ... 12 more
Caused by: java.lang.ClassNotFoundException: Class org.apache.oozie.action.hadoop.OozieLauncherOutputCommitter not found
    at org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:2105)
    at org.apache.hadoop.conf.Configuration.getClass(Configuration.java:2197)
    ... 13 more
2017-12-15 10:22:51,207 INFO [main] org.apache.hadoop.util.ExitUtil: Exiting with status 1

    VERSION*(&container_1513308088308_0001_02_000001none����data:BCFile.indexnone��

data:TFile.indexnone��66data:TFile.metanone�����h��׶9�A@���P

检查oozie日志中的oozie share lib:

2017-12-15 10:21:54,427  INFO ShareLibService:520 - SERVER[namenode] Share lib for oozie:[hdfs://namenode:8020/user/oozie/share/lib/lib_20171215093741/oozie/json-simple-1.1.jar, hdfs://namenode:8020/user/oozie/share/lib/lib_20171215093741/oozie/oozie-hadoop-utils-2.5.0-cdh5.2.0.oozie-4.0.0-cdh5.2.0.jar, hdfs://namenode:8020/user/oozie/share/lib/lib_20171215093741/oozie/oozie-sharelib-oozie-4.0.0-cdh5.2.0.jar]

通过命令检查oozie sharelib

sudo -u oozie oozie admin -shareliblist -oozie http://namenode:11000/oozie
[Available ShareLib]
hive
distcp
mapreduce-streaming
oozie
hcatalog
hive2
sqoop
pig

更多日志我找到
enter image description here

2017-12-15 10:53:48,473  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@:start:] Start action [0000001-171215102152358-oozie-oozi-W@:start:] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-12-15 10:53:48,473  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@:start:] [***0000001-171215102152358-oozie-oozi-W@:start:***]Action status=DONE
2017-12-15 10:53:48,473  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@:start:] [***0000001-171215102152358-oozie-oozi-W@:start:***]Action updated in DB!
2017-12-15 10:53:48,561  INFO WorkflowNotificationXCommand:520 - SERVER[namenode] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@:start:] No Notification URL is defined. Therefore nothing to notify for job 0000001-171215102152358-oozie-oozi-W@:start:
2017-12-15 10:53:48,562  INFO WorkflowNotificationXCommand:520 - SERVER[namenode] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[] No Notification URL is defined. Therefore nothing to notify for job 0000001-171215102152358-oozie-oozi-W
2017-12-15 10:53:48,576  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@pig-5760] Start action [0000001-171215102152358-oozie-oozi-W@pig-5760] with user-retry state : userRetryCount [0], userRetryMax [0], userRetryInterval [10]
2017-12-15 10:53:48,990  INFO PigActionExecutor:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@pig-5760] checking action, hadoop job ID [job_1513308088308_0002] status [RUNNING]
2017-12-15 10:53:48,992  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@pig-5760] [***0000001-171215102152358-oozie-oozi-W@pig-5760***]Action status=RUNNING
2017-12-15 10:53:48,993  INFO ActionStartXCommand:520 - SERVER[namenode] USER[hdfs] GROUP[-] TOKEN[] APP[pig-app-hue-script] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@pig-5760] [***0000001-171215102152358-oozie-oozi-W@pig-5760***]Action updated in DB!
2017-12-15 10:53:49,028  INFO WorkflowNotificationXCommand:520 - SERVER[namenode] USER[-] GROUP[-] TOKEN[-] APP[-] JOB[0000001-171215102152358-oozie-oozi-W] ACTION[0000001-171215102152358-oozie-oozi-W@pig-5760] No Notification URL is defined. Therefore nothing to notify for job 0000001-171215102152358-oozie-oozi-W@pig-5760

登录yarn-ndoe-manager:

2017-12-15 10:53:54,841 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exit code from container container_1513308088308_0002_02_000001 is : 1
2017-12-15 10:53:54,841 WARN org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor: Exception from container-launch with container ID: container_1513308088308_0002_02_000001 and exit code: 1
ExitCodeException exitCode=1:
        at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
        at org.apache.hadoop.util.Shell.run(Shell.java:504)
        at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
        at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
        at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
        at java.util.concurrent.FutureTask.run(FutureTask.java:266)
        at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
        at java.lang.Thread.run(Thread.java:748)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exception from container-launch.
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Container id: container_1513308088308_0002_02_000001
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Exit code: 1
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor: Stack trace: ExitCodeException exitCode=1:
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.util.Shell.runCommand(Shell.java:601)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.util.Shell.run(Shell.java:504)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.util.Shell$ShellCommandExecutor.execute(Shell.java:786)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.yarn.server.nodemanager.DefaultContainerExecutor.launchContainer(DefaultContainerExecutor.java:213)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:302)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch.call(ContainerLaunch.java:82)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at java.util.concurrent.FutureTask.run(FutureTask.java:266)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-12-15 10:53:54,841 INFO org.apache.hadoop.yarn.server.nodemanager.ContainerExecutor:       at java.lang.Thread.run(Thread.java:748)
2017-12-15 10:53:54,842 WARN org.apache.hadoop.yarn.server.nodemanager.containermanager.launcher.ContainerLaunch: Container exited with a non-zero exit code 1
2017-12-15 10:53:54,842 INFO org.apache.hadoop.yarn.server.nodemanager.containermanager.container.Container: Container container_1513308088308_0002_02_000001 transitioned from RUNNING to EXITED_WITH_FAILURE

我怎么能修好!!!

回答(0)