带Hive的NiFi PutHiveStreaming处理器：连接到EndPoint失败

有人会在Nifi 1.3.0和Hive上帮助解决这个问题 . 我得到了与hive 1.2和Hive 2.1.1相同的错误 . 配置单元格是 partioned ， bucketed 并存储为 ORC 格式 .

The partition is created on hdfs but data failed on writing stage . 请检查日志如下：

[5:07 AM] papesdiop: Failed connecting to EndPoint {metaStoreUri='thrift://localhost:9083', database='mydb', table='guys', partitionVals=[dev] }
[5:13 AM] papesdiop: I get in log see next, hope it might help too:
[5:13 AM] papesdiop: Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable to acquire lock on {metaStoreUri='thrift://localhost:9083', database='mydb', table='guys', partitionVals=[dev] }
  at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:578)

FULL TRACE LOGS:

重新连接 . org.apache.thrift.transport.TTransportException：null org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:132）at org.apache.thrift.transport.TTransport.readAll（TTransport.java:86）at org.apache.thrift.protocol.TBinaryProtocol.readAll（TBinaryProtocol.java:429）在org.apache.thrift.protocol.TBinaryProtocol.readI32（TBinaryProtocol.java:318）在org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin（ TBinaryProtocol.java:219）在org.apache.thrift.TServiceClient.receiveBase（TServiceClient.java:69）在org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore $ Client.recv_lock（ThriftHiveMetastore.java:3906）在组织.apache.hadoop.hive.metastore.api.ThriftHiveMetastore $ Client.lock（ThriftHiveMetastore.java:3893）在org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock（HiveMetaStoreClient.java:1863）在sun.reflect.GeneratedMethodAccessor380来自java.lang.r的sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）的.invoke（未知来源） eflect.Method.invoke（Method.java:498）atg.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke（RetryingMetaStoreClient.java:152）at com.sun.proxy . $ Proxy126.lock（Unknown Source）at org .apache.hive.hcatalog.streaming.HiveEndPoint $ TransactionBatchImpl.beginNextTransactionImpl（HiveEndPoint.java:573）在org.apache.hive.hcatalog.streaming.HiveEndPoint $ TransactionBatchImpl.beginNextTransaction（HiveEndPoint.java:547）在org.apache.nifi .util.hive.HiveWriter.nextTxnBatch（HiveWriter.java:261）在org.apache.nifi.util.hive.HiveWriter . （HiveWriter.java:73）在org.apache.nifi.util.hive.HiveUtils.makeHiveWriter（ HiveUtils.java:46）在org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter（PutHiveStreaming.java:964）在org.apache.nifi.processors.hive.PutHiveStreaming.getOrCreateWriter（PutHiveStreaming.java:875）在组织.apache.nifi.processors.hive.PutHiveStreaming.lambda $ null $ 40（PutHiveStreaming.java:676）org.apache.nifi.processor.util.pattern.ExceptionHandler.execute（ExceptionHandler . java：127）atg.apache.nifi.processors.hive.PutHiveStreaming.lambda $ onTrigger $ 44（PutHiveStreaming.java:673）org.apache.nifi.controller.repository.StandardProcessSession.read（StandardProcessSession.java:2136）at at org.apache.nifi.controller.repository.StandardProcessSession.read在org.apache.nifi.processors（StandardProcessSession.java:2106）在org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger（PutHiveStreaming.java:627） . 在org.apache.nifi.processor.util.pattern hive.PutHiveStreaming.lambda $ onTrigger $ 36（PutHiveStreaming.java:551）在org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger（PartialFunctions.java:114） .RollbackOnFailure.onTrigger（RollbackOnFailure.java:184）在org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger（PutHiveStreaming.java:551）在org.apache.nifi.controller.StandardProcessorNode.onTrigger（StandardProcessorNode.java:1120 ）org.apach的org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call（ContinuallyRunProcessorTask.java:147） e.nifi.controller.tasks.ContinuallyRunProcessorTask.call（ContinuallyRunProcessorTask.java:47）在org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent $ 1.run（TimerDrivenSchedulingAgent.java:132）在java.util.concurrent.Executors $ RunnableAdapter .CALL（Executors.java:511）在java.util.concurrent.FutureTask.runAndReset（FutureTask.java:308）在java.util.concurrent.ScheduledThreadPoolExecutor中$ ScheduledFutureTask.access $ 301（ScheduledThreadPoolExecutor.java:180）在java.util中.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:294）在java.util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1149）在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:624 ）在java.lang.Thread.run（Thread.java:748)2017-09-07 06:41:31,015 DEBUG [Timer-4] oanprocessors.hive.PutHiveStreaming PutHiveStreaming [id = 13ed53d2-015e-1000-c7b1- 5af434c38751]开始向所有作家发送心跳2017-09-07 06：41：31,890 INFO [时间r-Driven Process Thread-3] hive.metastore尝试使用URI thrift连接到Metastore：// localhost：9083 2017-09-07 06：41：31,893 INFO[定时器驱动的进程线程-3] hive.metastore连接到Metastore . 2017年9月7日06：41：31911 ERROR [定时器驱动进程线程-3] oanprocessors.hive.PutHiveStreaming PutHiveStreaming [ID = 13ed53d2-015e-1000-c7b1-5af434c38751]未能创建HiveWriter endpoints ：{metaStoreUri = 'thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals = [dev]}：org.apache.nifi.util.hive.HiveWriter $ ConnectFailure：连接到EndPoint失败{metaStoreUri = 'thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals = [dev]} org.apache.nifi.util.hive.HiveWriter $ ConnectFailure：连接到EndPoint失败{metaStoreUri =' thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals = [dev]} org.apache.nifi.util.hive.HiveWriter . （HiveWriter.java:79）at org . 位于org.apache.nifi.processors.hive的org.apache.nifi.processors.hive.PutHiveStreaming.makeHiveWriter（PutHiveStreaming.java:964）的apache.nifi.util.hive.HiveUtils.makeHiveWriter（HiveUtils.java:46） . 在org.apache.ni上的PutHiveStreaming.getOrCreateWriter（PutHiveStreaming.java:875） fi.processors.hive.PutHiveStreaming.lambda $在org.apache.nifi.processors在org.apache.nifi.processor.util.pattern.ExceptionHandler.execute（ExceptionHandler.java:127）空$ 40（PutHiveStreaming.java:676） .hive.PutHiveStreaming.lambda $ onTrigger $ 44（PutHiveStreaming.java:673）位于org.apache.nifi.controller.repository.StandardProcessSession的org.apache.nifi.controller.repository.StandardProcessSession.read（StandardProcessSession.java:2136） . 读取（StandardProcessSession.java:2106）org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger（PutHiveStreaming.java:627）org.apache.nifi.processors.hive.PutHiveStreaming.lambda $ onTrigger $ 36（PutHiveStreaming.java） ：551）org.apache.nifi.processor.util.pattern.PartialFunctions.onTrigger（PartialFunctions.java:114）at org.apache.nifi.processor.util.pattern.RollbackOnFailure.onTrigger（RollbackOnFailure.java:184）at at org.apache.nifi.processors.hive.PutHiveStreaming.onTrigger（PutHiveStreaming.java:551）位于org.apache.nifi.controller.StandardProcessorNode . ongrigger（StandardProcessorNode.java:1120）org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call（ContinuallyRunProcessorTask.java:147）org.apache.nifi.controller.tasks.ContinuallyRunProcessorTask.call（ContinuallyRunProcessorTask.java:47） at org.apache.nifi.controller.scheduling.TimerDrivenSchedulingAgent $ 1.run（TimerDrivenSchedulingAgent.java:132）at java.util.concurrent.Executors $ RunnableAdapter.call（Executors.java:511）at java.util.concurrent.FutureTask . runAndReset（FutureTask.java:308）at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.access $ 301（ScheduledThreadPoolExecutor.java:180）at java.util.concurrent.ScheduledThreadPoolExecutor $ ScheduledFutureTask.run（ScheduledThreadPoolExecutor.java:294）at java . util.concurrent.ThreadPoolExecutor.runWorker（ThreadPoolExecutor.java:1149）在java.util.concurrent.ThreadPoolExecutor中的$ Worker.run（ThreadPoolExecutor.java:624）在java.lang.Thread.run（Thread.java:748）引起的：org.apache.nifi.util.hive.HiveWriter $ TxnBatchFailure：无法从EndPoint获取事务批处理：{metaStoreUri ='thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals = [dev]} org.apache.nifi.util.hive .hiveWriter.nextTxnBatch（HiveWriter.java:264）在org.apache.nifi.util.hive.HiveWriter . （HiveWriter.java:73）... 24个常见框架省略引起：org.apache.hive.hcatalog.streaming .TransactionError：无法在org.apache.hive.hcatalog.streaming上获取{metaStoreUri ='thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals = [dev]}的锁定 . HiveEndPoint $ TransactionBatchImpl.beginNextTransactionImpl（HiveEndPoint.java:578）位于org.apache.ncn . HiveWriter.java:261）...省略了25个常见框架引起：org.apache.thrift.transport.TTransportException：null org.apache.thrift.transport.TIOStreamTransport.read（TIOStreamTransport.java:13 2）在org.apache.thrift.transport.TTransport.readAll（TTransport.java:86）在org.apache.thrift.protocol.TBinaryProtocol.readAll（TBinaryProtocol.java:429）在org.apache.thrift.protocol.TBinaryProtocol .readI32（TBinaryProtocol.java:318）在org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin（TBinaryProtocol.java:219）在org.apache.thrift.TServiceClient.receiveBase（TServiceClient.java:69）在org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore $ Client.recv_lock（ThriftHiveMetastore.java:3906）org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore $ Client.lock（ThriftHiveMetastore.java:3893）at at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock（HiveMetaStoreClient.java:1863）at sun.reflect.GeneratedMethodAccessor380.invoke（Unknown Source）at sun.reflect.DelegatingMethodAccessorImpl.invoke（DelegatingMethodAccessorImpl.java:43）at java . 位于com.sun.proxy的org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.invoke（RetryingMetaStoreClient.java:152）的lang.reflect.Method.invoke（Method.java:498） . $ Proxy126.lock（未知来源）在org.apache.hive.hcatalog.streaming.HiveEndPoint $ TransactionBatchImpl.beginNextTransactionImpl（HiveEndPoint.java:573）...省略了27个常见帧2017-09-07 06：41：31,911 ERROR [定时器驱动的进程线程-3] oanprocessors.hive.PutHiveStreaming PutHiveStreaming [id = 13ed53d2-015e-1000-c7b1-5af434c38751]连接到Hive endpoints 时出错：ta在thrift：// localhost：9083 2017-09-07 06：41：31,911 DEBUG [计时器驱动的进程线程-3] oanprocessors.hive.PutHiveStreaming PutHiveStreaming [id = 13ed53d2-015e-1000-c7b1-5af434c38751]选择了产出资源;将不会被安排再次运行1000毫秒2017-09-07 06：41：31,912错误[计时器驱动的进程线程-3] oanprocessors.hive.PutHiveStreaming PutHiveStreaming [id = 13ed53d2-015e-1000-c7b1-5af434c38751] Hive Streaming连接/写入错误，流文件将受到处罚并路由到重试 . org.apache.nifi.util.hive.HiveWriter $ ConnectFailure：连接到EndPoint失败{metaStoreUri ='thrift：// localhost：9083'，database ='default'，table ='guys'，partitionVals =

The Hive table

CREATE TABLE mydb.guys（firstname string，lastname string）PARTITIONED BY（作业字符串）CLUSTERED BY（firstname）INTO 10 BUCKETS ROW FORMAT SERDE'org.apache.hadoop.hive.ql.io.orc.OrcSerde'ORORED AS ORC LOCATION'hdfs：// localhost：9000 / user / papesdiop / guys'TBLPROPERTIES（'transactional'='true'）

提前致谢