基本问题:我可以使用Presto作为我的Hive Metastore的基础吗?
我正在尝试在亚马逊aws上安装presto和hive,我在创建Hive Metastore时遇到了麻烦 . 除了未能连接到Metastore之外,Presto似乎已安装并正常工作 . 问题是当我运行Hive时,它无法创建一个Metastore .
当我运行Hive时,我收到以下错误:
Exception in thread "main" java.lang.RuntimeException: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:444)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.lang.RuntimeException: Unable to instantiate org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1449)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
... 7 more
Caused by: java.lang.reflect.InvocationTargetException
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
... 12 more
Caused by: MetaException(message:Could not connect to meta store using any of the URIs provided. Most recent failure: org.apache.thrift.transport.TTransportException: java.net.ConnectException: Connection refused
at org.apache.thrift.transport.TSocket.open(TSocket.java:185)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:358)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native Method)
at sun.reflect.NativeConstructorAccessorImpl.newInstance(NativeConstructorAccessorImpl.java:62)
at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
at java.lang.reflect.Constructor.newInstance(Constructor.java:408)
at org.apache.hadoop.hive.metastore.MetaStoreUtils.newInstance(MetaStoreUtils.java:1447)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.<init>(RetryingMetaStoreClient.java:63)
at org.apache.hadoop.hive.metastore.RetryingMetaStoreClient.getProxy(RetryingMetaStoreClient.java:73)
at org.apache.hadoop.hive.ql.metadata.Hive.createMetaStoreClient(Hive.java:2661)
at org.apache.hadoop.hive.ql.metadata.Hive.getMSC(Hive.java:2680)
at org.apache.hadoop.hive.ql.session.SessionState.start(SessionState.java:425)
at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:672)
at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:616)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:483)
at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.net.ConnectException: Connection refused
at java.net.PlainSocketImpl.socketConnect(Native Method)
at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:345)
at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:206)
at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:188)
at java.net.SocksSocketImpl.connect(SocksSocketImpl.java:392)
at java.net.Socket.connect(Socket.java:589)
at org.apache.thrift.transport.TSocket.open(TSocket.java:180)
... 20 more
)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.open(HiveMetaStoreClient.java:404)
at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.<init>(HiveMetaStoreClient.java:215)
at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.<init>(SessionHiveMetaStoreClient.java:73)
... 17 more
我的conf / hive-site.xml如下:
<property>
<name>hive.metastore.uris</name>
<value>thrift://ec2..........amazonaws.com:8080</value>
<description>Thrift URI for the remote metastore. Used by metastore client to connect to remote metastore. </description>
<property>
<name>javax.jdo.option.ConnectionURL</name>
<value>jdbc:presto://ec2...........amazonaws.com:9083/hive;databaseName=metastore_db;create=true</value>
<description>JDBC connect string for a JDBC metastore</description>
<property>
<name>javax.jdo.option.ConnectionDriverName</name>
<value>presto-jdbc-0.90.jar</value>
<description>Driver class name for a JDBC metastore</description>
我可以使用presto来创建我的Metastore,还是必须使用mySQL或derby?如果可以的话,我将非常感谢能够解决这个问题的任何帮助 .
谢谢!
4 回答
从理论上讲,你应该能够做到这一点 . 但Presto需要一个后端;哪个支持Presto用于您的Metastore数据库?无论您使用哪个后端,都需要支持插入 .
有一个Presto bootstrap action for EMR可以为您处理所有设置 .
感谢到目前为止的回复 .
@Dain Sundstrom:我尝试了你对引导动作的建议,但没有真正的快乐 .
在对蜂巢结构进行了更多的研究后,我认为不可能像我一样尝试使用presto . 根据我现在的理解,presto使用hive作为其数据库,但是hive本身需要mySQL或derby的底层数据库,这是我出错的地方 . 所以我的计划是现在安装mySQL,链接配置单元,然后就这样做 . 该计划的第2步是祈祷它没有太多麻烦 .
谢谢你的帮助 .
将Metastore存储到MySQL时,可以使用以下步骤:
完成后,将以下内容添加到$ HIVE_HOME / conf /中的hive-site.xml
然后启动Metastore服务:
蜂巢 - 服务Metastore&
如果您想了解更多或者您遇到困难,请发表评论 .
此致,德维普拉萨德 .