pyspark模块在spark正在使用的python实例中不可用-Java 学习之路

我正在使用我自己没有在AWS中设置的EMR .

我试图理解Python解释器火花正在使用什么，在我的.bashrc中我有以下设置 export PYSPARK_PYTHON=/mnt/anaconda/bin/python

当我运行spark-submit命令时，我使用sys.executable打印到python解释器的路径，这确实是它使用的解释器 .

但是，当我专门去那个文件夹时，用 ./python 启动那个Python实例，然后尝试 import pyspark 我得到：

>>> import pyspark
Traceback (most recent call last):
  File "<stdin>", line 1, in <module>
ImportError: No module named pyspark

这里发生了什么？这个解释器不应该安装pyspark模块吗？

1 回答

当你调用spark-submit时，它会将它自己的python文件夹和py4j库添加到PYTHONPATH中 . 他们住在安装的spark文件夹中，它们不是直接安装到你正在使用的python的site-packages中 . 例如，如果我pyspark打印python路径：

$ pyspark
Python 2.6.6 (r266:84292, Aug 18 2016, 15:13:37) 
[GCC 4.4.7 20120313 (Red Hat 4.4.7-17)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
17/10/17 15:34:04 WARN NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
17/10/17 15:34:04 WARN Utils: Your hostname, rwidmaier resolves to a loopback address: 127.0.0.1; using 10.4.248.126 instead (on interface em1)
17/10/17 15:34:04 WARN Utils: Set SPARK_LOCAL_IP if you need to bind to another address
/home/rwidmaier/apps/spark/python/pyspark/context.py:195: UserWarning: Support for Python 2.6 is deprecated as of Spark 2.0.0
  warnings.warn("Support for Python 2.6 is deprecated as of Spark 2.0.0")
17/10/17 15:34:09 WARN ObjectStore: Version information not found in metastore. hive.metastore.schema.verification is not enabled so recording the schema version 1.2.0
17/10/17 15:34:09 WARN ObjectStore: Failed to get database default, returning NoSuchObjectException
17/10/17 15:34:09 WARN ObjectStore: Failed to get database global_temp, returning NoSuchObjectException
Welcome to
      ____              __
     / __/__  ___ _____/ /__
    _\ \/ _ \/ _ `/ __/  '_/
   /__ / .__/\_,_/_/ /_/\_\   version 2.2.0
      /_/

Using Python version 2.6.6 (r266:84292, Aug 18 2016 15:13:37)
SparkSession available as 'spark'.
>>> import sys
>>> from pprint import pprint
>>> pprint(sys.path)
['',
 u'/tmp/spark-0a08efa7-714a-498b-b5e9-bba48684d52d/userFiles-52012e25-d7af-4599-a214-9637141ed4ec',
 '/usr/lib64/python2.6/site-packages/deluge-1.3.11-py2.6-linux-x86_64.egg',
 '/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip',
 '/home/ryan/apps/spark/python',
 '/usr/lib64/python26.zip',
 '/usr/lib64/python2.6',
 '/usr/lib64/python2.6/plat-linux2',
 '/usr/lib64/python2.6/lib-tk',
 '/usr/lib64/python2.6/lib-old',
 '/usr/lib64/python2.6/lib-dynload',
 '/usr/lib64/python2.6/site-packages',
 '/usr/lib64/python2.6/site-packages/gst-0.10',
 '/usr/lib64/python2.6/site-packages/gtk-2.0',
 '/usr/lib64/python2.6/site-packages/webkit-1.0',
 '/usr/lib/python2.6/site-packages']

您可以在列表中看到它已手动添加：

/home/ryan/apps/spark/python/lib/py4j-0.10.4-src.zip
/ home / ryan / apps / spark / python

要确保始终使用正确的库设置解释器，您只需将PYSPARK_PYTHON设置为指向您要使用的python安装，然后运行“pyspark”而不是直接运行python . 或者，如果您想提供驱动程序，则可以使用spark-submit .

回复于 2024-04-20T12:23:43+08:00

pyspark模块在spark正在使用的python实例中不可用

1 回答

相关问题