首页 文章

当我将local的值设置为大于1时，为什么pyspark会使用错误

提问于
浏览 次
1

当我将 local 的值设置为1时，操作正常，但设置为2时，错误消息报告如下

from pyspark import SparkContext
# Changing 1 to 2 will give you an error
sc = SparkContext("local[2]", "sort")


class MySort:
    def __init__(self, tup):
        self.tup = tup

    def __gt__(self, other):
        if self.tup[0] > other.tup[0]:
            return True
        elif self.tup[0] == other.tup[0]:
            if self.tup[1] >= other.tup[1]:
                return True
            else:
                return False
        else:
            return False


r1 = sc.parallelize([(1, 2), (2, 2), (2, 3), (2, 1), (1, 3)])
r2 = r1.sortBy(MySort)
print(r2.collect())

Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\worker.py", line 230, in main
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\worker.py", line 225, in process
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\serializers.py", line 376, in dump_stream
    bytes = self.serializer.dumps(vs)
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\serializers.py", line 555, in dumps
    return pickle.dumps(obj, protocol)
_pickle.PicklingError: Can't pickle : attribute lookup MySort on __main__ failed
python pyspark

2 回答

  • 0

    我认为你需要在你的 class 中使用文件添加参数spark-submit：

    --py-files your_file.py

    因为spark需要将此类传递给另一个worker .

    回复于
  • 0

    它真正有趣的属性我以前不知道它 . 我认为当你使用单核时，类不会被腌制（在其他地方使用类需要pickle） . 但你仍然可以使用函数（我假设你按前两个值排序值）：

    key_func = lambda tup : tup[:2]

r1 = sc.parallelize([(1, 2), (2, 2), (2, 3), (2, 1), (1, 3)])
r2 = r1.sortBy(key_func)
print(r2.collect())
    回复于

相关问题