当我将local的值设置为大于1时,为什么pyspark会使用错误

当我将 local 的值设置为1时,操作正常,但设置为2时,错误消息报告如下

from pyspark import SparkContext
# Changing 1 to 2 will give you an error
sc = SparkContext("local[2]", "sort")


class MySort:
    def __init__(self, tup):
        self.tup = tup

    def __gt__(self, other):
        if self.tup[0] > other.tup[0]:
            return True
        elif self.tup[0] == other.tup[0]:
            if self.tup[1] >= other.tup[1]:
                return True
            else:
                return False
        else:
            return False


r1 = sc.parallelize([(1, 2), (2, 2), (2, 3), (2, 1), (1, 3)])
r2 = r1.sortBy(MySort)
print(r2.collect())
Caused by: org.apache.spark.api.python.PythonException: Traceback (most recent call last):
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\worker.py", line 230, in main
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\worker.py", line 225, in process
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\serializers.py", line 376, in dump_stream
    bytes = self.serializer.dumps(vs)
  File "E:\spark2.3.1\spark-2.3.1-bin-hadoop2.7\python\lib\pyspark.zip\pyspark\serializers.py", line 555, in dumps
    return pickle.dumps(obj, protocol)
_pickle.PicklingError: Can't pickle : attribute lookup MySort on __main__ failed

回答(2)

2 years ago

我认为你需要在你的 class 中使用文件添加参数spark-submit:

--py-files your_file.py

因为spark需要将此类传递给另一个worker .

2 years ago

它真正有趣的属性我以前不知道它 . 我认为当你使用单核时,类不会被腌制(在其他地方使用类需要pickle) . 但你仍然可以使用函数(我假设你按前两个值排序值):

key_func = lambda tup : tup[:2]

r1 = sc.parallelize([(1, 2), (2, 2), (2, 3), (2, 1), (1, 3)])
r2 = r1.sortBy(key_func)
print(r2.collect())