首页 文章

在编写张量流变换元数据时,管道将在GCP上失败

提问于
浏览
0

我希望这里有人可以提供帮助 . 我一直在谷歌搜索这个错误,但没有找到任何东西 .

我有一个在本地执行时工作正常但在GCP上执行时失败的管道 . 以下是我收到的错误消息 .

工作流程失败 . 原因:S03:写入变换fn / WriteMetadata / ResolveBeamFutures / CreateSingleton / Read Write transform fn / WriteMetadata / ResolveBeamFutures / ResolveFutures / Do Write transform fn / WriteMetadata / WriteMetadata failed . ,尝试了4次工作项而没有成功 . 每次 Worker 最终都失去与服务的联系 . 试图工作项:Traceback(最近一次调用最后一次):文件“preprocess.py”,第491行,在main()文件“preprocess.py”,第487行,在主transform_data(args,pipeline_options,runner)文件中“preprocess.py”,第451行,在transform_data中eval_data | ='Identity eval'>> beam.ParDo(Identity())文件“/Library/Python/2.7/site-packages/apache_beam/pipeline.py”,第335行,在退出self.run() . wait_until_finish()文件“/Library/Python/2.7/site-packages/apache_beam/runners/dataflow/dataflow_runner.py”,第897行,在wait_until_finish(self.state,getattr(self . _runner,'last_error_msg',无)),self)apache_beam.runners.dataflow.dataflow_runner.DataflowRuntimeException:数据流管道失败 . 状态:FAILED,错误:回溯(最近一次调用最后一次):文件“/usr/local/lib/python2.7/dist-packages/dataflow_worker/batchworker.py”,第582行,在do_work work_executor.execute()文件中/usr/local/lib/python2.7/dist-packages/dataflow_worker/executor.py“,第166行,执行op.start()文件”apache_beam / runners / worker / operations.py“,第294行,在apache_beam中.runners.worker.operations.DoOperation.start(apache_beam / runners / worker / operations.c:10607)def start(self):文件“apache_beam / runners / worker / operations.py”,第295行,在apache_beam.runners中 . self.coped_start_state的worker.operations.DoOperation.start(apache_beam / runners / worker / operations.c:10501):文件“apache_beam / runners / worker / operations.py”,第300行,位于apache_beam.runners.worker.operations中 . DoOperation.start(apache_beam / runners / worker / operations.c:9702)pickler.loads(self.spec.serialized_fn))文件“/usr/local/lib/python2.7/dist-packages/apache_beam/internal/pickler . py“,第225行,在load中返回dill.loads(s)文件”/ usr /local/lib/python2.7/dist-packages/dill/dill.py“,第277行,在load load load(file)文件中”/usr/local/lib/python2.7/dist-packages/dill/dill .py“,第266行,在load obj = pik.load()文件”/usr/lib/python2.7/pickle.py“,第858行,在load dispatchkey文件”/usr/lib/python2.7/pickle .py“,第1083行,在load_newobj中obj = cls.new(cls,* args)TypeError:new()需要4个参数(给定1个)

有任何想法吗??

谢谢,

佩德罗

1 回答

  • 0

    如果管道在本地工作但在GCP上失败,则可能是您遇到版本不匹配 .

    您在本地和GCP上运行什么TF,tf.Transform,梁版本?

相关问题