如何在python中使用pyarrow从S3读取分区的镶木地板文件

3 回答

• 7

  我设法使用最新版本的fastparquet和s3fs . 以下是相同的代码：

  import s3fs
  import fastparquet as fp
  s3 = s3fs.S3FileSystem()
  fs = s3fs.core.S3FileSystem()
  
  #mybucket/data_folder/serial_number=1/cur_date=20-12-2012/abcdsd0324324.snappy.parquet 
  s3_path = "mybucket/data_folder/*/*/*.parquet"
  all_paths_from_s3 = fs.glob(path=s3_path)
  
  myopen = s3.open
  #use s3fs as the filesystem
  fp_obj = fp.ParquetFile(all_paths_from_s3,open_with=myopen)
  #convert to pandas dataframe
  df = fp_obj.to_pandas()
  

  通过我们conversation向马丁指出我正确的方向

  NB ：基于benchmark，这比使用pyarrow要慢 . 一旦通过ARROW-1213在pyarrow中实现s3fs支持，我将更新我的答案

  我对单个迭代进行了快速基准测试，并将pyarrow和文件列表作为glob发送到fastparquet . 对于s3fs vs pyarrow，fastparquet更快，我的hackish代码 . 但我认为pyarrow s3fs一旦实现就会更快 .

  代码和基准如下：

  >>> def test_pq():
  ...     for current_file in list_parquet_files:
  ...         f = fs.open(current_file)
  ...         df = pq.read_table(f).to_pandas()
  ...         # following code is to extract the serial_number & cur_date values so that we can add them to the dataframe
  ...         #probably not the best way to split :)
  ...         elements_list=current_file.split('/')
  ...         for item in elements_list:
  ...             if item.find(date_partition) != -1:
  ...                 current_date = item.split('=')[1]
  ...             elif item.find(dma_partition) != -1:
  ...                 current_dma = item.split('=')[1]
  ...         df['serial_number'] = current_dma
  ...         df['cur_date'] = current_date
  ...         list_.append(df)
  ...     frame = pd.concat(list_)
  ...
  >>> timeit.timeit('test_pq()',number =10,globals=globals())
  12.078817503992468
  
  >>> def test_fp():
  ...     fp_obj = fp.ParquetFile(all_paths_from_s3,open_with=myopen)
  ...     df = fp_obj.to_pandas()
  
  >>> timeit.timeit('test_fp',number =10,globals=globals())
  2.0100269466638565e-06
  

  参考：

  • fastparquet

  • s3fs

  回复于 2024-05-04T15:25:08+08:00
• 1

  我们来讨论https://issues.apache.org/jira/browse/ARROW-1213和https://issues.apache.org/jira/browse/ARROW-1119 . 我们必须添加一些代码以允许pyarrow识别s3fs文件系统并添加一个shim /兼容类以符合S3FS 's slightly different filesystem API to pyarrow' .

  回复于 2024-05-04T15:25:08+08:00
• 2

  此问题已在2017年的this pull request中得到解决 .

  对于那些只想使用pyarrow从S3读取实木复合地板的人，这里有一个例子：

  import s3fs
  import pyarrow.parquet as pq
  from pyarrow.filesystem import S3FSWrapper
  
  fs = s3fs.S3FileSystem()
  bucket = "your-bucket"
  path = "your-path"
  
  # Python 3.6 or later
  p_dataset = pq.ParquetDataset(
      f"s3://{bucket}/{path}",
      filesystem=fs
  )
  df = p_dataset.read().to_pandas()
  
  # Pre-python 3.6
  p_dataset = pq.ParquetDataset(
      "s3://{0}/{1}".format(bucket, path),
      filesystem=fs
  )
  df = p_dataset.read().to_pandas()
  

  回复于 2024-05-04T15:25:08+08:00