将文件解压缩到s3

3 回答

• 0

  您可以尝试https://www.cloudzipinc.com/将几种不同格式的存档从S3解压缩/扩展到存储桶中的目标 . 我用它将数字目录的组件解压缩到S3中 .

  回复于 2024-05-10T20:38:32+08:00
• 0

  通过使用ec2实例解决了 . 将s3文件复制到ec2中的本地目录，并将该目录复制回S3存储桶 .

  回复于 2024-05-10T20:38:32+08:00
• 0

  示例解压缩到ec2实例中的本地目录

  def s3Unzip(srcBucket,dst_dir):  
  '''
  function to decompress the s3 bucket contents to local machine 
  
  Args:
  srcBucket (string): source bucket name 
  dst_dir (string): destination location in the local/ec2 local file system
  
  Returns:
  None
  '''      
  #bucket = s3.lookup(bucket)
  s3=s3Conn
  path=''
  
  bucket = s3.lookup(bucket_name)
  for key in bucket:
      path = os.path.join(dst_dir, key.name)
      key.get_contents_to_filename(path)
      if path.endswith('.zip'):
          opener, mode = zipfile.ZipFile, 'r'
      elif path.endswith('.tar.gz') or path.endswith('.tgz'):
          opener, mode = tarfile.open, 'r:gz'
      elif path.endswith('.tar.bz2') or path.endswith('.tbz'):
          opener, mode = tarfile.open, 'r:bz2'
      else: 
          raise ValueError ('unsuppported format')
  
      try:
          os.mkdir(dst_dir)
          print ("local directories created")
      except Exception:
          logger_s3.warning ("Exception in creating local directories to extract zip file/ folder already existing")    
      cwd = os.getcwd()
      os.chdir(dst_dir)
  
      try:
          file = opener(path, mode)
          try: file.extractall()
          finally: file.close()
          logger_s3.info('(%s) extracted successfully to %s'%(key ,dst_dir))
      except Exception as e:
          logger_s3.error('failed to extract (%s) to %s'%(key ,dst_dir))
          os.chdir(cwd)   
  s3.close
  

  示例代码上传到mysql实例

  使用“LOAD DATA LOCAL INFILE”查询直接上传到mysql

  def upload(file_path,timeformat):
  '''
  function to upload a  csv file data to mysql rds 
  
  Args:
  file_path (string): local file path
  timeformat (string): destination bucket to copy data
  
  Returns:
  None    
  '''  
  for file in file_path:
      try:
          con = connect()
          cursor = con.cursor()
  
          qry="""LOAD DATA LOCAL INFILE '%s' INTO TABLE xxxx FIELDS TERMINATED BY ',' LINES TERMINATED BY '\n' (col1 , col2 ,col3, @datetime , col4 ) set datetime = str_to_date(@datetime,'%s');""" %(file,timeformat)
          cursor.execute(qry)
          con.commit()
          logger_rds.info ("Loading file:"+file)
      except Exception:
          logger_rds.error ("Exception in uploading "+file)
           ##Rollback in case there is any error
          con.rollback()
  cursor.close()
  # disconnect from server
  con.close()
  

  回复于 2024-05-10T20:38:32+08:00