首页 文章

如何使用PHP和aws-sdk v3将大型档案上传到Amazon Glacier?

提问于
浏览
0

这是我第一次使用亚马逊的任何东西 . 我正在尝试使用PHP SDK V3将多个文件上传到Amazon Glacier . 然后,这些文件需要由亚马逊合并为一个 .

这些文件存储在cPanel的主目录中,必须通过cron作业上传到Amazon Glacier .

我知道我必须使用上传多部分方法,但我不确定它需要哪些其他功能才能使其工作 . 我也不确定我计算和传递变量的方式是否正确 .

这是我到目前为止的代码:

<?php
require 'aws-autoloader.php';

use Aws\Glacier\GlacierClient;
use Aws\Glacier\TreeHash;

//############################################
//DEFAULT VARIABLES
//############################################
$key = 'XXXXXXXXXXXXXXXXXXXX';
$secret = 'XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX';   
$accountId = '123456789123';
$vaultName = 'VaultName';
$partSize = '4194304';
$fileLocation = 'path/to/files/';

//############################################
//DECLARE THE AMAZON CLIENT
//############################################
$client = new GlacierClient([
    'region' => 'us-west-2',
    'version' => '2012-06-01',
    'credentials' => array(
        'key'    => $key,
        'secret' => $secret,
  )
]);

//############################################
//GET THE UPLOAD ID
//############################################
$result = $client->initiateMultipartUpload([
    'partSize' => $partSize,
    'vaultName' => $vaultName
]);
$uploadId = $result['uploadId'];

//############################################
//GET ALL FILES INTO AN ARRAY
//############################################
$files = scandir($fileLocation);
unset($files[0]);
unset($files[1]);
sort($files);

//############################################
//GET SHA256 TREE HASH (CHECKSUM)
//############################################
$th = new TreeHash();
//GET TOTAL FILE SIZE
foreach($files as $part){
    $filesize = filesize($fileLocation.$part);
    $total = $filesize;
    $th = $th->update(file_get_contents($fileLocation.$part));
}
$totalchecksum = $th->complete();

//############################################
//UPLOAD FILES
//############################################
foreach ($files as $key => $part) {
    //HASH CONTENT
    $filesize = filesize($fileLocation.$part);
    $rangeSize = $filesize-1;
    $range = 'bytes 0-'.$rangeSize.'/*';
    $sourcefile = $fileLocation.$part;

    $result = $client->uploadMultipartPart([
        'accountId' => $accountId,
        'checksum' => '',
        'range' => $range,
        'sourceFile' => $sourcefile,
        'uploadId' => $uploadId,
        'vaultName' => $vaultName
    ]);
}

//############################################
//COMPLETE MULTIPART UPLOAD
//############################################
$result = $client->completeMultipartUpload([
    'accountId' => $accountId,
    'archiveSize' => $total,
    'checksum' => $totalchecksum,
    'uploadId' => $uploadId,
    'vaultName' => $vaultName,
]);
?>

似乎新Glacier客户端的声明正在工作,我确实收到了一个上传ID,但如果我做得对,我不是100% . 文件需要上传到然后合并的Amazon Glacier Vault仍然是空的,我不确定文件是否只显示completeMultipartUpload已成功执行的文件 .

运行代码时,我还收到以下错误:

致命错误:未捕获异常'Aws \ Glacier \ Exception \ GlacierException',消息'执行错误'CompleteMultipartUpload“on”https://glacier.us-west-2.amazonaws.com/XXXXXXXXXXXX/vaults/XXXXXXXXXXX/multipart-uploads / cTI0Yfk6xBYIQ0V-rhq6AcdHqd3iivRJfyYzK6-NV1yn9GQvJyYCoSrXrrrx4kfyGm6m9PUEAq4M0x6duXm5MD8abn-M“; AWS HTTP错误:客户端错误:403 InvalidSignatureException(client):我们计算的请求签名与您提供的签名不匹配 . 检查您的AWS Secret Access Key和签名方法 . 有关详细信息,请参阅服务文档此请求的规范字符串应该是'POST / XXXXXXXXXXX / vaults / XXXXXXXXX / multipart-uploads / cTI0Yfk6xBYIQ0V-rhq6AcdHqd3iivRJfyYzK6-NV1yn9GQvJyYCoSrXrrrx4kfyGm6m9PUEAq4M0x6duXm5MD8abn-M host:glacier.us-west-2.amazonaws.com x-amz-archive-size: 1501297 x-amz-date:20151016T081455Z x-amz-glacier-version:2012-06-01 x-amz-sha256-tree-hash:?[qiu茫茫氓虏脕脕鹿陇陋陋陇[; K脳T host; x-amz-archive-size; x-amz-date; x-amz-glacier-version; x-am in /home/XXXXXXXXXXX/public_html/XXXXXXXXXX/Aws/WrappedHttpHandler.php on 152

有没有更简单的方法来做到这一点?如果有帮助的话,我也有完整的SSH访问权限 .

2 回答

  • 1

    我觉得你误解了uploadMultipartPart . uploadMultipartPart表示您在多个部分上传1个大文件 . 然后执行completeMultipartUpload以标记您已完成上传一个文件 .

    从您的代码看起来您正在上传多个文件 .

    您可能实际上不需要使用uploadMultipartPart

    也许你可以使用常规的“uploadArchive”?

    参考:

    https://blogs.aws.amazon.com/php/post/Tx7PFHT4OJRJ42/Uploading-Archives-to-Amazon-Glacier-from-PHP

  • 1

    我已经在PHP SDK V3(版本3)中管理了这个,我在研究中一直找到这个问题,所以我想我也会发布我的解决方案 . 使用风险自负,错误检查或处理非常少 .

    <?php
    require 'vendor/autoload.php';
    
    use Aws\Glacier\GlacierClient;
    use Aws\Glacier\TreeHash;
    
    
    // Create the glacier client to connect with
    $glacier = new GlacierClient(array(
          'profile' => 'default',
          'region' => 'us-east-1',
          'version' => '2012-06-01'
          ));
    
    $fileName = '17mb_test_file';         // this is the file to upload
    $chunkSize = 1024 * 1024 * pow(2,2);  // 1 MB times a power of 2
    $fileSize = filesize($fileName);      // we will need the file size (in bytes)
    
    // initiate the multipart upload
    // it is dangerous to send the filename without escaping it first
    $result = $glacier->initiateMultipartUpload(array(
          'archiveDescription' => 'A multipart-upload for file: '.$fileName,
          'partSize' => $chunkSize,
          'vaultName' => 'MyVault'
          ));
    
    // we need the upload ID when uploading the parts
    $uploadId = $result['uploadId'];
    
    // we need to generate the SHA256 tree hash
    // open the file so we can get a hash from its contents
    $fp = fopen($fileName, 'r');
    // This class can generate the hash
    $th = new TreeHash();
    // feed in all of the data
    $th->update(fread($fp, $fileSize));
    // generate the hash (this comes out as binary data)...
    $hash = $th->complete();
    // but the API needs hex (thanks). PHP to the rescue!
    $hash = bin2hex($hash);
    
    // reset the file position indicator
    fseek($fp, 0);
    
    // the part counter
    $partNumber = 0;
    
    print("Uploading: '".$fileName
        ."' (".$fileSize." bytes) in "
        .(ceil($fileSize/$chunkSize))." parts...\n");
    while ($partNumber * $chunkSize < ($fileSize + 1))
    {
      // while we haven't written everything out yet
      // figure out the offset for the first and last byte of this chunk
      $firstByte = $partNumber * $chunkSize;
      // the last byte for this piece is either the last byte in this chunk, or
      // the end of the file, whichever is less
      // (watch for those Obi-Wan errors)
      $lastByte = min((($partNumber + 1) * $chunkSize) - 1, $fileSize - 1);
    
      // upload the next piece
      $result = $glacier->uploadMultipartPart(array(
            'body' => fread($fp, $chunkSize),  // read the next chunk
            'uploadId' => $uploadId,          // the multipart upload this is for
            'vaultName' => 'MyVault',
            'range' => 'bytes '.$firstByte.'-'.$lastByte.'/*' // weird string
            ));
    
      // this is where one would check the results for error.
      // This is left as an exercise for the reader ;)
    
      // onto the next piece
      $partNumber++;
      print("\tpart ".$partNumber." uploaded...\n");
    }
    print("...done\n");
    
    // and now we can close off this upload
    $result = $glacier->completeMultipartUpload(array(
      'archiveSize' => $fileSize,         // the total file size
      'uploadId' => $uploadId,            // the upload id
      'vaultName' => 'MyVault',
      'checksum' => $hash                 // here is where we need the tree hash
    ));
    
    // this is where one would check the results for error.
    // This is left as an exercise for the reader ;)
    
    
    // get the archive id.
    // You will need this to refer to this upload in the future.
    $archiveId = $result->get('archiveId');
    
    print("The archive Id is: ".$archiveId."\n");
    
    
    ?>
    

相关问题