C# - 从 Amazon S3 存储桶下载文件,并在文件不存在时捕获错误

C# - Downloading a file from Amazon S3 bucket and catch error when it doesn't exist

提问人:eek 提问时间:8/20/2023 更新时间:8/21/2023 访问量:228

问:

我生成了一个需要从 S3 下载的文件名列表。一切正常,但是当它遇到不存在的文件时会中断。当我尝试列出存储桶的内容时,我收到一个拒绝访问错误,即使根据供应商的说法,这应该可以正常工作......所以我放弃了列出内容的尝试。

相反,有没有办法在异步方法尝试下载不存在的东西时捕获异步方法的错误而不会使我的程序崩溃?如果一个文件失败,我想移动到列表中的下一个文件。

当我尝试一个不存在的文件时出现的错误:

Amazon.S3.AmazonS3Exception: Access Denied --->   
Amazon.Runtime.Internal.HttpErrorResponseException: The remote server returned an error: (403) 
Forbidden. ---> System.Net.WebException: The remote server returned an error: (403) Forbidden. at 
System.Net.HttpWebRequest.GetResponse() at Amazon.Runtime.Internal.HttpRequest.GetResponse() --- 
End of inner exception stack trace 

我的代码:

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Threading.Tasks;
using System.IO;
using System.IO.Compression;
using Amazon;
using Amazon.S3;
using Amazon.S3.Model;
using Amazon.S3.IO;
using System.Threading;

namespace AlgoSeekDecompressor
{
    class Program
    {
        static void Main(string[] args)
        {
            var list = new List<string>()
            {
                "Apple.csv",
                "Banana.csv",
                "Canteloupe.csv",
            };

            var downloadPath = @"E:\My_Folder";

            Task.Run(async () => { await DownloadFilesAsync(list, downloadPath); }).Wait();

            Console.WriteLine("Done downloading");
        }

        static async Task DownloadFilesAsync(List<string> list, string downloadPath)
        {
            var accessKey = "#################";
            var secretAccessKey = "#########################";

            var client = new AmazonS3Client(
                accessKey,
                secretAccessKey,
                RegionEndpoint.USEast1
            );

            foreach (var file in list)
            {
                var bucketName = "amazon_s3_bucket";

                //There is a prefix that is the first letter of the filename
                var filename = file.Substring(0, 1).ToUpper() + "/" + file + ".csv.gz";

                var request = new GetObjectRequest()
                {
                    BucketName = bucketName,
                    Key = filename,
                    RequestPayer = RequestPayer.Requester
                };

                if (!Directory.Exists(downloadPath))
                {
                    Directory.CreateDirectory(downloadPath);
                    Console.WriteLine("Directory {0} does not exist - creating directory", downloadPath);
                }

                var filePath = Path.Combine(downloadPath, filename);
                var appendToFile = false;

                using (var response = await client.GetObjectAsync(request))
                {
                    await response.WriteResponseStreamToFileAsync(filePath, appendToFile, CancellationToken.None);
                }
            }
        }
    }
}
C# Amazon-S3 错误处理 async-await

评论

2赞 Etienne de Martel 8/20/2023
403 错误是指您没有对文件的正确权限。丢失的文件会给你一个 404。
0赞 eek 8/20/2023
感谢您的见解。有没有办法抓住它并移动到下一个文件?
0赞 Etienne de Martel 8/20/2023
你在问如何抓住一个?因为这可能涉及某种声明。AmazonS3Exceptioncatch(AmazonS3Exception)

答:

1赞 sa-es-ir 8/20/2023 #1

我认为最好不要使用 因为没有理由使用它,您可以直接调用您的方法。Task.Run

关于捕获异常,这将起作用:

static async Task Main(string[] args) {
    ///....

    var messages = await DownloadFilesAsync(list, downloadPath);

    if (!messages.Any())
      Console.WriteLine("Done");
    else
      Console.WriteLine("Some of files failed");
  }

  static async Task<List<string>> DownloadFilesAsync(List < string > list, string downloadPath) {
    //....

    var errorMessages = new List<string>();
    foreach(var file in list) {
      //.....

      var filePath = Path.Combine(downloadPath, filename);
      var appendToFile = false;

      using GetObjectResponse response = await client.GetObjectAsync(request);

      try {

        await response.WriteResponseStreamToFileAsync(filePath, appendToFile, CancellationToken.None);

        if(response.HttpStatusCode != System.Net.HttpStatusCode.OK)
            errorMessages.Add($"Failed to download {objectName} with status {response.HttpStatusCode}");

      } 
     catch (AmazonS3Exception ex) {

        errorMessages.Add($"Error saving {objectName}: {ex.Message}");
      }

     catch (Exception ex) {

        errorMessages.Add($"Unhandled exception on saving {objectName}: {ex.Message}");
      }
    }

    return errorMessages;
  }

评论

1赞 Sir Rufo 8/20/2023
这并不能解决如果失败,我只想移动到列表中的下一个文件。
0赞 sa-es-ir 8/20/2023
@SirRufo 感谢您更新的观点和答案
0赞 eek 8/21/2023
感谢您的帮助,但这无法编译,因为它说我必须以异步方法运行它。当我尝试使用任务运行它时,它仍然会因错误文件而中断。
0赞 sa-es-ir 8/21/2023
@eek我忘了将方法设置为异步。答案已更新Main