Remarks/suggestions from Reddit:
Also a couple pointers on the S3 side:
-
I'm not 100% positive, but I think you can use the HeadObject method for GetFile, since it will get you the metadata you need without actually downloading the full file
-
Every place you use ListObjectsV2Async, you should check the response for a continuation or pagination token. The response you get will be limited to 1000 keys IIRC, so you'll need to loop through and make repeated requests until some stopping condition (either nextpagetoken is null, or istruncated is false or something like that)
-
also it looks like sometimes you use ListObjectsV2Async and other times you use ListObjectsAsync
-
this is more on the feature request side: some places will have strict security requirements and likely have certain settings enabled on their s3 buckets. you might want to expose a way for a caller to customize the Put or other requests before they are sent, so they can modify it, such as adding encryption/kms settings etc
-
lastly, just a word of caution - you'll see on r/aws and other places a lot that "s3 is not a file system". you can map some concepts like files and directories like you have here, and it can work ok. but don't go too far trying to replicate a full file system, or s3 will not work well. file systems can write bytes to arbitrary sections of a file easily enough, but in s3 you'd need to download and reupload the entire file (like you are already doing for Append). also trying to get a full directory listing and file size for each file does not map well to s3 operations etc. People who use tools like some s3 FUSE mount often have a hard time with this
-
on that note, someone trying to use this to for example Append to a log file in a tight loop might expect it to work quickly, since appending to a file on a local file system can handle that. but they'll see much slower performance writing to s3 like this since each write is going to be 2 network calls, and each write will be bigger and slower than the previous
Remarks/suggestions from Reddit:
Also a couple pointers on the S3 side:
I'm not 100% positive, but I think you can use the HeadObject method for GetFile, since it will get you the metadata you need without actually downloading the full file
Every place you use ListObjectsV2Async, you should check the response for a continuation or pagination token. The response you get will be limited to 1000 keys IIRC, so you'll need to loop through and make repeated requests until some stopping condition (either nextpagetoken is null, or istruncated is false or something like that)
also it looks like sometimes you use ListObjectsV2Async and other times you use ListObjectsAsync
this is more on the feature request side: some places will have strict security requirements and likely have certain settings enabled on their s3 buckets. you might want to expose a way for a caller to customize the Put or other requests before they are sent, so they can modify it, such as adding encryption/kms settings etc
lastly, just a word of caution - you'll see on r/aws and other places a lot that "s3 is not a file system". you can map some concepts like files and directories like you have here, and it can work ok. but don't go too far trying to replicate a full file system, or s3 will not work well. file systems can write bytes to arbitrary sections of a file easily enough, but in s3 you'd need to download and reupload the entire file (like you are already doing for Append). also trying to get a full directory listing and file size for each file does not map well to s3 operations etc. People who use tools like some s3 FUSE mount often have a hard time with this
on that note, someone trying to use this to for example Append to a log file in a tight loop might expect it to work quickly, since appending to a file on a local file system can handle that. but they'll see much slower performance writing to s3 like this since each write is going to be 2 network calls, and each write will be bigger and slower than the previous