AWS S3 : Is there any way to count rows of uploade

问题:

We'd like to check the row count of uploaded files on AWS S3, if the uploading task is sure to be end correctly.

Now we're simply sending exported files by Windows AWS CLI command as follows, after exporting from SQL Server;

aws s3 cp !SEND_FILE! %S3_DIR%/ 

Kindly let us know if there is any way to check the count of uploaded files on S3 bucket. If there is no appropriate way to count them, to get any uploading error code as altanative way would be also appreciated.

Thanks for your advice in advance.

回答1:

Without re-downloading the file there isn't a way to count the lines on S3 directly. However, this is not needed. From the REST API documentation:

Amazon S3 never adds partial objects; if you receive a success response, Amazon S3 added the entire object to the bucket.

The primary way this is accomplished is by verifying the MD5 hash that is calculated on the client (i.e. where you're running the CLI) is the same as the object it stored. From the same docs:

To ensure that data is not corrupted traversing the network, use the Content-MD5 header. When you use this header, Amazon S3 checks the object against the provided MD5 value and, if they do not match, returns an error.

The CLI will automatically calculate the MD5 hash for you and send it to S3. So if the object (file) did not upload correctly you would get an error back in the CLI letting you know that.