Using gzip to compress files to transfer with aws

Question

$ gzip file.txt | aws s3 cp file.txt.gz s3://my_bucket/

I am trying to gzip file.txt to file.txt.gz and the passing it to aws program which has s3 as a command and cp as a subcommand.

Generates : warning: Skipping file file.txt.gz. File does not exist.

I'm newbie in linux. Can anyone help on this please?

Answer 1

Replace the | with &&. The | means pipe, which runs the aws command immediately without waiting for gzip to finish or even start. Also the | will do nothing here, since its purpose is to send the stdout output of gzip to the stdin input of aws. There is no stdout output from gzip in that form.

If you really want gzip to send its output to stdout and not write the file file.txt.gz, then you need to use gzip -c file.txt. Then you need a way for aws to take in that data. The typical way this is specified in Unix utilities is to replace the file name with -. However I don't know if gzip -c file.txt | aws s3 cp - s3://my_bucket/ will work.

Answer 2

$ gzip -c file.txt | aws s3 cp - s3://my_bucket/file.txt.gz

Unless you desire to have a .gz locally of file.txt, this allows you to accomplish the gzip and transfer in one step, leaving file.txt in tact.

Newer versions of the AWS CLI now allow you to steam, UNIX style, via '-' character.