Checking if a file is in a S3 bucket using the s3c

Question

I having a program that successfully uploads all of the files that I need. I have new files everyday that I need to upload. After I have uploaded the files I no longer need them and thus am not looking to sync them.

I am curious if there is a way to check if given a path and file name if that exists within S3 using the s3cmd.

Answer 1

We can use s3cmd ls , Take one flag flag_exists true if file is there and false if file is not there.

FLAG_EXISTS=false
    for j in $(s3cmd ls s3://abc//abc.txt); do
      if [[ "$j" == "s3://abc//abc.txt" ]]; then
        FLAG_EXISTS=true
        break
      fi
    done
    if [ "$FLAG_EXISTS" = false ]; then
      echo 'file not exists'
    else
      echo 'file exists'
    fi

Explanation - Since ls can return many values like if u search for s3cmd ls abc.txt , then it can return values like abc.txt abcd.txt and so on , so looping and checking using if condition if file exists.

Answer 2

You can use the ls command in s3cmd to know if a file is present or not in S3.

Bash code

path=$1
count=`s3cmd ls $path | wc -l`

if [[ $count -gt 0 ]]; then
        echo "exist"
else
        echo "do not exist"
fi

Usage: ./s3_exist.sh s3://foo/bar.txt

Edit:

As cocoatomo pointed out in comments, s3cmd ls $path lists all file that begins with $path. A safer approach would be to use s3cmd info $path and check the exit code.

New Bash code

path=$1
s3cmd info $path >/dev/null 2>&1

if [[ $? -eq 0 ]]; then
    echo "exist"
else
    echo "do not exist"
fi

Answer 3

In the newer version of AWS CLI, you can use the following code to detect the existence of a file or directory

count=$(aws s3 ls $path | wc -l)
if [ $count -gt 0 ]
then
  (>&2 echo "$path already exists!")
  return
fi

Answer 4

Assuming that bar.txt and bar.txt.bak exist in a bucket s3://foo, "s3cmd ls s3://foo/bar.txt" shows a following output.

$ s3cmd ls s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt
2013-11-11 11:11    5   s3://foo/bar.txt.bak

Since we should remove 2nd line from the command result, we use "awk" command to filter unnecessary lines.

$ filename=s3://foo/bar.txt
$ s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }"
2013-11-11 11:11    5   s3://foo/bar.txt

Finally, we build up all commands.

filename=s3://foo/bar.txt
count=$(s3cmd ls ${filename} | awk "\$4 == \"${filename}\" { print \$4 }" | wc -l)

if [ $count -eq 0 ]; then
  echo "file does not exist"
else
  echo "file exists"
fi