Bash Script that downloads file from FTP only if i

Question

I currently have a bash script that downloads files from an FTP Site. I would like to modify the script to keep a manifest of every file it downloads and only download any new files that are not in the manifest. Is there a way to do this in bash or am I pushing the limitations of what bash can actually do?

Answer 1

My suggestion would be to go grab the list of files that you might want to download, compare that to your manifest, then download the ones not in the manifest. Something like this should work:

ftp username@hostname <<EOF
ls * filelist.txt
EOF
cat filelist.txt | awk '{print $9}' | tail -n +3 > filenames.txt
while IFS= read -r file
  do
    if [[ ! $(grep "$file" manifest.txt) ]]; then
       echo "$file is not in manifest"
    fi
  done < "filenames.txt"

Of course, you'll need to change filelist.txt and filenames.txt to include an appropriate path. Also, change the echo command to add the filename to another file or variable that you'll use to do the actual download.

Here's an explanation of what's going on:

The first three lines use ftp to grab a list of files (You'll want to change the * to point to the correct directory.)

The cat line parses the output of the ftp ls command to grab just the filename part and put it into another file.

Then the while loop goes through the list of files and uses grep to see if the filename exists in the manifest file. If it doesn't, you can then download it and add it to the manifest.

Let me know if you have any questions.

Hope this helps!