How to skip already existing files when downloadin

Question

I want to curl to download a link, but I want it to skip files that already exist. Right now, the line of code I have will continue to overwrite it no mater what:

curl '$url' -o /home/$outputfile &>/dev/null &

How this can be achieved?

Answer 1

You could just put your call to curl inside an if block:

if ! [ -f /home/$outputfile ]; then
  curl -o /home/$outputfile "url"
fi

Also note that in your example, you've got $url inside single quotes, which won't do what you want. Compare:

echo '$HOME'

To:

echo "$HOME"

Also, curl has a --silent option that can be useful in scripts.

Answer 2

The curl may support skipping the files when you use it with -O and -J, but its behaviour is inconsistent.

The -J (--remote-header-name) basically tells the -O (--remote-name) option to use the server-specified Content-Disposition filename instead of extracting a filename from the URL. In that way the curl doesn't really know what file name the server will return, so it may ignore the existing file for a safety precaution.

^{Source: Re: -J "Refusing to overwrite..."}

For example:

$ curl -LJO -H 'Accept: application/octet-stream' 'https://api.github.com/repos/x/y/releases/assets/12345
Warning: Refusing to overwrite my_file.bin: File 
Warning: exists
curl: (23) Failed writing body (0 != 16384)

However as mentioned already, its behaviour is unpredictable and it doesn't work for all the files.

Answer 3

You can use curl option -C -. This option is used to resume a broken download, but will skip the download if the file is already complete. Note that the argument to -C is a single dash. A disadvantage might be that curl still briefly contacts the remote server to ask for the file size.

Answer 4

Use wget with --no-clobber instead:

-nc, --no-clobber: skip downloads that would download to existing files.

Example:

wget -nc -q -O "/home/$outputfile" "$url"