multiple wget -r a site simultaneously?

Question

any command / wget with options?

For multithreaded download a site recursively and simultaneously?

Answer 1

Use axel to download with multi connections

apt-get install axel

axel http://example.com/file.zip

Answer 2

I found a decent solution.

Read original at http://www.linuxquestions.org/questions/linux-networking-3/wget-multi-threaded-downloading-457375/

wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &
wget -r -np -N [url] &

copied as many times as you deem fitting to have as much processes downloading. This isn't as elegant as a properly multithreaded app, but it will get the job done with only a slight amount of over head. the key here being the "-N" switch. This means transfer the file only if it is newer than what's on the disk. This will (mostly) prevent each process from downloading the same file a different process already downloaded, but skip the file and download what some other process hasn't downloaded. It uses the time stamp as a means of doing this, hence the slight overhead.

It works great for me and saves a lot of time. Don't have too many processes as this may saturate the web site's connection and tick off the owner. Keep it around a max of 4 or so. However, the number is only limited by CPU and network bandwidth on both ends.

Answer 3

With the use of parallel wget utilizing the xargs switch, this solution seems so much better:

https://stackoverflow.com/a/11850469/1647809

Answer 4

Well, you can always run multiple instances of wget, no?

Example:

wget -r http://somesite.example.org/ &
wget -r http://othersite.example.net/ &

etc. This syntax will work in any Unix-like environment (e.g. Linux or MacOS); not sure how to do this in Windows.

Wget itself does not support multithreaded operations - at least, neither the manpage nor its website has any mention of this. Anyway, since wget supports HTTP keepalive, the bottleneck is usually the bandwidth of the connection, not the number of simultaneous downloads.