Download all pdf files using wget

问题:

I have the following site http://www.asd.com.tr. I want to download all PDF files into one directory. I've tried a couple of commands but am not having much luck.

$ wget --random-wait -r -l inf -nd -A pdf http://www.asd.com.tr/

With this code only four PDF files were downloaded. Check this link, there are over several thousand PDFs available:

• http://www.asd.com.tr/Default.aspx

For instance, hundreds of files are in the following folder:

• http://www.asd.com.tr/Folders/asd/…

But I can't figure out how to access them correctly to see and download them all, there are some of folders in this subdirectory, http://www.asd.com.tr/Folders/, and thousands of PDFs in these folders.

I've tried to mirror site using -m command but it failed too.

Any more suggestions?

回答1:

First, verify that the TOS of the web site permit to crawl it. Then, one solution is :

mech-dump --links 'http://domain.com' |
    grep pdf$ |
    sed 's/\s+/%20/g' |
    xargs -I% wget http://domain.com/%

The mech-dump command comes with Perl's module WWW::Mechanize (libwww-mechanize-perl package on debian & debian likes distros)