Automatic download file from web page

Question

I am looking for a method to download automatically a file from a website.

Currently the process is really manual and heavy. I go on a webpage, I enter my pass and login. It opens a pop up, where I have to click a download button to save a .zip file.

Do you have any advice on how I could automate this task ?

I am on windows 7, and I can use mainly MS dos batch, or python. But I am open to other ideas.

Answer 1

You'll want to take a look at requests (to fetch the html and the file), Beautifulsoup (to parse the html and find the links)

requests has built in auth: http://docs.python-requests.org/en/latest/ Beautifulsoup is quite easy to use: http://www.crummy.com/software/BeautifulSoup/bs4/doc/

Pseudocode: use request to download the sites html and auth. Go through the links by parsing. If a link meets the criteria -> save in a list, else continue. When all the links have been scrapped, go through them and download the file using requests (req = requests.get('url_to_file_here', auth={'username','password'}), if req.status_code in [200], file = req.text

If you can post the link of the site you want to download from, maybe we can do more.

Answer 2

You can use selenium web driver to automate the downloading. You can use below snippet for browser download preferences in java.

FirefoxProfile profile = new FirefoxProfile();
profile.setPreference("browser.download.folderList", 2);
profile.setPreference("browser.download.manager.showWhenStarting", false);
profile.setPreference("browser.download.dir", "C:\\downloads");
profile.setPreference("browser.helperApps.neverAsk.openFile","text/csv,application/x-msexcel,application/excel,application/x-excel,application/vnd.ms-excel,text/html,text/plain,application/msword,application/xml");

To handle the popup using this class when popup comes.

Robot robot = new Robot();
robot.keyPress(KeyEvent.VK_DOWN); 
robot.keyRelease(KeyEvent.VK_DOWN);
robot.keyPress(KeyEvent.VK_ENTER); 
robot.keyRelease(KeyEvent.VK_ENTER);