从多个网站下载文件。(Downloading files from multiple website

这是我第一次Python项目所以它是很基本的，初步的。我经常要清理掉的朋友和我使用经常更新的免费程序病毒。不用手动下载每个程序，我试图创建一个简单的方法实现流程的自动化。因为我也在努力学习Python我认为这将是实践的好机会。

问题：

我必须找到一些链接的.exe文件。我能够找到正确的网址，但是当它尝试下载我得到一个错误。

有没有办法将所有链接添加到列表中，然后创建一个函数来浏览清单，并运行在每个URL的功能？我Google'd相当多的，我只是似乎无法使它发挥作用。也许我不是在正确的方向在想什么？

import urllib, urllib2, re, os
from BeautifulSoup import BeautifulSoup

# Website List
sas = 'http://cdn.superantispyware.com/SUPERAntiSpyware.exe'
tds = 'http://support.kaspersky.com/downloads/utils/tdsskiller.exe'
mbam = 'http://www.bleepingcomputer.com/download/malwarebytes-anti-malware/dl/7/?1'
tr = 'http://www.simplysup.com/tremover/download.html'
urllist = [sas, tr, tds, tr]
urrllist2 = []

# Find exe files to download

match = re.compile('\.exe')
data = urllib2.urlopen(urllist)
page = BeautifulSoup(data)

# Check links
#def findexe():
for link in page.findAll('a'):
    try:
        href = link['href']
        if re.search(match, href):
            urllist2.append(href)

    except KeyError:
        pass

os.chdir(r"C:\_VirusFixes")
urllib.urlretrieve(urllist2, os.path.basename(urllist2))

正如你所看到的，我已经离开注释掉了，因为我不能让它正常工作的功能。

我应该放弃列表，只需下载他们的个人？我想是有效的。

任何建议或如果你能在正确的方向指向我，那将是非常赞赏。

Answer 1:

除了mikez302的答案，这里是一个稍微更可读的方式编写代码：

import os
import re
import urllib
import urllib2

from BeautifulSoup import BeautifulSoup

websites = [
    'http://cdn.superantispyware.com/SUPERAntiSpyware.exe'
    'http://support.kaspersky.com/downloads/utils/tdsskiller.exe'
    'http://www.bleepingcomputer.com/download/malwarebytes-anti-malware/dl/7/?1'
    'http://www.simplysup.com/tremover/download.html'
]

download_links = []

for url in websites:
    connection = urllib2.urlopen(url)
    soup = BeautifulSoup(connection)
    connection.close()

    for link in soup.findAll('a', {href: re.compile(r'\.exe$')}):
        download_links.append(link['href'])

for url in download_links:
    urllib.urlretrieve(url, r'C:\_VirusFixes', os.path.basename(url))

Answer 2:

urllib2.urlopen是用于访问单个URL的功能。如果您要访问多个的，你应该在列表循环。你应该做这样的事情：

for url in urllist:
    data = urllib2.urlopen(url)
    page = BeautifulSoup(data)

    # Check links
    for link in page.findAll('a'):
        try:
            href = link['href']
            if re.search(match, href):
                urllist2.append(href)

        except KeyError:
            pass

    os.chdir(r"C:\_VirusFixes")
    urllib.urlretrieve(urllist2, os.path.basename(urllist2))

Answer 3:

上面的代码没有工作对我来说，在我的情况，这是因为该页面通过脚本组装的链接，而不是包括它的代码。当我遇到了这个问题我用下面的代码这仅仅是一个刮板：

import os
import re
import urllib
import urllib2

from bs4 import BeautifulSoup

url = ''

connection = urllib2.urlopen(url)
soup = BeautifulSoup(connection) #Everything the same up to here 
regex = '(.+?).zip'       #Here we insert the pattern we are looking for
pattern = re.compile(regex)
link = re.findall(pattern,str(soup)) #This finds all the .zip (.exe) in the text
x=0
for i in link:
    link[x]=i.split(' ')[len(i.split(' '))-1] 
# When it finds all the .zip, it usually comes back with a lot of undesirable 
# text, luckily the file name is almost always separated by a space from the 
# rest of the text which is why we do the split
    x+=1  

os.chdir("F:\Documents")
# This is the filepath where I want to save everything I download

for i in link:
    urllib.urlretrieve(url,filename=i+".zip") # Remember that the text we found doesn't include the .zip (or .exe in your case) so we want to reestablish that.

这是不是像以前的答案代码一样高效，但它会为大多数几乎所有的现场工作。

文章来源: Downloading files from multiple websites.