urllib2 proxy does not work with tor

2019-08-03 23:29发布

问题:

I want to write some script with python which uses tor/proxy addresses to access web, for the test I have the following script:

import urllib2
from BeautifulSoup import BeautifulSoup

protocol = 'socks4'
ip = '127.0.0.1:9050'

proxy = urllib2.ProxyHandler({protocol:ip})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

page = urllib2.urlopen("http://www.ifconfig.me/ip").read()

print(page)

The problem is it shows my own IP address, while when run directly from terminal:

proxychains curl ifconfig.me/ip

shows different IP, how can I fix it?

when http used instead of socks 4 it gives the following error:

Traceback (most recent call last):
  File "proxy_test.py", line 11, in <module>
    page = urllib2.urlopen("http://www.ifconfig.me/ip").read()
  File "/usr/lib/python2.7/urllib2.py", line 126, in urlopen
    return _opener.open(url, data, timeout)
  File "/usr/lib/python2.7/urllib2.py", line 400, in open
    response = meth(req, response)
  File "/usr/lib/python2.7/urllib2.py", line 513, in http_response
    'http', request, response, code, msg, hdrs)
  File "/usr/lib/python2.7/urllib2.py", line 438, in error
    return self._call_chain(*args)
  File "/usr/lib/python2.7/urllib2.py", line 372, in _call_chain
    result = func(*args)
  File "/usr/lib/python2.7/urllib2.py", line 521, in http_error_default
    raise HTTPError(req.get_full_url(), code, msg, hdrs, fp)
urllib2.HTTPError: HTTP Error 501: Tor is not an HTTP Proxy

回答1:

I use http (not sock) and it works

import urllib2
from BeautifulSoup import BeautifulSoup

protocol = 'http'
ip = '127.0.0.1:8118'

proxy = urllib2.ProxyHandler({protocol:ip})
opener = urllib2.build_opener(proxy)
urllib2.install_opener(opener)

page = urllib2.urlopen("http://www.ifconfig.me/ip").read()

print(page)