Using Tor proxy with scrapy

Question

I need help setting up Tor in Ubuntu and to use it within scrapy framework.

I did some research and found out this guide:

class RetryChangeProxyMiddleware(RetryMiddleware):

    def _retry(self, request, reason, spider):
        log.msg('Changing proxy')
        tn = telnetlib.Telnet('127.0.0.1', 9051)
        tn.read_until("Escape character is '^]'.", 2)
        tn.write('AUTHENTICATE "267765"\r\n')
        tn.read_until("250 OK", 2)
        tn.write("signal NEWNYM\r\n")
        tn.read_until("250 OK", 2)
        tn.write("quit\r\n")
        tn.close()
        time.sleep(3)
        log.msg('Proxy changed')
        return RetryMiddleware._retry(self, request, reason, spider)

then use it in settings.py:

DOWNLOADER_MIDDLEWARE = {
                         'spider.middlewares.RetryChangeProxyMiddleware': 600,
                         }

and then you just want to send requests through local tor proxy (polipo) which could be done with:

tsocks scrapy crawl spirder 

does anyone can confirm, that this method works and you get different IPs?

Answer 1

I was using this snippet: http://snipplr.com/view/66992/use-a-random-user-agent-for-each-request/

Update: broken link fixed