How to avoid redirection of the webcrawler to the

Question

I subclassed a CrawlSpider and want to extract data from website. However, I always get redirected to the site's mobile version. I tried to change the USER_AGENT variable in scrapy's settings to Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.8.0.1) Gecko/20060111 Firefox/1.5.0.1, but still get redirected.

Is there another way to signal another client and avoid redirection?

Answer 1

There are two types of redirection supported in Scrapy:

• RedirectMiddleware - Handle redirection of requests based on response status
• MetaRefreshMiddleware - Handle redirection of requests based on meta-refresh html tag

So, maybe your html page uses second type of redirection?

See also:

• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#redirectmiddleware-settings
• http://doc.scrapy.org/en/latest/topics/downloader-middleware.html#metarefreshmiddleware-settings