crawl a list of sites one by one with scrapy

Question

I am trying to crawl a list of sites with scrapy. I tried to put the list of website urls as the start_urls, but then I found I couldn't afford so much memory with it. Is there any way to set the scrapy crawling one or two sites at a time?

Answer 1

You can try using concurrent_requests = 1 so that you don't overloaded with data

http://doc.scrapy.org/en/latest/topics/settings.html#concurrent-requests

Answer 2

You can define a start_requests method which iterates through requests to your URLs. This should avoid the overhead of holding all your start urls in memory at once and is the simplest approach to solve the problem you described.

If this is still a lot of URLs for scrapy to hold in memory during the crawl, you can enable persistence support.

If you really want to only feed scrapy a few URLs at a time, this is possible by registering for the spider_idle signal and in you callback function add the next few URLs and raise DontCloseSpider.