requests disappear after queueing in scrapy

问题:

Scrapy seems to complete without processing all the requests. I know this because i am logging before and after queueing the request and I can clearly see that.

I am logging in both parse and error callback methods and none of them got called for those missing requests.

How can I debug what happened to those requests?

回答1:

You need to add dont_filter=True when re-queueing the request. Though the request may not match other request but Scrapy remembers what requests it has already made and it will filter out if you re-queue it. It will assume it was by mistake.