Scrapy Project with Multiple Spiders - Custom Sett

Question

Scenario:

• Single scrapy project with multiple spiders.
• Custom settings defined per spider.

Issue:

• Upon execution (i.e. scrapy crawl ...) custom settings of the spider being executed is being overriden by custom settings of another spider in the same project.

spider01.py

class FirstXmlSpider(XMLFeedSpider):

# Spider Name
name = 'spider01'

# Initialise Settings
now = datetime.datetime.now()
settings.set('LOG_FILE', './logs/' + name + '_' + now.strftime("%Y%m%d_%H%M%S") + '.txt')

spider02.py

class SecondXmlSpider(XMLFeedSpider):

# Spider Name
name = 'spider02'

# Initialise Settings
now = datetime.datetime.now()
settings.set('LOG_FILE', './logs/' + name + '_' + now.strftime("%Y%m%d_%H%M%S") + '.txt')

Steps to Reproduce

1. Execute scrapy crawl spider01
2. Check logs directory (expecting to see log file prefixed with spider01_)
3. See log file with correct contents but wrong file name (spider02_).

Any ideas? I've setup scrapy projects in the past with multiple spiders without problem. Not sure why I am getting issues now?

Answer 1

Since Scrapy 1.0, you can add custom settings for spiders with no need to change project's global settings. Just add an attribute called custom_settings into your spiders.

Take a look at the docs: http://doc.scrapy.org/en/latest/topics/settings.html#settings-per-spider