I'm working with scrapy. I have a pipieline that starts with:
class DynamicSQLlitePipeline(object):
@classmethod
def from_crawler(cls, crawler):
# Here, you get whatever value was passed through the "table" parameter
table = getattr(crawler.spider, "table")
return cls(table)
def __init__(self,table):
try:
db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db"
db = dataset.connect(db_path)
table_name = table[0:3] # FIRST 3 LETTERS
self.my_table = db[table_name]
My spider starts with:
class For_Spider(Spider):
name = "for"
table = 'hello' # creating dummy attribute. will be overwritten
def start_requests(self):
self.table = self.dc # dc is passed in
When I start the spider with:
scrapy crawl for -a dc=mystring -a records=1
dc is passed in as a instance attribute (not a class attribute)
How do I pass this instance attribute to my pipeline so that I can dynamically set the table name in my pipeline? What I have now works for a class attribute.