Passing scrapy instance (not class) attribute to p

2019-06-13 23:55发布

I'm working with scrapy. I have a pipieline that starts with:

class DynamicSQLlitePipeline(object):

    @classmethod
    def from_crawler(cls, crawler):
        # Here, you get whatever value was passed through the "table" parameter
        table = getattr(crawler.spider, "table")
        return cls(table)

    def __init__(self,table):
        try:
            db_path = "sqlite:///"+settings.SETTINGS_PATH+"\\data.db"
            db = dataset.connect(db_path)
            table_name = table[0:3]  # FIRST 3 LETTERS
            self.my_table = db[table_name]

My spider starts with:

class For_Spider(Spider):

    name = "for"
    table = 'hello' # creating dummy attribute. will be overwritten

    def start_requests(self):

        self.table = self.dc # dc is passed in

When I start the spider with:

scrapy crawl for -a dc=mystring -a records=1

dc is passed in as a instance attribute (not a class attribute)

How do I pass this instance attribute to my pipeline so that I can dynamically set the table name in my pipeline? What I have now works for a class attribute.

标签: python scrapy
0条回答
登录 后发表回答