Yield multiple items using scrapy

问题:

I'm scraping data from the following URL:
http://www.indexmundi.com/commodities/?commodity=gasoline

There are two sections which contain price: Gulf Coast Gasoline Futures End of Day Settlement Price and Gasoline Daily Price

I want to scrape data from both sections as two different items. Here is the code which I've written:

if dailyPrice:
        item['description'] = u''.join(dailyPrice.xpath(".//h1/text()").extract())
        item['price'] = u''.join(dailyPrice.xpath(".//span/text()").extract())
        item['unit'] =  dailyPrice.xpath(".//div/p/text()").extract()[0].split(',')[-1]
        regex = re.compile("Source:(.*)",re.IGNORECASE|re.UNICODE)
        result = re.search(regex, u''.join(dailyPrice.xpath(".//div/p/text()").extract()))
        if result:
            item['source'] = result.group(1).strip()

        yield item


if futurePrice:
        item['description'] = u''.join(futurePrice.xpath(".//h1/text()").extract())
        item['price'] = u''.join(futurePrice.xpath(".//span/text()").extract())
        item['unit'] =  u''.join(futurePrice.xpath(".//div[2]/table//tr[1]/td/text()").extract())
        source = futurePrice.xpath(".//div[2]/table//tr[4]/td/a/text()").extract()
        if source:
            item['source'] = u' - '.join(source)
        else:
            item['source'] = ''

        yield item

I want to know if this code will work fine or what should be correct way to do this?

回答1:

It should work just fine. You can yield as many items from a parse callback as you need. Just some notes:

1. In the second case it's better to create a new item then reusing the old one. Because you never know what has happened to the old item reference. Maybe you are overwriting and losing the previous data.

2. You can create different item types for your two cases. And in the pipeline treat them differently.