Im writing a Scrapy CrawlSpider that reads a list of ADs on first page, takes some info like thumbs of the listings and AD urls, then yields a request to each of this AD urls to take their details.
It was working and paginating apparently well on test enviroment, but today trying to make a complete run I realized that in log:
Crawled 3852 pages (at 228 pages/min), scraped 256 items (at 15 items/min)
I'm not understanding the reason of this big difference between Crawled pages and Scraped items. Anybody can help me to realize where that items are getting lost?
My spider code:
class MySpider(CrawlSpider):
name = "myspider"
allowed_domains = ["myspider.com", "myspider.co"]
start_urls = [
"http://www.myspider.com/offers/myCity/typeOfAd/?search=fast",
]
#Pagination
rules = (
Rule (
SgmlLinkExtractor()
, callback='parse_start_url', follow= True),
)
#1st page
def parse_start_url(self, response):
hxs = HtmlXPathSelector(response)
next_page = hxs.select("//a[@class='pagNext']/@href").extract()
offers = hxs.select("//div[@class='hlist']")
for offer in offers:
myItem = myItem()
myItem['url'] = offer.select('.//span[@class="location"]/a/@href').extract()[0]
myItem['thumb'] = oferta.select('.//div[@class="itemFoto"]/div/a/img/@src').extract()[0]
request = Request(myItem['url'], callback = self.second_page)
request.meta['myItem'] = myItem
yield request
if next_page:
yield Request(next_page[0], callback=self.parse_start_url)
def second_page(self,response):
myItem = response.meta['myItem']
loader = myItemLoader(item=myItem, response=response)
loader.add_xpath('address', '//span[@itemprop="streetAddress"]/text()')
return loader.load_item()