I want to be able to use scrapy to crawl links on a sitemap. I don't know much about this application, so I would be interested in any links/info/documentation you could provide.
Thanks
I want to be able to use scrapy to crawl links on a sitemap. I don't know much about this application, so I would be interested in any links/info/documentation you could provide.
Thanks
A new generic spider has just been added to Scrapy trunk, for this purpose. It will be available on next release (Scrapy 0.14)
All of the documentation is at http://doc.scrapy.org/. The tutorials can be found at scrapy.org also.
As for your question, see this SO question: how to parse a sitemap.xml file using scrapy's XmlFeedSpider?