from scrapy.selector import selector error

Question

I am unable to do the following:

from scrapy.selector import Selector

The error is:

File "/Desktop/KSL/KSL/spiders/spider.py", line 1, in from scrapy.selector import Selector ImportError: cannot import name Selector

It is as if LXML is not installed on my machine, but it is. Also, I thought this was a default module built into scrapy. Maybe not?

Thoughts?

Answer 1

Try importing HtmlXPathSelector instead.

    from scrapy.selector import HtmlXPathSelector

And then use the .select() method to parse out your html. For example,

    sel = HtmlXPathSelector(response)
    site_names = sel.select('//ul/li')

If you are following the tutorial on the Scrapy site (http://doc.scrapy.org/en/latest/intro/tutorial.html), the updated example would look like this:

    from scrapy.spider import BaseSpider
    from scrapy.selector import HtmlXPathSelector

    class DmozSpider(BaseSpider):
        name = "dmoz"
        allowed_domains = ["dmoz.org"]
        start_urls = [
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Books/",
            "http://www.dmoz.org/Computers/Programming/Languages/Python/Resources/"
        ]

        def parse(self, response):
            sel = HtmlXPathSelector(response)
            sites = sel.select('//ul/li')

            for site in sites:
                title = site.select('a/text()').extract()
                link = site.select('a/@href').extract()
                desc = site.select('text()').extract()
                print title, link, desc

Hope this helps!

Answer 2

I encounter the same problem. I think there is something wrong with your scrapy version.

You could type scrapy version -v into cmd to check the version. As far as I know, the newest version is 0.24.4 (2014.10.23). You could visit http://scrapy.org/ to find the newest.

Answer 3

Had the same issue which was due to the way I installed scrapy (my OS is Ubuntu). I did it through

sudo apt-get install python-scrapy

Install it from python instead. If you use Anaconda just do

conda install -c scrapinghub scrapy

If not, then

pip install Scrapy

http://doc.scrapy.org/en/latest/intro/install.html