Scrapy spider not found error

Question

This is Windows 7 with python 2.7

I have a scrapy project in a directory called caps (this is where scrapy.cfg is)

My spider is located in caps\caps\spiders\campSpider.py

I cd into the scrapy project and try to run

scrapy crawl campSpider -o items.json -t json

I get an error that the spider can't be found. The class name is campSpider

...
    spider = self.crawler.spiders.create(spname, **opts.spargs)
  File "c:\Python27\lib\site-packages\scrapy-0.14.0.2841-py2.7-win32.egg\scrapy\spidermanager.py", l
ine 43, in create
    raise KeyError("Spider not found: %s" % spider_name)
KeyError: 'Spider not found: campSpider'

Am I missing some configuration item?

Answer 1

Ensure same name attribute is used in the command line for running spider ...

scrapy crawl

Answer 2

Ahh Yes, you should enter the value of your 'name variable value'.

I.e.

import scrapy

class QuoteSpider(scrapy.Spider):
    name = 'quotes'
    start_urls = [
        'http://quotes.toscrape.com/'
    ]

    def parse(self, response):
        title = response.css('title').extract()
        yield {'titleText' : title}

So in this case, the name = 'quotes'. Then in your command line you enter: 'scrapy crawl quotes'

That was my problem.

Answer 3

make sure that your spider file is saved in your spider directory. the Crawler looks for the spider name in the spider directory

Answer 4

Sometime this strange behaviour is caused by LOG_STDOUT = True

It defaults to False though, so check it and if it is set to True - try to set it to default

LOG_STDOUT = False

This is a logged issue

Answer 5

Try running scrapy list on the command line. If there is any error on the spider it will detect it.

In my case, I was bluntly copy code from another project and forget to change the project name from the spider module import

Answer 6

I fixed by fixing my filename.

Originally, my.spider.py. Fixed, myspider.py.

I'm very new to python and scrapy so I'm not sure if this is a dumb mistake on my part.