Error while creating Scrapy project through startp

Question

I am learning the Scrapy framework. I just started, so this may seem like a novice doubt. However, here goes.

I use the startproject command to create a project. However, the project cannot be created. It gives me the following error.

Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 11, in <module>
sys.exit(execute())
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 122, in execute
cmds = _get_commands_dict(settings, inproject)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 46, in _get_commands_dict
cmds = _get_commands_from_module('scrapy.commands', inproject)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 29, in _get_commands_from_module
for cmd in _iter_command_classes(module):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/cmdline.py", line 20, in _iter_command_classes
for module in walk_modules(module_name):
  File "/usr/local/lib/python2.7/dist-packages/scrapy/utils/misc.py", line 68, in walk_modules
submod = import_module(fullpath)
  File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
__import__(name)
  File "/usr/local/lib/python2.7/dist-packages/scrapy/commands/bench.py", line 2, in         <module>
    from scrapy.tests.spiders import FollowAllSpider
ImportError: No module named tests.spiders

What can be the reason for the failure in creating the project? I searched google for this error but I cannot get anything.

How can this problem be solved?

Answer 1

I'm not sure of the exact cause yet, but this seems to be specific to Scrapy 0.24.3. If you've installed via pip, running

pip install scrapy==0.24.2

fixes the issue for me on Windows.

On Ubuntu (I installed from Scrapy's repositories as their instructions dicate) I had to pip uninstall scrapy and then run the above command.

I went to report the bug, but it looks like it's already there: https://github.com/scrapy/scrapy/issues/848