Installing Python module with pip

Question

I'm trying to install a module called Scrapy. I installed it using

pip install Scrapy

I see the 'scrapy' folder in my /usr/local/lib/python2.7/site-packages, but when I try to import it in a Python program, is says there is no module by that name. Any ideas as to why this might be happening?

EDIT: Here is the output of the pip command:

    Downloading/unpacking Scrapy
  Downloading Scrapy-0.20.0.tar.gz (745kB): 745kB downloaded
  Running setup.py egg_info for package Scrapy

    no previously-included directories found matching 'docs/build'
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.2 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /usr/local/lib/python2.7/site-packages (from Scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.4.1 in /usr/local/lib/python2.7/site-packages (from w3lib>=1.2->Scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages/setuptools-1.1.6-py2.7.egg (from zope.interface>=3.6.0->Twisted>=10.0.0->Scrapy)
Installing collected packages: Scrapy
  Running setup.py install for Scrapy
    changing mode of build/scripts-2.7/scrapy from 644 to 755

    no previously-included directories found matching 'docs/build'
    changing mode of /usr/local/bin/scrapy to 755
Successfully installed Scrapy
Cleaning up...

When I run /usr/local/bin/scrapy I get the usage for the command and the available commands. I noticed that I have a python2.7 and python2.7-32 in my /usr/local/bin, and I remember installing the 32 bit version because of a problem with Mavericks.

Here is the output of python /usr/local/bin/scrapy:

Traceback (most recent call last): File "/usr/local/bin/scrapy", line 3, in <module> from scrapy.cmdline import execute ImportError: No module named scrapy.cmdline 

And head /usr/local/bin/scrapy:

#!/usr/local/opt/python/bin/python2.7 from scrapy.cmdline import execute execute()

Answer 1

EDIT: You can force pip to install to an alternate location. The details are here: Install a Python package into a different directory using pip?. If you do indeed have extra Python folders on your system, maybe you can try directing scrapy to those, even if just for a temporary solution.

Can you post the output of the pip command? Perhaps it is failing somewhere?

Also, is it possible you have two versions of Python on your machine? Pip only installs to one location, but perhaps the version of Python on your path is different.

Finally, sometimes package names given to pip are not exactly the same as the name used to import. Check the documentation of the package. I took a quick look and the import should be lowercase:

import scrapy

Answer 2

if you run on Ubuntu:

use the official Ubuntu Packages, which already solve all dependencies for you and are continuously updated with the latest bug fixes.

Optionally, even if it solves your problem, it is always better to install python libraries on a virtual environment, using virtualenvwrapper to keep the libraries separated, try to examine the apt-get installation log to find out what tools where added, then remove scrapy python library and reinstall it in the virtual env. using pip

Answer 3

Are you using Homebrew or MacPorts or something? As @J.F.Sebastian said, it sounds like you are having issues mixing the default python that comes with OS X, and one that is installed via a package manager... Try /usr/local/opt/python/bin/python2.7 -m scrapy and see if that throws an ImportError.

If that works, then you may want to consider making that python executable your default. Something like alias python2.7=/usr/local/opt/python/bin/python2.7 and then always use python2.7 instead of the default python. You can likewise just point python to the /urs/local... bit, but then you won't have easy access to the system (OS X-supplied) python if you ever needed it for some reason.

Answer 4

When all else fails you can always set the environment variable PYTHONPATH (see Permanently add a directory to PYTHONPATH for help) to the path where you installed Scrapy. (pending you're not using virtualenv -- and if you are please specify so we can help, it's generally a good idea to provide OS too)

Answer 5

It appears that the scrapy module that is installed on the Python path is an executable file that will bootstrap a Scrapy project directory for you.

The Python code in the scrapy executable looks like this:

   #!/usr/bin/env python

   from scrapy.cmdline import execute
   execute()

That's intended to be run from the command line rather than imported into your own Python project module.

According to the documentation for the project, running the scrapy executable with this syntax:

scrapy startproject <your-project-name>

will bootstrap a Scrapy project that has the following directory structure:

your-project-name/
scrapy.cfg
tutorial/
    __init__.py
    items.py
    pipelines.py
    settings.py
    spiders/
        __init__.py
        ...

There are a number of examples in the documentation that demonstrate how you create and run your own spiders, link extractors, etc., and how to manipulate the data that you retrieve with the application. They each demonstrate the appropriate Python imports from subdirectories in the scrapy package to get you up and running.

Hope that this helps.