Mac OS X El Capitan - Scrapy/Python ImportError: c

Question

I am trying to use Scrapy on Mac OS X El Capitan. I have zsh installed and I have tried everything that I could find online to fix this issue. I have also looked at Scrapy throws ImportError: cannot import name xmlrpc_client and could not resolve my problem!

Python installed via brew and added "pip install scrapy":

➜  DriverEBV  which python
/usr/local/bin/python

My .zshrc has the following line:

export PATH=/usr/local/bin:$PATH
export PYTHONPATH="/Library/Python/2.7/site-packages"

This is the error I get:

➜  DriverEBV  scrapy runspider DriverEBV.py
Traceback (most recent call last):
  File "/usr/local/bin/scrapy", line 7, in <module>
    from scrapy.cmdline import execute
  File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48, in <module>
    from scrapy.spiders import Spider
  File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py", line 10, in <module>
    from scrapy.http import Request
  File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line 12, in <module>
    from scrapy.http.request.rpc import XmlRpcRequest
  File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py", line 7, in <module>
    from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

When I run "pip install scrapy" now this is what I see:

➜  DriverEBV  pip install scrapy
Requirement already satisfied (use --upgrade to upgrade): scrapy in /Library/Python/2.7/site-packages
Requirement already satisfied (use --upgrade to upgrade): cssselect>=0.9 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): queuelib in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyOpenSSL in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): w3lib>=1.8.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): lxml in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): Twisted>=10.0.0 in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): six>=1.5.2 in /usr/local/lib/python2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): service-identity in /Library/Python/2.7/site-packages (from scrapy)
Requirement already satisfied (use --upgrade to upgrade): cryptography>=0.7 in /usr/local/lib/python2.7/site-packages (from pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): zope.interface>=3.6.0 in /usr/local/lib/python2.7/site-packages (from Twisted>=10.0.0->scrapy)
Requirement already satisfied (use --upgrade to upgrade): characteristic>=14.0.0 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1-modules in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pyasn1 in /Library/Python/2.7/site-packages (from service-identity->scrapy)
Requirement already satisfied (use --upgrade to upgrade): setuptools in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): enum34 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): ipaddress in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): idna>=2.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): cffi>=1.1.0 in /usr/local/lib/python2.7/site-packages (from cryptography>=0.7->pyOpenSSL->scrapy)
Requirement already satisfied (use --upgrade to upgrade): pycparser in /usr/local/lib/python2.7/site-packages (from cffi>=1.1.0->cryptography>=0.7->pyOpenSSL->scrape)

Anybody able to help me?

Answer 1

Try uninstalling via pip and then reinstall using easy_install command. I had the same trouble with another python module and doing it this way fixed the issue for me on Mac OS X El Capitan.

Answer 2

What helped me was to uninstall six and scrapy and then install again:

pip uninstall six
pip uninstall scrapy

pip install six
pip install scrapy

Run with sudo if necessary.

---

Or, you can also try upgrading six and scrapy:

pip install --upgrade scrapy
pip install --upgrade six

Answer 3

I've had nothing but pain mucking with the Mac OS X system Python libraries installed in the /Library/Python directory. What has worked well for me is a combination of MacPorts and virtualenv:

1. Install MacPorts

2. Install Python, pip, and virtualenv from MacPorts:

   /opt/local/bin/port install python27
   /opt/local/bin/port install py27-pip
   /opt/local/bin/port install py27-virtualenv
   

3. Setup virtualenv:

   /opt/local/bin/virtualenv-2.7 myenv
   

4. Activate virtualenv (don't forget the dot!)

   . myenv/bin/activate
   

5. Install scrapy

   pip install scrapy
   

This way, the system Python libraries are untouched and you can install whatever packages you like without having to remove or upgrade existing packages.

Answer 4

I believe the best solution on OS X should be "Don’t use system python". It will make life easier. This link shows how to do this.

There’s a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

(Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:

1. Install homebrew following the instructions in http://brew.sh/
2. Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell):

echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

3. Reload .bashrc to ensure the changes have taken place:

source ~/.bashrc

4. Install python:

brew install python

5. Latest versions of python have pip bundled with them so you won’t need to install it separately. If this is not the case, upgrade python:

brew update; brew upgrade python