Scrapy throws ImportError: cannot import name xmlr

Question

After install Scrapy via pip, and having Python 2.7.10:

scrapy
Traceback (most recent call last):
File "/usr/local/bin/scrapy", line 7, in <module>
from scrapy.cmdline import execute
File "/Library/Python/2.7/site-packages/scrapy/__init__.py", line 48,  
in <module>
from scrapy.spiders import Spider
File "/Library/Python/2.7/site-packages/scrapy/spiders/__init__.py",    
line 10, in <module>
from scrapy.http import Request
File "/Library/Python/2.7/site-packages/scrapy/http/__init__.py", line   
12, in <module>
from scrapy.http.request.rpc import XmlRpcRequest
File "/Library/Python/2.7/site-packages/scrapy/http/request/rpc.py",  
line 7, in <module>
from six.moves import xmlrpc_client as xmlrpclib
ImportError: cannot import name xmlrpc_client

But I can import module:

Python 2.7.10 (default, Jun 10 2015, 19:42:47) 
[GCC 4.2.1 Compatible Apple LLVM 6.1.0 (clang-602.0.53)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
>>> import scrapy
>>> 

What's going on?

Answer 1

I just had the same issue. Try this:

sudo pip uninstall scrapy

sudo pip install scrapy==0.24.2

Then give it a shot

Answer 2

This is a known issue on Mac OSX for Scrapy. You can refer to this link.

Basically the issue is with the PYTHONPATH in your system. To solve the issue change the current PYTHONPATH to point to the newer or none Mac OSX version of Python. Before running Scrapy, try:

export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH

If that worked you can change the .bashrc file permanently:

echo "export PYTHONPATH=/Library/Python/2.7/site-packages:$PYTHONPATH" >> ~/.bashrc

If none of this works, take a look at the link above.

Answer 3

I've just fixed this issue on my OS X.

Please backup your files first.

sudo rm -rf /Library/Python/2.7/site-packages/six*
sudo rm -rf /System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six*
sudo pip install six

Scrapy 1.0.0 is ready to go.

---

If you encounter an error rm: /System/Library/... Operation not permitted

Please try to disable System Integrity Protection
See Operation Not Permitted when on root El capitan (rootless disabled)

Answer 4

Better than downgrading Scrapy it's to upgrade your version of six:

pip install --upgrade six scrapy

This will allow you to use all the goodies from Scrapy 1.0 ;)

Answer 5

I had the same exact problem when upgrading to Scrapy 1.0. After numerous work arounds the solution that worked for me was uninstalling six with pip:

sudo pip uninstall six

then re-installing six via easy_install

easy_install six

Hope that works!

Answer 6

I believe the best solution on OS X should be "Don’t use system python". It will make life easier. This link shows how to do this.

There’s a known issue that prevents pip from updating system packages. This has to be addressed to successfully install Scrapy and its dependencies. Here are some proposed solutions:

(Recommended) Don’t use system python, install a new, updated version that doesn’t conflict with the rest of your system. Here’s how to do it using the homebrew package manager:

1. Install homebrew following the instructions in http://brew.sh/
2. Update your PATH variable to state that homebrew packages should be used before system packages (Change .bashrc to .zshrc accordantly if you’re using zsh as default shell):

echo "export PATH=/usr/local/bin:/usr/local/sbin:$PATH" >> ~/.bashrc

3. Reload .bashrc to ensure the changes have taken place:

source ~/.bashrc

4. Install python:

brew install python

5. Latest versions of python have pip bundled with them so you won’t need to install it separately. If this is not the case, upgrade python:

brew update; brew upgrade python