On OS X El Capitan I can not upgrade a python pack

Question

I am trying to use scrape, but I have a problem.

from six.moves import xmlrpc_client as xmlrpclib

ImportError: cannot import name xmlrpc_client

Then, I tried pip install --upgrade six scrape, but:

  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip/basecommand.py", line 211, in main
status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip/commands/install.py", line 311, in run
root=options.root_path,
  File "/Library/Python/2.7/site-packages/pip/req/req_set.py", line 640, in install
requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip/req/req_install.py", line 716, in uninstall
paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip/req/req_uninstall.py", line 125, in remove
renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip/utils/__init__.py", line 315, in renames
shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
os.chflags(dst, st.st_flags)

OSError: [Errno 1] Operation not permitted: '/var/folders/3h/r_2cxlvd1sjgzfgs4xckc__c0000gn/T/pip-5h86J8-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

Answer 1

Quick Fix:

I just got around what I think was the same problem. You might consider trying this (sudo, if necessary):

pip install scrape --upgrade --ignore-installed six

Github is ultimately where I got this answer (and there are a few more suggestions you may consider if this one doesn't solve your problem). It also seems as though this is an El Capitan problem.

Also, this technically might be a duplicate. But the answer the other post came up with was installing your own Python rather than relying on the default osx Python, which strikes me as more laborious.

---

Longer, Real Fix:

Eventually I ran into a situation where I actually needed to upgrade six in order to install some other libraries, at which point ignoring wasn't enough. The easiest way to upgrade six for me was to not use pip, but instead to manually download the .tar.gz from the six page, decompress, move to the newly decompressed six package directory, and run the installation manually (sudo, if necessary):

python setup.py install

Thanks to this answer for the guidance.

Answer 2

I don't think this is a duplicate, but actually this issue discussed here on the pip GitHub repository issues list.

NOTE: Depending on which package you're installing, you may need to modify the command referenced in the solution. In my case below, I was trying to setup virtualenv and virtualwrapper. If you're failing on installing aws-cli or any other package, just insert that into the script

The core of the problem is tied to Apple's new SIP that they shipped with El Capitan. More specifically,

OS X 10.11's python retains its own copy of six which is unremoveable, because of modifications Apple has done to their python distribution. 1.4.1 is not the latest, 1.10.0 is. It also comes early on their python's import path, so it will typically override later versions you install.

I would suggest using a different python for now. Python.org's, or installed via Homebrew, or Anaconda Python.

There is an incredibly detailed discussion on the Ask Different Stack Exchange that covers how the problems with SIP have been identified, addressed, and evolved since the original release of El Capitan. Although I found it fascinating, you'll spend less time following the instructions below than it would take you to read it, so I'd reccomend checking it out AFTER you finish the following...

I ran into the exact same error when attempting to upgrade VirtualEnv & VirtualEnvWrapper. There were several suggestions kicked around on that above thread, but in the end the most stable was to

1. Leverage the built-in support for the sudo OPTION to specify a HOME environment variable

$ man sudo

  -H   The -H (HOME) option option sets the HOME environment variable 
        to the home directory of the target user (root by default) as specified 
        HOME environment variable depends on sudoers(5) settings.  By default, 
        sudo will set HOME if env_reset or always_set_home are set, or if
        set_home is set and the -s option is specified on the command line.

2. Leverage pip's options to force an upgrade and ignore any pre-existing packages

$ pip install --help | grep upgrade

  -U, --upgrade   Upgrade all specified packages to the newest available 
        version. This process is recursive regardless of whether a dependency 
        is already satisfied.

beejhuff@ignatius:~/mac_setup$ pip install --help | grep ignore-installed

  -I, --ignore-installed  Ignore the installed packages (reinstalling instead).

First, my original attempt & error:

$ sudo pip install virtualenv virtualenvwrapper

The directory '/Users/beejhuff/Library/Caches/pip/http' or its parent directory 
is not owned by the current user and the cache has been disabled. 
Please check the permissions and owner of that directory. If executing 
pip with sudo, you may want sudo's -H flag.

The directory '/Users/beejhuff/Library/Caches/pip' or its parent directory 
is not owned by the current user and caching wheels has been disabled. 
check the permissions and owner of that directory. If executing pip with 
sudo, you may want sudo's -H flag.

Collecting virtualenv
  Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 335kB/s 
Collecting virtualenvwrapper
  Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl
Collecting virtualenv-clone (from virtualenvwrapper)
  Downloading virtualenv-clone-0.2.6.tar.gz
Collecting stevedore (from virtualenvwrapper)
  Downloading stevedore-1.12.0-py2.py3-none-any.whl
Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)
  Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 362kB/s 
Collecting six>=1.9.0 (from stevedore->virtualenvwrapper)
  Downloading six-1.10.0-py2.py3-none-any.whl
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
  Found existing installation: six 1.4.1
    DEPRECATION: Uninstalling a distutils installed project (six) has been deprecated and will be removed in a future version. This is due to the fact that uninstalling a distutils project will only partially uninstall the project.
    Uninstalling six-1.4.1:
Exception:
Traceback (most recent call last):
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/basecommand.py", line 209, in main
    status = self.run(options, args)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/commands/install.py", line 317, in run
    prefix=options.prefix_path,
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_set.py", line 726, in install
    requirement.uninstall(auto_confirm=True)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_install.py", line 746, in uninstall
    paths_to_remove.remove(auto_confirm)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/req/req_uninstall.py", line 115, in remove
    renames(path, new_path)
  File "/Library/Python/2.7/site-packages/pip-8.1.0-py2.7.egg/pip/utils/__init__.py", line 267, in renames
    shutil.move(old, new)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 302, in move
    copy2(src, real_dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 131, in copy2
    copystat(src, dst)
  File "/System/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/shutil.py", line 103, in copystat
    os.chflags(dst, st.st_flags)
OSError: [Errno 1] Operation not permitted: '/tmp/pip-GQL8Gi-uninstall/System/Library/Frameworks/Python.framework/Versions/2.7/Extras/lib/python/six-1.4.1-py2.7.egg-info'

The Solution

It required modifying my installation command in THREE specific ways:

1. I had to add the -H flag to sudo
2. I had to add the --upgrade option AFTER the name of the package I was upgrading (virtualenv)
3. I had to use the --ignore-installed flag and specify the six package was the one to be ignored.
4. *Note: general command is $ sudo -H pip install <packagename> --upgrade --ignore-installed six - replace<packagename> with the specific package you need to install`

Final Working Example

1st Upgrade virtualenv

$ sudo -H pip install virtualenv --upgrade --ignore-installed six

Password:

Collecting virtualenv
  Using cached virtualenv-15.0.0-py2.py3-none-any.whl
Collecting six
  Using cached six-1.10.0-py2.py3-none-any.whl
Installing collected packages: virtualenv, six
Successfully installed six-1.4.1 virtualenv-15.0.0

2nd Upgrade virtualenvwrapper

$ sudo -H pip install virtualenvwrapper --upgrade --ignore-installed six

Password:

  Downloading virtualenvwrapper-4.7.1-py2.py3-none-any.whl
Collecting six
  Downloading six-1.10.0-py2.py3-none-any.whl
Collecting virtualenv (from virtualenvwrapper)
  Downloading virtualenv-15.0.0-py2.py3-none-any.whl (1.8MB)
    100% |████████████████████████████████| 1.8MB 751kB/s 
Collecting virtualenv-clone (from virtualenvwrapper)
  Downloading virtualenv-clone-0.2.6.tar.gz
Collecting stevedore (from virtualenvwrapper)
  Downloading stevedore-1.12.0-py2.py3-none-any.whl
Collecting pbr>=1.6 (from stevedore->virtualenvwrapper)
  Downloading pbr-1.8.1-py2.py3-none-any.whl (89kB)
    100% |████████████████████████████████| 92kB 417kB/s 
Installing collected packages: virtualenv, virtualenv-clone, pbr, six, stevedore, virtualenvwrapper
  Running setup.py install for virtualenv-clone ... done
Successfully installed pbr-1.8.1 six-1.4.1 stevedore-1.12.0 virtualenv-15.0.0 virtualenv-clone-0.2.6 virtualenvwrapper-4.7.1

Answer 3

This would work!!

sudo pip install awsebcli --upgrade --ignore-installed six

Answer 4

Both of previous answers don't work for me. Finally, I got the solution from GitHub, aws/aws-cli, as excerpted below.

On OS X, if you see an error regarding the version of six that came with distutils in El Capitan, use the --ignore-installed option:

$ sudo pip install awscli --ignore-installed six