How do I install from a local cache with pip?

Question

I install a lot of the same packages in different virtualenv environments. Is there a way that I can download a package once and then have pip install from a local cache?

This would reduce download bandwidth and time.

Answer 1

A simpler option is basket.

Given a package name, it will download it and all dependencies to a central location; without any of the drawbacks of pip cache. This is great for offline use.

You can then use this directory as a source for pip:

pip install --no-index -f file:///path/to/basket package

Or easy_install:

easy_install -f ~/path/to/basket -H None package

You can also use it to update the basket whenever you are online.

Answer 2

PIP_DOWNLOAD_CACHE has some serious problems. Most importantly, it encodes the hostname of the download into the cache, so using mirrors becomes impossible.

The better way to manage a cache of pip downloads is to separate the "download the package" step from the "install the package" step. The downloaded files are commonly referred to as "sdist files" (source distributions) and I'm going to store them in a directory $SDIST_CACHE.

The two steps end up being:

pip install --no-install --use-mirrors -I --download=$SDIST_CACHE <package name>

Which will download the package and place it in the directory pointed to by $SDIST_CACHE. It will not install the package. And then you run:

pip install --find-links=file://$SDIST_CACHE --no-index --index-url=file:///dev/null <package name> 

To install the package into your virtual environment. Ideally, $SDIST_CACHE would be committed under your source control. When deploying to production, you would run only the second pip command to install the packages without downloading them.

Answer 3

In my opinion, pip2pi is a much more elegant and reliable solution for this problem.

From the docs:

pip2pi builds a PyPI-compatible package repository from pip requirements

pip2pi allows you to create your own PyPI index by using two simple commands:

1. To mirror a package and all of its requirements, use pip2tgz:

   $ cd /tmp/; mkdir package/
   $ pip2tgz packages/ httpie==0.2
   ...
   $ ls packages/
   Pygments-1.5.tar.gz
   httpie-0.2.0.tar.gz
   requests-0.14.0.tar.gz
   

2. To build a package index from the previous directory:

   $ ls packages/
   bar-0.8.tar.gz
   baz-0.3.tar.gz
   foo-1.2.tar.gz
   $ dir2pi packages/
   $ find packages/
   /httpie-0.2.0.tar.gz
   /Pygments-1.5.tar.gz
   /requests-0.14.0.tar.gz
   /simple
   /simple/httpie
   /simple/httpie/httpie-0.2.0.tar.gz
   /simple/Pygments
   /simple/Pygments/Pygments-1.5.tar.gz
   /simple/requests
   /simple/requests/requests-0.14.0.tar.gz
   

3. To install from the index you built in step 2., you can simply use:

   pip install --index-url=file:///tmp/packages/simple/ httpie==0.2
   

You can even mirror your own index to a remote host with pip2pi.

Answer 4

pip wheel is an excellent option that does what you want with the extra feature of pre-compiling the packages. From the official docs:

Build wheels for a requirement (and all its dependencies):

$ pip wheel --wheel-dir=/tmp/wheelhouse SomePackage

Now your /tmp/wheelhouse directory has all your dependencies precompiled, so you can copy the folder to another server and install everything with this command:

$ pip install --no-index --find-links=/tmp/wheelhouse SomePackage

Note that not all the the packages will be completely portable across machines. Some packages will be built specifically for the Python version, OS distribution and/or hardware architecture that you're using. That will be specified in the file name, like -cp27-none-linux_x86_64 for CPython 2.7 on a 64-bit Linux, etc.