How do I install from a local cache with pip?

Question

I install a lot of the same packages in different virtualenv environments. Is there a way that I can download a package once and then have pip install from a local cache?

This would reduce download bandwidth and time.

Answer 1

Using pip only (my version is 1.2.1), you can also build up a local repository like this:

if ! pip install --find-links="file://$PIP_SDIST_INDEX" --no-index <package>; then
    pip install --download-directory="$PIP_SDIST_INDEX" <package>
    pip install --find-links="file://$PIP_SDIST_INDEX" --no-index <package>
fi

In the first call of pip, the packages from the requirements file are looked up in the local repository (only), and then installed from there. If that fails, pip retrieves the packages from its usual location (e.g. PyPI) and downloads it to the PIP_SDIST_INDEX (but does not install anything!). The first call is "repeated" to properly install the package from the local index.

(--download-cache creates a local file name which is the complete (escaped) URL, and pip cannot use this as an index with --find-links. --download-cache will use the cached file, if found. We could add this option to the second call of pip, but since the index already functions as a kind of cache, it does not necessarily bring a lot. It would help if your index is emptied, for instance.)

Answer 2

I think the package "pip-accel" must be a good choice.

Answer 3

For newer Pip versions:

Newer Pip versions now cache downloads by default. See this documentation:

https://pip.pypa.io/en/stable/reference/pip_install/#caching

For older Pip versions:

Create a configuration file named ~/.pip/pip.conf, and add the following contents:

[global]
download_cache = ~/.cache/pip

On OS X, a better path to choose would be ~/Library/Caches/pip since it follows the convention other OS X programs use.

Answer 4

Updated Answer 19-Nov-15

According to the Pip documentation:

Starting with v6.0, pip provides an on by default cache which functions similarly to that of a web browser. While the cache is on by default and is designed do the right thing by default you can disable the cache and always access PyPI by utilizing the --no-cache-dir option.

Therefore, the updated answer is to just use pip with its defaults if you want a download cache.

Original Answer

From the pip news, version 0.1.4:

Added support for an environmental variable $PIP_DOWNLOAD_CACHE which will cache package downloads, so future installations won’t require large downloads. Network access is still required, but just some downloads will be avoided when using this.

To take advantage of this, I've added the following to my ~/.bash_profile:

export PIP_DOWNLOAD_CACHE=$HOME/.pip_download_cache

or, if you are on a Mac:

export PIP_DOWNLOAD_CACHE=$HOME/Library/Caches/pip-downloads

Notes

1. If a newer version of a package is detected, it will be downloaded and added to the PIP_DOWNLOAD_CACHE directory. For instance, I now have quite a few Django packages.
2. This doesn't remove the need for network access, as stated in the pip news, so it's not the answer for creating new virtualenvs on the airplane, but it's still great.

Answer 5

There is a new solution to this called pip-accel, a drop-in replacement for pip with caching built in.

The pip-accel program is a wrapper for pip, the Python package manager. It accelerates the usage of pip to initialize Python virtual environments given one or more requirements files. It does so by combining the following two approaches:

• Source distribution downloads are cached and used to generate a local index of source distribution archives.

• Binary distributions are used to speed up the process of installing dependencies with binary components (like M2Crypto and LXML). Instead of recompiling these dependencies again for every virtual environment we compile them once and cache the result as a binary *.tar.gz distribution.

Paylogic uses pip-accel to quickly and reliably initialize virtual environments on its farm of continuous integration slaves which are constantly running unit tests (this was one of the original use cases for which pip-accel was developed). We also use it on our build servers.

We've seen around 10x speedup from switching from pip to pip-accel.

Answer 6

Starting in version 6.0, pip now does it's own caching:

• DEPRECATION pip install --download-cache and pip wheel --download-cache command line flags have been deprecated and the functionality removed. Since pip now automatically configures and uses it’s internal HTTP cache which supplants the --download-cache the existing options have been made non functional but will still be accepted until their removal in pip v8.0. For more information please see https://pip.pypa.io/en/latest/reference/pip_install.html#caching

More information from the above link:

Starting with v6.0, pip provides an on by default cache which functions similarly to that of a web browser. While the cache is on by default and is designed do the right thing by default you can disable the cache and always access PyPI by utilizing the --no-cache-dir option.