Easy_install cache downloaded files

Question

Is there a way to configure easy_install to avoid having to download the files again when an installation fails?

Answer 1

Here is my solution using pip, managing even installation of binary packages and usable on both, linux and Windows. And as requested, it will limit download from PyPi to the mininum, and as extra bonus, on Linux, it allows to speed up repeated installation of packages usually requiring compilation to a fraction of a second.

Setup takes few steps, but I thing, it is worth to do.

Create pip config file

Create pip configuration file (on linux: ~/.pip/pip.conf, on Windows %HOME%\pip\pip.ini)

My one has this content:

[global]
download-cache = /home/javl/.pip/cache
find-links = /home/javl/.pip/packages

[install]
use-wheel = yes

[wheel]
wheel-dir = /home/javl/.pip/packages

Populating cache dir - goes automatically

The cache dir will get cached version of data downloaded from pypi each time, pip attempts to get some package from pypi. It is easy to get it there (no special care needed), but note, that from pip point of view, these are just cashed data downloaded from PyPi, not packages, so in case you use an option --no-index, it will not work.

pip install --download to populate packages dir

The packages dir is place to put real package files to. E.g. for my favorite package plac, I would do:

$ pip install --download ~/.pip/packages plac

and the plac package file would appeare in that dir. You may even use -r requirements.txt file to do this for multiple packages at once.

These packages are used even whith $ pip install --no-index <something>.

Prevent repeated compilation of the same package on Linux

E.g. lxml package requires compliation, and download and compile may take from 45 seconds to minutes. Using wheel format, you may save here a lot.

Install wheel tool, if you do not have it yet:

$ pip install wheel

Create the wheel for lxml (assuming, you have managed to install lxml in past - it requires some libs in the system to be installed):

$ pip wheel lxml

This goes over download, compile, but finally results in lxml whl file being in packgages dir.

Since then

$ pip install lxml

or even faster

$ pip install --no-index lxml

will take fraction of a second, as it uses wheel formatted package.

Prepare wheel package from Window setup exe package

(note: this can be prepared even on Linux machine, there is no compilation, only some repacking from exe file into whl.)

1. download the exe form of the package from pypi, e.g:

   $ wget https://pypi.python.org/packages/2.7/l/lxml/lxml-3.2.3.win32-py2.7.exe#md5=14ab978b7f0a3382719b65a1ca938d33 $ dir lxml-3.2.3.win32-py2.7.exe

2. convert it to whl

   $ wheel convert lxml-3.2.3.win32-py2.7.exe $ dir lxml-3.2.3.win32-py2.7.exe lxml-3.2.3-cp27-none-win32.whl

3. Test it:

   $ pip install lxml

or

$ pip install --no-index lxml

shall be very quick.

Note, that wheel convert can do exactly the same conversion for egg formatted packages.

Let easy_install and setup.py install reuse your packages dir

easy_install and $ python setup.py install do not seem to offer download cache, but allow to use packages we have in our packages dir.

To do so, edit config file for these two tools:

On Linux: $HOME/.pydistutils.cfg

On Windows: %HOME%\pydistutils.cfg

In my case I have here in /home/javl/.pydistutils.cfg:

[easy_install]
find_links = /home/javl/.pip/packages

This config may help even some cases of pip install calls, when pip attempts to install a package, declaring dependency on other ones. As it delegates this task to setup.py call, without the .pydistutils.cfg config it would download the file from PyPi.

Unfortunatelly, wheel format is not supported in this case (as far as I am aware of).

Answer 2

pip (http://pypi.python.org/pypi/pip/) is a drop-in replacement for the easy_install tool and can do that.

Just run easy_install pip and set an environment variable PIP_DOWNLOAD_CACHE to the path you want pip to store the files. Note that the cache won't work with dependencies that checkout from a source code repository (like svn/git/hg/bzr).

Then use pip install instead of easy_install