I'm trying to setup a local directory with packages that can be reused for installation on a machine without an internet connection but I'm having problems with some of the packages.
I first download the packages with
pip download -r requirements.txt -d my_packages --no-binary :all:
And then I try to install them with
pip install -r requirements.txt --no-index -f my_packages
One of the packages I'm having trouble installing is elasticsearch-dsl==6.1.0:
pip install -r requirements --no-index -f my_packages
Looking in links: my_packages
Collecting elasticsearch-dsl==6.1.0
Collecting six (from elasticsearch-dsl==6.1.0)
Collecting python-dateutil (from elasticsearch-dsl==6.1.0)
Installing build dependencies ... error
Complete output from command /Users/Oskar/.pyenv/versions/2.7.15/envs/no_internet/bin/python2.7 -m pip install --ignore-installed --no-user --prefix /private/var/folders/36/t0_t6_td2f560t2j0149vjmw0000gn/T/pip-build-env-moib0N --no-warn-script-location --no-binary :none: --only-binary :none: --no-index --find-links my_packages -- setuptools wheel:
Looking in links: my_packages
Collecting setuptools
Could not find a version that satisfies the requirement setuptools (from versions: )
No matching distribution found for setuptools
Sure, setuptools I can manually install but there is more packages than that which is required for all the other packages. django-guardian==1.4.9 is another example which requires pytest-runner which for some reason is not downloaded with pip download
Use pip wheel, instead of pip download, to pre-download and compile your dependencies.
$ pip install wheel
$ pip wheel -w my_wheels python-dateutil --no-binary :all:
$ pip install -f my_wheels --no-index python-dateutil # works
$ pip install -f my_packages --no-index python-dateutil # breaks
pip wheel builds the python-dateutil package, so you don't need setuptools_scm later during pip install.
According to the docs,
Wheel is a built-package format, and offers the advantage of not
recompiling your software during every install.
Therefore, I surmise that pip wheel will use build-time dependencies, such as setuptools_scm, but pip install won't, since the .whl has been built.
The --no-binary :all: option still does the right thing: downloads the source in .tar.gz instead of any binary distributions.
(jwodder was shrewd in pointing out the differences between run-time dependencies i.e. install_requires and build-time dependencies i.e. setup_requires.)
I tested this on a local environment, and no setuptools_scm and no pytest-runner, no problem.
(py3) j@computer:~/so-examples|⇒ pip freeze
django-guardian==1.4.9
python-dateutil==2.7.3
six==1.11.0
Because you specified --no-binary :all:, pip only downloads sdists for the packages and does not download any wheels. However, pip still needs each package in wheel form when it comes time to install, and so pip install tries to build a wheel for each package from the sdist, and this step requires setuptools, wheel, and anything listed in the package's setup_requires. I don't know why your environment doesn't have setuptools, but any healthy Python environment should have both it and wheel installed when the environment is created.
Unfortunately for you, packages listed in setup_requires are installed by setuptools itself rather than by pip, so pip download has no way to capture these requirements. If you insist on using --no-binary :all:, then the only solution is to manually compile a list of packages that are missing when pip install runs, download them, and then either install the missing packages before the ones that depend on them or else configure setuptools to look for the downloads in my_packages. Fortunately, the list will likely be rather short (probably just pytest-runner and setuptools_scm).
You can use pip2pi to avoid reinventing the wheel, this is a very handy tool for creating your own pip repos. I maintain a partial pypi mirror for my team and had the best results with this project.
From its Github page: "pip2pi builds a PyPI-compatible package repository from pip requirements"
$ pip install pip2pi
$ mkdir packages
$ pip2tgz packages/ -r requirements.txt
$ dir2pi -n packages/ # <--- this builds/updates the index, i.e. the "simple/" directory
Now just serve the packages/simple/ folder through http and use it as your index.
$ pip install -i http://myserver.net/simple/ -r requirements.txt
To avoid the hassle of pip security check you can tune your pip.conf like this:
[global]
format=columns
disable_pip_version_check = 1
index-url=file:http://myserver.net/simple/
Now you can install from your own mirror with the regular pip install command.
