Install-time dependencies in requirements.txt

Question

I'm using tox to prepare venv and run unit tests and my application needs openopt library which in turn imports numpy.distutils.core in its setup.py.

No matter how I order numpy and openopt in my requirements.txt I can't ensure numpy is installed before setup.py from openopt is executed and exit with ImportError: No module named numpy.distutils.core

How can I fix that? For development I can add numpy to requirements.txt, run tox, add openopt and run tox again but it's not production-ready setup.

Answer 1

UPDATE There is an issue in the tox project that might be implemented that would add functionality to deal with these kinds of problems in a more "official" way. Discussion is here: Add an option to run commands after virtualenv creation but before other steps

UPDATE (a bit more background): The main problem is that it is a BadThing(TM) to assume that some other package is installed already in setup.py. these kinds of problems fall into the area of bootstrapping and they can be hellish to handle properly, but usually this is possible with some extra effort. If you really need a different package at setup time, you can look into setup_requires and some additional magic (have a look e.g. at setuptools_scm for inspiration). In the worst case and if the package is not to complicated, you can make it part of your package (which comes with its own problems though, like keeping it up to date and possible licensing conflicts).

Original answer:

If you use requirements.txt already, an easy (but admittedly ugly) solution would be:

1. create two (or more) requirements files (e.g. requirements-0.txt and requirements-1.txt (hopefully with better names)).
2. sort the packages by dependency into those files
3. use commands instead of deps to install them in the right order

.e.g.

[testenv]
deps = 
    pytest
    # whatever else where order does not matter

commands =
    pip install -r {toxinidir}/requirements-0.txt
    pip install -r {toxinidir}/requirements-1.txt
    # ... and more if needed

    # now do your actual testing ...
    pytest tests/unit

... or if you want to keep it even simpler, just stick the package that is being imported in setup.py of another package right in front of your single requirements.txt

[...]
commands =
    pip install <package that needs to be installed first (e.g. numpy)>
    pip install -r {toxinidir}/requirements.txt        
    pytest tests/unit

Answer 2

It's documented in https://testrun.org/tox/latest/example/basic.html#depending-on-requirements-txt

deps = -rrequirements.txt

According to common practice on github, common trick is:

deps =
    setuptools
    -r{toxinidir}/requirements.txt

Answer 3

I have a generic way to bootstrap build-time dependencies in setup.py. You can use this even if you are not using tox. For this case, add the following snippet to the top of the setup.py script.

from setuptools.dist import Distribution

# Bootstrapping dependencies required for the setup
Distribution(dict(setup_requires=['numpy']))

Warning: This will install numpy using easy_install. Installing numpy with this method is somewhat tricky.