How do you test different Python versions with Tox from within Travis-CI?
I have a tox.ini:
[tox]
envlist = py{27,33,34,35}
recreate = True
[testenv]
basepython =
py27: python2.7
py33: python3.3
py34: python3.4
py35: python3.5
deps =
-r{toxinidir}/pip-requirements.txt
-r{toxinidir}/pip-requirements-test.txt
commands = py.test
which runs my Python unittests in several Python versions and works perfectly.
I want to setup a build in Travis-CI to automatically run this when I push changes to Github, so I have a .travis.yml:
language: python
python:
- "2.7"
- "3.3"
- "3.4"
- "3.5"
install:
- pip install tox
script:
- tox
This technically seems to work, but it redundantly runs all my tests in each version of Python...from each version of Python. So a build that takes 5 minutes now takes 45 minutes.
I tried removing the python list from my yaml file, so Travis will only run a single Python instance, but that causes my Python3.5 tests to fail because the 3.5 interpreter can't be found. Apparently, that's a known limitation as Travis-CI won't install Python3.5 unless you specify that exact version in your config...but it doesn't do that for the other versions.
Is there a way I can workaround this?
For this I would consider using tox-travis. This is a plugin which allows use of Travis CI’s multiple python versions and Tox’s full configurability.
To do this you will configure the .travis.yml file to test with Python:
sudo: false
language: python
python:
- "2.7"
- "3.4"
install: pip install tox-travis
script: tox
This will run the appropriate testenvs, which are any declared env with py27 or py34 as factors of the name by default. Py27 or py34 will be used as fallback if no environments match the given factor.
Further Reading
For more control and flexibility you can manually define your matrix so that the Python version and tox environment match up:
language: python
matrix:
include:
- python: 2.7
env: TOXENV=py27
- python: 3.3
env: TOXENV=py33
- python: 3.4
env: TOXENV=py34
- python: 3.5
env: TOXENV=py35
- python: pypy
env: TOXENV=pypy
- env: TOXENV=flake8
install:
- pip install tox
script:
- tox
In case it's not obvious, each entry in the matrix starts on a line which begins with a hyphen (-). Any items following that line which are indented are additional lines for that single item.
For example, all entries except for the last, are two lines. the last entry is only one line and does not contain a python setting; therefore, it simply uses the default Python version (Python 2.7 according to the Travis documentation). Of course, a specific Python version is not as important for that test. If you wanted to run such a test against both Python 2 and 3 (once each), then it is recommended to use the versions Travis installs by default (2.7 and 3.4) so that the tests complete more quickly as they don't need to install a non-standard Python version first. For example:
- python: 2.7
env: TOXENV=flake8
- python: 3.4
env: TOXENV=flake8
The same works with pypy (second to last entry on matrix) and pypy3 (not shown) in addition to Python versions 2.5-3.6.
While the various other answers provide shortcuts which give you this result in the end, sometimes its helpful to define the matrix manually. Then you can define specific things for individual environments within the matrix. For example, you can define dependencies for only a single environment and avoid the wasted time installing that dependency in every environment.
- python: 3.5
env: TOXENV=py35
- env: TOXENV=checkspelling
before_install: install_spellchecker.sh
- env: TOXENV=flake8
In the above matrix, the install_spellchecker.sh script is only run for the relevant environment, but not the others. The before_install setting was used (rather than install), as using the install setting would have overridden the global install setting. However, if that's what you want (to override/replace a global setting), simply redefine it in the matrix entry. No doubt, various other settings could be defined for individual environments within the matrix as well.
Manually defining the matrix can provide a lot of flexibility. However, if you don't need the added flexibility, one of the various shortcuts in the other answers will keep your config file simpler and easier to read and edit later on.
Travis provides the python version for each test as TRAVIS_PYTHON_VERSION, but in the form '3.4', while tox expects 'py34'.
If you don't want to rely on an external lib (tox-travis) to do the translation, you can do that manually:
language: python
python:
- "2.7"
- "3.3"
- "3.4"
- "3.5"
install:
- pip install tox
script:
- tox -e $(echo py$TRAVIS_PYTHON_VERSION | tr -d .)
Search this pattern in a search engine and you'll find many projects using it.
This works for pypy as well:
tox -e $(echo py$TRAVIS_PYTHON_VERSION | tr -d . | sed -e 's/pypypy/pypy/')
Source: flask-mongoengine's .travis.yml.
TOXENV environment variable can be used to select subset of tests for each version of Python via specified matrix:
language: python
python:
- "2.7"
- "3.4"
- "3.5"
env:
matrix:
- TOXENV=py27-django-19
- TOXENV=py27-django-110
- TOXENV=py27-django-111
- TOXENV=py34-django-19
- TOXENV=py34-django-110
- TOXENV=py34-django-111
- TOXENV=py35-django-19
- TOXENV=py35-django-110
- TOXENV=py35-django-111
install:
- pip install tox
script:
- tox -e $TOXENV
In tox config specify to skip missing versions of Python:
[tox]
skip_missing_interpreters=true
