Running unit tests in parallel with pytest?

Question

How can I parallelize the execution of the unit tests written with pytest? Which tactics of parallelism can I choose from?

Answer 1

In order to run pytests in parallel you are going to need to install pytest-xdist. Please see the different parallelism tactics listed below, you can use any of those (however I can bet that one of those suits best for your particular case):

pip install pytest-xdist

# The most primitive case, sending tests to multiple CPUs:
pytest -n NUM

# Execute tests within 3 subprocesses.
pytest --dist=each --tx 3*popen//python=python3.6

# Execute tests in 3 forked subprocess. Won't work on windows.
pytest --dist=each --tx 3*popen//python=python3.6 --boxed

# Sending tests to the ssh slaves
pytest --dist=each --tx ssh=first_slave --tx ssh=seconds_slave --rsyncdir package package

# Sending tests to the socket server, link is available below.
python socketserver.py :8889 &
python socketserver.py :8890 &
pytest --dist=each --tx socket=localhost:8889 --tx socket=localhost:8890

You can provide different values for --dist(-d) parameter, which handles the way tests are distributed across workers, see documentation for more information regarding the usage of --dist.

NOTE: Once tests are executed the socket_server.py is down. I suggest you running socket server from separate terminal windows for debugging purposes

You can introduce more complicated flows, for instance running tests inside docker containers with have started socket servers kind of "pytest workers" and another docker container which communicates to them and serve as "pytest runner".