Does Maven surefire plugin run tests using multipl

Question

I'm wondering if the Maven surefire plugin either runs tests multi-threaded by default (and if so can the number of threads be controlled? ) or if it runs tests from the Test classes in a random order or predictable order, or if the order can dictated by some means.

I haven't verified this yet (I'll do so tomorrow just looking for some heads up guidance and verification at this point), but it looks as if my various JUnit Test classes are getting the tests run in some intermixed order. Which makes it a real pain to orchestrate the creating of the test resources (which are quite hefty in my case).

Its probably a classic problem I run my suite with the Eclipse JUnit runner and everything runs very linear and plays nice. I go to Maven cmd line and things seems to be stepping all over each other.

Answer 1

First of all, your unit tests should be independent of each other. This is because the order of execution is not guaranteed even by JUnit, so each test should set up and tear down its context (aka test fixture) independent of what happens before or after.

The order of execution is definitely not random though, in JUnit it tends to be the same (I would guess alphabetical order), but you should not build on it - it can change anytime, and apparently in Surefire the order is different.

Here is a good link on why interacting tests are not a good idea.

Answer 2

JUnit runs tests in the order in which they appear in the .java file (not alphabetically). Maven-surefire runs them in a different order, but not predictably (as far as I can tell).

Ideally, tests would be independent of one another, but singletons and static context can complicate things. A helpful way to get new static contexts between executions of separate TestCase's (but not individual tests) is to set the forkMode variable in your pom.xml..

<forkMode>always</forkMode>

Answer 3

By default, Maven runs your tests in a separate ("forked") process, nothing more (this can be controlled using the forkMode optional parameter).

If you are using TestNG or Junit 4.7+ (since SUREFIRE-555), it is possible to run tests in parallel (see the parallel and the threadCount optional parameters) but that's not a default.

Now, while I'm not sure if the surefire plugin behaves the same as JUnit, it is possible to get some control by manually creating a TestSuite and specify the order in which tests are executed:

TestSuite suite= new TestSuite();
suite.addTest(new MathTest("testAdd"));
suite.addTest(new MathTest("testDivideByZero")); 

You are however strongly advised never to depend upon test execution order, unit tests really should be indeed independent.

P.S.: Just in case, there is also this request SUREFIRE-321 (to run tests in alphabetical order) that you might want to vote for.

Answer 4

I have found that if you are using the -T option in your maven command, Surefire will then fork into forkCount * <specified number of threads by the -T option> number of concurrent processes.

To make them all run in one process despite having multiple threads specified by -T, you can force forkCount to be 0 by adding the option -Dsurefire.forkCount=0