Is there a way to “fail fast” for junit with the m

Question

I'm currently working on a java project using maven. We use the maven surefire plugin to run our junit suite as part of the build process.

Our test suite is rapidly growing, in both coverage and execution time. The execution time is very frustrating and time consuming when you end up waiting ten minutes to find out that a test failed in the first minute of testing.

I would like to find a way to make the build process fail upon the first error/failure in the test suite. I know that this is possible for other build tools, but I have been unable to find a way to do this with maven surefire.

I know that there is an unresolved ticket for this functionality in the surefire jira, but I'm hoping that there is an existing solution out there for this.

Answer 1

As of September 6th 2015 it appears there is, -Dsurefire.skipAfterFailureCount=1.

As of October 19th 2015 version 2.19 has been released.

Answer 2

A few ways to speed it up if not exactly what you need:

• If it's a multi module build, add --fail-fast to the command line to drop out after the first module.

• Look into failsafe to move long running integration tests onto a different phase of the lifecycle.

• Look into a profile based solution for fast and slow tests - Is there a way to tell surefire to skip tests in a certain package?.

Answer 3

There may be a suitable workaround, but it depends on your requirements and you need to use a CI server which can handle jvm process return codes.

The basic idea is to stop Maven's JVM process altogether and let the OS know that the process has stopped unexpectedly. Then, a continuous integration server like Jenkins/Hudson should be able to check for a non-zero exit code and let you know that a test has failed.

The first step is to make surefire exit the JVM at the first test failure. You can do that with JUnit 4.7 or higher by using a custom RunListener (put it in src/test/java):

package org.example
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class FailFastListener extends RunListener {
        public void testFailure(Failure failure) throws Exception {
                System.err.println("FAILURE: " + failure);
                System.exit(-1);
        }
}

Then, you need to configure that class so surefire will register it with the JUnit 4 Runner. Edit your pom.xml and add the listener configuration property to the maven-surefire-plugin. You will also need to configure surefire to not fork a new JVM process for executing the tests. Otherwise, it will just go on with the next test cases.

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-surefire-plugin</artifactId>
    <version>2.10</version>
    <configuration>
        <forkMode>never</forkMode>
        <properties>
            <property>
                <name>listener</name>
                <value>org.example.FailFastListener</value>
            </property>
        </properties>
    </configuration>
</plugin>

If this does not help, I'd try to fork the maven surefire junit provider plugin.

Btw, unit tests, by definition, should run faster than 0.1 seconds. If your build really takes so long due to the unit tests, you will have to make them run faster in the future.

Answer 4

As far as I know, no, and this really requires the resolution of SUREFIRE-580. If you want to make this happen faster, you should at least vote for the issue and, optionally, submit a patch ;)

Answer 5

This doesn't solve the question exactly, but the solution that my workplace ultimately came up with was to use Atlassian's Clover to run specialized builds of just tests pertaining to changed code.

We have a Clover build that runs the tests for the changed code and in turn kicks off the full test build.

This has proven to be a satisfactory solution.

Answer 6

You can run maven with --fail-fast option:

The -ff option is very useful for developers running interactive builds who want to have rapid feedback during the development cycle.

an example can be:

mvn clean test -ff

http://books.sonatype.com/mvnref-book/reference/running-sect-options.html