I'm currently working on a java project using maven. We use the maven surefire plugin to run our junit suite as part of the build process.
Our test suite is rapidly growing, in both coverage and execution time. The execution time is very frustrating and time consuming when you end up waiting ten minutes to find out that a test failed in the first minute of testing.
I would like to find a way to make the build process fail upon the first error/failure in the test suite. I know that this is possible for other build tools, but I have been unable to find a way to do this with maven surefire.
I know that there is an unresolved ticket for this functionality in the surefire jira, but I'm hoping that there is an existing solution out there for this.
As of September 6th 2015 it appears there is, -Dsurefire.skipAfterFailureCount=1.
As of October 19th 2015 version 2.19 has been released.
As far as I know, no, and this really requires the resolution of SUREFIRE-580. If you want to make this happen faster, you should at least vote for the issue and, optionally, submit a patch ;)
There may be a suitable workaround, but it depends on your requirements and you need to use a CI server which can handle jvm process return codes.
The basic idea is to stop Maven's JVM process altogether and let the OS know that the process has stopped unexpectedly. Then, a continuous integration server like Jenkins/Hudson should be able to check for a non-zero exit code and let you know that a test has failed.
The first step is to make surefire exit the JVM at the first test failure. You can do that with JUnit 4.7 or higher by using a custom RunListener (put it in src/test/java):
package org.example
import org.junit.runner.notification.Failure;
import org.junit.runner.notification.RunListener;
public class FailFastListener extends RunListener {
public void testFailure(Failure failure) throws Exception {
System.err.println("FAILURE: " + failure);
System.exit(-1);
}
}
Then, you need to configure that class so surefire will register it with the JUnit 4 Runner. Edit your pom.xml and add the listener configuration property to the maven-surefire-plugin. You will also need to configure surefire to not fork a new JVM process for executing the tests. Otherwise, it will just go on with the next test cases.
<plugin>
<groupId>org.apache.maven.plugins</groupId>
<artifactId>maven-surefire-plugin</artifactId>
<version>2.10</version>
<configuration>
<forkMode>never</forkMode>
<properties>
<property>
<name>listener</name>
<value>org.example.FailFastListener</value>
</property>
</properties>
</configuration>
</plugin>
If this does not help, I'd try to fork the maven surefire junit provider plugin.
Btw, unit tests, by definition, should run faster than 0.1 seconds. If your build really takes so long due to the unit tests, you will have to make them run faster in the future.
You can run maven with --fail-fast option:
The -ff option is very useful for developers running interactive
builds who want to have rapid feedback during the development cycle.
an example can be:
mvn clean test -ff
http://books.sonatype.com/mvnref-book/reference/running-sect-options.html
A few ways to speed it up if not exactly what you need:
If it's a multi module build, add
--fail-fast to the command line to drop out after the first module.
Look into failsafe to move long
running integration tests onto a
different phase of the lifecycle.
Look into a profile based solution for fast and slow tests -
Is there a way to tell surefire to skip tests in a certain package?.
This isn't a direct answer to the question, but it can also be useful to feed the output of maven through grep, to strip out most stuff and help you to see where the test failures are.
Like this:
mvn test | grep -w 'Running\|Tests'
Which produces output (for my code) like this:
Running scot.mygov.pp.test.dashboard.DashboardJsonSerDesCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.029 sec
Running scot.mygov.pp.test.dashboard.DashboardBaselineCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.026 sec
Running scot.mygov.pp.test.dashboard.DashboardDatabaseValidationTest
Tests run: 7, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 2.264 sec
Running scot.mygov.pp.test.dashboard.DashboardServiceWebServiceIsolationCRUDTest
Tests run: 26, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 0.032 sec
Much easier to see where the first error of failure is.
This doesn't solve the question exactly, but the solution that my workplace ultimately came up with was to use Atlassian's Clover to run specialized builds of just tests pertaining to changed code.
We have a Clover build that runs the tests for the changed code and in turn kicks off the full test build.
This has proven to be a satisfactory solution.
