maven-failsafe-plugin Errors and BUILD SUCCESS?

Question

my question is very similar to this one: maven-failsafe-plugin Failures and BUILD SUCCESS?

and I manage to set up failsafe plugin to fail if tests fail.

But if test goes into error state, failsafe plugin still does not break the build.

.................
-------------------------------------------------------
   T E S T S
-------------------------------------------------------
Running xxxxx.IntegrationTierFunctionalTestCase
Tests run: 1, Failures: 0, Errors: 1, Skipped: 0, Time elapsed: 0.054 sec <<< FAILURE!

Results :

Tests in error:
  testException(xxxxx.IntegrationTierFunctionalTestCas

Tests run: 1, Failures: 0, Errors: 1, Skipped: 0

[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is
[INFO] [failsafe:verify {execution: functional-test-1024}]
[INFO] Failsafe report directory: C:\projects\oec-integration-server\trunk\oec-integrati
[WARNING] File encoding has not been set, using platform encoding Cp1252, i.e. build is
[INFO] [failsafe:integration-test {execution: functional-test-24}]
[INFO] Failsafe report directory: C:\projects\oec-integration-server\trunk\oec-integrati
.............
[INFO] ------------------------------------------------------------------------
[INFO] ------------------------------------------------------------------------
[INFO] BUILD SUCCESSFUL
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 58 seconds
[INFO] Finished at: Tue May 28 17:48:13 BST 2013
[INFO] Final Memory: 114M/781M
[INFO] ------------------------------------------------------------------------

for simplicy IntegrationTierFunctionalTestCase contains only this code

import org.junit.Test;
import static org.junit.Assert.fail;
public class IntegrationTierFunctionalTestCase 
{

    @Test
    public void testException(){
        //fail();
        throw new RuntimeException("super error");
    }
}

if I uncomment fail() whole build fails correctly, with build failed. but if I just throw an exception, it fails as on shown above.

oour plugin configuration looks like this

<plugin>
    <groupId>org.apache.maven.plugins</groupId>
    <artifactId>maven-failsafe-plugin</artifactId>
    <version>2.7</version>
    <configuration>
        <redirectTestOutputToFile>true</redirectTestOutputToFile>
        <systemPropertyVariables>
            <oec.env>TEST</oec.env>
            <mule.test.timeoutSecs>2400</mule.test.timeoutSecs>
        </systemPropertyVariables>
        <additionalClasspathElements>
            <additionalClasspathElement>${basedir}/src/main/resources/config</additionalClasspathElement>
            </additionalClasspathElement>
        </additionalClasspathElements>
    </configuration>
    <executions>
        <execution>
            <id>functional-test-1024</id>
            <phase>test</phase>
            <goals>
                <goal>integration-test</goal>
                <goal>verify</goal>
            </goals>
            <configuration>
                <includes>
                    <include>**/IntegrationTierFunctionalTestCase.java</include>
                </includes>
                <forkMode>once</forkMode>
                <argLine>-XX:MaxPermSize=256M -Xmx1024M</argLine>
            </configuration>
        </execution>
    </executions>
</plugin>

What am I missing? And no I do not want to wrap it in try-catch blocks and fail tests manually.

Answer 1

There is one possible additional reason why it happens. Failsafe verify is looking for test classes and expects them to be in ${project.build.directory}/test-classes - if you have a different setup it will just print "No tests to run." and ends if build success regardless of what the result of integration-test phase was.

You have to set testClassesDirectory in plugin configuration :

<plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.18.1</version>

            <configuration>
                <systemPropertyVariables>
                    <environmentType>${environmentType}</environmentType>
                </systemPropertyVariables>
                <summaryFile>${project.build.directory}/failsafe-reports/failsafe-summary.xml</summaryFile>
                <testClassesDirectory>${project.build.directory}/classes</testClassesDirectory>
                <suiteXmlFiles>
                    <suiteXmlFile>${suiteXml}</suiteXmlFile>
                </suiteXmlFiles>
            </configuration>

Answer 2

Please verify that the maven property "maven.test.failure.ignore" is not set to "true" in any of your maven pom.xml files, as it can be the only reason to not stop the build after test failure.

Answer 3

You need having two executions blocks, cause the verify goal of the maven-failsafe-plugin is intended to check the results of the integration tests.

  <executions>
    <execution>
        <id>functional-test-1024</id>
        <phase>test</phase>
        <goals>
            <goal>integration-test</goal>
        </goals>
        <configuration>
            <includes>
                <include>**/IntegrationTierFunctionalTestCase.java</include>
            </includes>
            <forkMode>once</forkMode>
            <argLine>-XX:MaxPermSize=256M -Xmx1024M</argLine>
        </configuration>
    </execution>
    <execution>
        <id>verify</id>
        <phase>verify</phase>
        <goals>
            <goal>verify</goal>
        </goals>
    </execution>
  </executions>

Furthermore you should update the version of the maven-failsafe-plugin to 2.14.1 instead of 2.7. Update: In the meantime update to 2.17.

Answer 4

If you're running the integration tests like this:

mvn test-compile failsafe:integration-test

Then you should know that according to the maven documentation on failsafe:

The Failsafe Plugin will not fail the build during the integration-test phase, thus enabling the post-integration-test phase to execute.

I was able to get the build to fail like this:

mvn test-compile failsafe:integration-test failsafe:verify

And here's my failsafe configuration for reference:

        <plugin>
            <groupId>org.apache.maven.plugins</groupId>
            <artifactId>maven-failsafe-plugin</artifactId>
            <version>2.19</version>
            <configuration>
                <includes>
                    <include>**/*IT.java</include>
                </includes>
            </configuration>
            <executions>
                <execution>
                    <goals>
                        <goal>integration-test</goal>
                    </goals>
                </execution>
            </executions>
        </plugin>