Rerun flaky JUnit test in case they failed

Question

I have a job A in Jenkins for my automated testing that is triggered if another job B build is successful. The job A run several tests. Some of the test are flaky so I would like to run them again few times and let them the chance to pass so my build won't be unstable/failed.

Is there any plugin I can use?

Answer 1

I would suggest to fix your tests or rewrite them so they will only fail if something is broken. Maybe you can mock away the things that tend to fail. If you are depnending on a database connection, maybe you could use a sqlite or smething which is local.

But there is also a plugin which can retry a build: https://wiki.jenkins-ci.org/display/JENKINS/Naginator+Plugin

Simply install the plugin, and then check the Post-Build action "Retry build after failure" on your project's configuration page.

If you want to rerun tests in JUnit-context, take a look here: SO: How to Re-run failed JUnit tests immediately?

Answer 2

Don't know of any plugin to run just the flaky/failed tests again, only the whole build. It should be possible, I just have not found any (and don't have enough time on my hand to write one). Here's what we did on a large java project where the build was ant based:

The build itself was pretty simple (using xml as formatter inside the junit ant task):
ant clean compile test

The build also accepted a single class name as parameter (using batchtest include section inside the junit ant task):
ant -Dtest.class.pattern=SomeClassName test

At the end of the jenkins job, we used the "Execute shell" build step. The idea was to search for all test results that had errors or failures, figure out the name of the class, then run that particular test class again. The file containing the failure will be overwritten, and the test collector at the end of the build will not see the flaky test failure, during the post build steps.

#!/bin/bash +x
cd ${WORKSPACE}
for i in $(seq 1 3); do 
    echo "Running failed tests $i time(s)"
    for file in `find -path '*/TEST-*.xml' | xargs grep 'errors\|failures' | grep '\(errors\|failures\)="[1-9]' | cut -d ':' -f 1`; do 
        class=`basename ${file} .xml | rev | cut -d '.' -f 1 | rev`
        ant -Dtest.class.pattern=${class} test
    done
done

After getting the build back under control, you definitely need to address the flaky tests. Don't let the green build fool you, there's still work to be done.