Mark as failed running too long JUnit tests

问题:

I would like to stop and mark as failed too long running junit tests (executed within Maven 3 build). I am aware of three ways of doing it:

1) Using Test annotation with timeout parameter:

@Test(timeout=100)
public void testWithTimeout() {
    ...
}

2) Using Rule annotation:

@Rule
public Timeout globalTimeout = new Timeout(100);

3) Configuring maven-surefire-plugin using following options:

forkedProcessTimeoutInSeconds=1
reuseForks=false

The clue is 1) and 2) requires to change each test (it hurts when you have many thousands). 3) solution is not acceptable as within many modules first test starts the context which is used by the test - tests performance would drastically decrease.

Do you have any other ideas how to achieve that? Any tricky solution (not involving patching JUnit :))?

回答1:

You can try to define your own test runner by writing a class which extends BlockJunit4ClassRunner and fail if the test execution exceeds the defined timeout. Then annotate your test classes with @RunWith(CustomizedTestRunner.class) You still need to modify the classes but the timeout value can be specified in a single location.

回答2:

Actually The Timeout Rule applies the same timeout to all test methods in a class.

If you don't want to add it to every Test class, you can define it once in a Test base class.

For something that doesn't require a change to all classes, you could implement a Junit custom suite runner ( See second code example )

Just for fun how about this hacky solution using the deprecated Thread.stop method?

Before function starts a watcher thread for every test, which after a timeout kills the Junit test thread.

If the test completes, cleanup kills the watcher thread.

Works for this little demo, not sure if this would work on production scale.

import static org.junit.Assert.*;

import org.junit.After;
import org.junit.Before;
import org.junit.Test;

// You could also make this a base class for test classes
public class TimeoutTest {

    Thread watcherThread ;
    Thread junitTestThread;
    final static int testTimeout = 2000;

    @Before
    public void myInit()
    {
        junitTestThread = Thread.currentThread();
        watcherThread = new Thread()
        {
            @Override 
            public void run()
            {
                    try {
                        Thread.sleep(testTimeout);
                        junitTestThread.stop();
                    } catch (InterruptedException e) {
                        // TODO Auto-generated catch block
                        e.printStackTrace();
                    }
            }
        };
        watcherThread.start();
    }

    @After
    public void myCleanup() throws InterruptedException
    {
        watcherThread.stop();
    }

    @Test
    public void testPassingFastTest() throws InterruptedException {
        Thread.sleep(1000);
    }

    @Test
    public void testFailingSlowTest() throws InterruptedException {
        Thread.sleep(3000);
    }

}

Or to do this for several test classes using a suite:

import java.util.Arrays;

import org.junit.runner.Description;
import org.junit.runner.RunWith;
import org.junit.runner.notification.RunListener;
import org.junit.runner.notification.RunNotifier;
import org.junit.runners.Suite;
import org.junit.runners.Suite.SuiteClasses;
import org.junit.runners.model.InitializationError;
import org.junit.runners.model.RunnerBuilder;

@RunWith(AllTests.class)
public class AllTests extends Suite{

    Thread watcherThread ;
    Thread junitTestThread;
    final static int testTimeout = 70;

    public AllTests(final Class<?> clazz) throws InitializationError {
        super(clazz, findClasses());
      }

    private static Class<?>[] findClasses() {
        // You could write code here to get the list of all test classes from specific directories
        return new  Class<?>[] {TimeoutTest.class,TimeoutTest2.class};
      }


    @Override
    public void run(final RunNotifier notifier) {


      notifier.addListener(new RunListener() {
        @Override
        public void testStarted(final Description description) {            
            System.out.println("Before test " + description.getDisplayName());
            junitTestThread = Thread.currentThread();
            watcherThread = new Thread()
            {
                @Override 
                public void run()
                {
                        try {
                            Thread.sleep(testTimeout);
                            junitTestThread.stop();
                        } catch (InterruptedException e) {
                            // TODO Auto-generated catch block
                            e.printStackTrace();
                        }
                }
            };
            watcherThread.start();

        }

        @Override
        public void testFinished(final Description description) {
            System.out.println("After test " + description.getDisplayName());
            watcherThread.stop();

        }

      }
    );

    super.run(notifier);


    }      
}

回答3:

Maybe not quite the solution you are looking for, but if you have your tests running in a continuous build server, there is usually a global timeout for the job. e.g. for Jenkins

回答4:

@Test and @Rule annotations only work with JUnit4. For JUnit3, you have to manage a new Thread manually to set a timeout:

public void testWithTimeout() throws InterruptedException, TimeoutException {
    final int factorialOf = 1 + (int) (30000 * Math.random());
    System.out.println("computing " + factorialOf + '!');

    Thread testThread = new Thread() {
        public void run() {
            System.out.println(factorialOf + "! = " + Utils.computeFactorial(factorialOf));
        }
    };

    testThread.start();
    Thread.sleep(1000);
    testThread.interrupt();

    if (testThread.isInterrupted()) {
        throw new TimeoutException("the test took too long to complete");
    }
}

Source and more info: https://netbeans.org/kb/docs/java/junit-intro.html#Exercise_24

Or a more compact solution is to use a Callable with a Future and ExecutorService. Info here: How can I wrap a method so that I can kill its execution if it exceeds a specified timeout?