How do I define an order to run junit tests in Int

Question

I have a flaky junit test that only fails if I run all my tests. I think that one test is causing another test to fail, I want to prove it before I try to fix it.

If I run all tests, it runs the "bad setup" then it runs the "test that fails after bad setup". It also runs a lot of irrelevant, slow tests in between. But if I use a pattern to only run these two, it runs "test that fails after bad setup" then "bad setup". As a result, both pass.

How do I only run "bad setup" and "test that fails after bad setup", in that order?

Answer 1

According to JUnit's wiki:

By design, JUnit does not specify the execution order of test method invocations. Until now, the methods were simply invoked in the order returned by the reflection API. However, using the JVM order is unwise since the Java platform does not specify any particular order, and in fact JDK 7 returns a more or less random order. Of course, well-written test code would not assume any order, but some do, and a predictable failure is better than a random failure on certain platforms.

From version 4.11, JUnit will by default use a deterministic, but not predictable, order (MethodSorters.DEFAULT). To change the test execution order simply annotate your test class using @FixMethodOrder and specify one of the available MethodSorters:

@FixMethodOrder(MethodSorters.JVM): Leaves the test methods in the order returned by the JVM. This order may vary from run to run.

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

You could use MethodSorters.NAME_ASCENDING and change your method names to match with your specific order. I know you're using this just for debugging sake but it's a Test Smell to rely on your test methods execution order and JUnit does not provide more finer grain control over test methods execution order

Answer 2

As said by Ali Dehghani, You can order the test method execution by

@FixMethodOrder(MethodSorters.NAME_ASCENDING): Sorts the test methods by method name, in lexicographic order.

Code:

@FixMethodOrder(MethodSorters.NAME_ASCENDING)
public class ApplicationTest extends ActivityInstrumentationTestCase2<MainActivity> {

    public ApplicationTest() {
        super(MainActivity.class);
    }

    @Rule
    public ActivityTestRule<MainActivity> mActivityTestRule = new ActivityTestRule<>(MainActivity.class);

    @Test
    void t1AttachUI(){
        // testing code goes here
    }

    @Test
    void t2InitializeViews(){
        // testing code goes here
    };

    @Test
    void t3SettingValues(){
        // testing code goes here
    };

    @Test
    void t4Validation(){
        // testing code goes here
    };

    @Test
    void t3AfterButtonPress(){
        // testing code goes here
    };
}

Answer 3

Unit tests ought to be independent so most frameworks don't guarantee or enforce the order in which they are run. But since you want to enforce an order the easiest way I've done it in the past it to create a "throw away" test suite or test method that calls the tests in whatever order I want them to run in. Unit tests are methods, just call them. This is easy to do if you're dealing with tens of tests, not at all appealing if you're dealing with hundreds or thousands.

Try to isolate the flaky interaction as much as possible, then swap around the order of the poorly interacting tests within the throwaway calling method.