JUnit test with dynamic number of tests

Question

In our project I have several JUnit tests that e.g. take every file from a directory and run a test on it. If I implement a testEveryFileInDirectory method in the TestCase this shows up as only one test that may fail or succeed. But I am interested in the results on each individual file. How can I write a TestCase / TestSuite such that each file shows up as a separate test e.g. in the graphical TestRunner of Eclipse? (Coding an explicit test method for each file is not an option.)

Compare also the question ParameterizedTest with a name in Eclipse Testrunner.

Answer 1

JUnit 3

public class XTest extends TestCase {

    public File file;

    public XTest(File file) {
        super(file.toString());
        this.file = file;
    }

    public void testX() {
        fail("Failed: " + file);
    }

}

public class XTestSuite extends TestSuite {

    public static Test suite() {
        TestSuite suite = new TestSuite("XTestSuite");
        File[] files = new File(".").listFiles();
        for (File file : files) {
            suite.addTest(new XTest(file));
        }
        return suite;
    }

}

JUnit 4

import org.junit.Test;
import org.junit.runner.RunWith;
import org.junit.runners.Parameterized;
import org.junit.runners.Parameterized.Parameters;

@RunWith(Parameterized.class)
public class TestY {

    @Parameters
    public static Collection<Object[]> getFiles() {
        Collection<Object[]> params = new ArrayList<Object[]>();
        for (File f : new File(".").listFiles()) {
            Object[] arr = new Object[] { f };
            params.add(arr);
        }
        return params;
    }

    private File file;

    public TestY(File file) {
        this.file = file;
    }

    @Test
    public void testY() {
        fail(file.toString());
    }

}

Answer 2

Take a look at Parameterized Tests in JUnit 4.

Actually I did this a few days ago. I'll try to explain ...

First build your test class normally, as you where just testing with one input file. Decorate your class with:

@RunWith(Parameterized.class)

Build one constructor that takes the input that will change in every test call (in this case it may be the file itself)

Then, build a static method that will return a Collection of arrays. Each array in the collection will contain the input arguments for your class constructor e.g. the file. Decorate this method with:

@Parameters

Here's a sample class.

@RunWith(Parameterized.class)
public class ParameterizedTest {

    private File file;

    public ParameterizedTest(File file) {
        this.file = file;
    }

    @Test
    public void test1() throws Exception {  }

    @Test
    public void test2() throws Exception {  }

    @Parameters
    public static Collection<Object[]> data() {
        // load the files as you want
        Object[] fileArg1 = new Object[] { new File("path1") };
        Object[] fileArg2 = new Object[] { new File("path2") };

        Collection<Object[]> data = new ArrayList<Object[]>();
        data.add(fileArg1);
        data.add(fileArg2);
        return data;
    }
}

Also check this example

Answer 3

You could consider using JUnitParams library, so you would have a few more (cleaner) options:

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test1(File file) throws Exception {  }

    @org.junit.Test
    @junitparams.Parameters(method = "data")
    public void test2(File file) throws Exception {  }

    public static File[] data() {
        return new File[] { new File("path1"), new File("path2") };
    }
}

---

@org.junit.runner.RunWith(junitparams.JUnitParamsRunner.class)
public class ParameterizedTest {

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test1(String path) throws Exception {
        File file = new File(path);
    }

    @org.junit.Test
    @junitparams.Parameters(value = { "path1", "path2" })
    public void test2(String path) throws Exception {
        File file = new File(path);
    }
}

---

You can see more samples of usage here.

In addition about JUnitParams, why writting parameterized tests with it is easier and more readable:

JUnitParams project adds a new runner to JUnit and provides much easier and readable parametrised tests for JUnit >=4.6.

Main differences to standard JUnit Parametrised runner:

• more explicit - params are in test method params, not class fields
• less code - you don't need a constructor to set up parameters
• you can mix parametrised with non-parametrised methods in one class
• params can be passed as a CSV string or from a parameters provider class
• parameters provider class can have as many parameters providing methods as you want, so that you can group different cases
• you can have a test method that provides parameters (no external classes or statics anymore)
• you can see actual parameter values in your IDE (in JUnit's Parametrised it's only consecutive numbers of parameters)

Answer 4

Should be possible in JUnit 3 by inheriting from TestSuite and overriding the tests() method to list the files and for each return an instance of a subclass of TestCase that takes the filename as constructor parameter and has a test method that tests the file given in the constructor.

In JUnit 4 it might be even easier.

Answer 5

I had a similar problem and ended up writing a simple JUnit 4 runner that allows med to dynamically generate tests.

https://github.com/kimble/junit-test-factory

Answer 6

If TestNG is an option, you could use Parameters with DataProviders.

Each individual file's test will have its result shown in the text-based report or Eclipse's TestNG plugin UI. The number of total tests run will count each of your files individually.

This behavior differs from JUnit Theories, in which all results are lumped under one "theory" entry and only count as 1 test. If you want separate result reporting in JUnit, you can try Parameterized Tests.

Test and inputs

public class FileTest {

    @DataProvider(name="files")
    public File[][] getFiles(){
        return new File[][] {
            { new File("file1") },
            { new File("file2") }
        };
        // or scan a directory
    }

    @Test(dataProvider="files")
    public void testFile(File file){
        //run tests on file
    }
}

Example output

PASSED: testFile(file1)
PASSED: testFile(file2)

===============================================
    Default test
    Tests run: 2, Failures: 0, Skips: 0
===============================================