Using JUnit categories vs simply organizing tests

Question

I have two logical categories of tests: plain functional unit tests (pass/fail) and benchmark performance tests that are just for metrics/diagnostics.

Currently, I have all test methods in a single class, call it MyTests:

public class MyTests 
{
    @Test
    public void testUnit1()
    {
        ...

        assertTrue(someBool);
    }

    @Test
    public void testUnit2()
    {
        ...

        assertFalse(someBool);
    }

    @Test
    @Category(PerformanceTest.class)
    public void bmrkPerfTest1()
    {
        ...
    }

    @Test
    @Category(PerformanceTest.class)
    public void bmrkPerfTest2()
    {
        ...
    }
}

Then I have a UnitTestSuite defined as

@RunWith(Categories.class)
@Categories.ExcludeCategory(PerformanceTest.class)
@SuiteClasses({ MyTests.class })
public class UnitTestSuite {} 

and a PerformanceTestSuite

@RunWith(Categories.class)
@Categories.IncludeCategory(PerformanceTest.class)
@SuiteClasses({ MyTests.class })
public class PerformanceTestSuite {} 

so that I can run the unit tests in Ant separately from performance tests (I don't think including Ant code is necessary).

This means I have a total of FOUR classes (MyTests, PerformanceTest, PerformanceTestSuite, and UnitTestSuite). I realize I could have just put all the unit tests in one class and benchmark tests in another class and be done with it, without the additional complexity with categories and extra annotations. I call tests by class name in Ant, i.e. don't run all tests in a package.

Does it make sense and what are the reasons to keep it organized by category with the annotation or would it be better if I just refactored it in two simple test classes?

Answer 1

If you only have two test classes then it probably doesn't matter. The project I work on has 50-60 classes. Listing them all by name would be exhausting. You could use filename patterns but I feel annotations are cleaner.

Answer 2

To the question of whether to split the tests in two classes:

As they are clearly very different kinds of tests (unit tests and performance tests), I would put them in different classes in any case, for that reason alone.

Some further musings:

I don't think using @Category annotations is a bad idea however. What I'd do, in a more typical project with tens or hundreds of classes containing tests, is annotate the test classes (instead of methods) with @Category, then use the ClassPathSuite library to avoid duplicated efforts of categorising the tests. (And maybe run the tests by category using Ant.)

If you will only ever have the two test classes, it of course doesn't matter much. You can keep the Categories and Suites, or throw them away (as you said the tests are run by class name in Ant) if having the extra classes bugs you. I'd keep them, and move towards the scenario described above, as usually (in a healthy project) more tests will accumulate over time. :-)