Check that JUnit Extension throws specific Excepti

Question

Suppose I develop an extension which disallows test method names to start with an uppercase character.

public class DisallowUppercaseLetterAtBeginning implements BeforeEachCallback {

    @Override
    public void beforeEach(ExtensionContext context) {
        char c = context.getRequiredTestMethod().getName().charAt(0);
        if (Character.isUpperCase(c)) {
            throw new RuntimeException("test method names should start with lowercase.");
        }
    }
}

Now I want to test that my extension works as expected.

@ExtendWith(DisallowUppercaseLetterAtBeginning.class)
class MyTest {

    @Test
    void validTest() {
    }

    @Test
    void TestShouldNotBeCalled() {
        fail("test should have failed before");
    }
}

How can I write a test to verify that the attempt to execute the second method throws a RuntimeException with a specific message?

Answer 1

After trying the solutions in the answers and the question linked in the comments, I ended up with a solution using the JUnit Platform Launcher.

class DisallowUppercaseLetterAtBeginningTest {

    @Test
    void should_succeed_if_method_name_starts_with_lower_case() {
        TestExecutionSummary summary = runTestMethod(MyTest.class, "validTest");

        assertThat(summary.getTestsSucceededCount()).isEqualTo(1);
    }

    @Test
    void should_fail_if_method_name_starts_with_upper_case() {
        TestExecutionSummary summary = runTestMethod(MyTest.class, "InvalidTest");

        assertThat(summary.getTestsFailedCount()).isEqualTo(1);
        assertThat(summary.getFailures().get(0).getException())
                .isInstanceOf(RuntimeException.class)
                .hasMessage("test method names should start with lowercase.");
    }

    private TestExecutionSummary runTestMethod(Class<?> testClass, String methodName) {
        SummaryGeneratingListener listener = new SummaryGeneratingListener();

        LauncherDiscoveryRequest request = request().selectors(selectMethod(testClass, methodName)).build();
        LauncherFactory.create().execute(request, listener);

        return listener.getSummary();
    }

    @ExtendWith(DisallowUppercaseLetterAtBeginning.class)
    static class MyTest {

        @Test
        void validTest() {
        }

        @Test
        void InvalidTest() {
            fail("test should have failed before");
        }
    }
}

JUnit itself will not run MyTest because it is an inner class without @Nested. So there are no failing tests during the build process.

Update

JUnit itself will not run MyTest because it is an inner class without @Nested. So there are no failing tests during the build process.

This is not completly correct. JUnit itself would also run MyTest, e.g. if "Run All Tests" is started within the IDE or within a Gradle build.

The reason why MyTest was not executed is because I used Maven and I tested it with mvn test. Maven uses the Maven Surefire Plugin to execute tests. This plugin has a default configuration which excludes all nested classes like MyTest.

See also this answer about "Run tests from inner classes via Maven" and the linked issues in the comments.

Answer 2

Another approach could be to use the facilities provided by the new JUnit 5 - Jupiter framework.

I put below the code which I tested with Java 1.8 on Eclipse Oxygen. The code suffers from a lack of elegance and conciseness but could hopefully serve as a basis to build a robust solution for your meta-testing use case.

Note that this is actually how JUnit 5 is tested, I refer you to the unit tests of the Jupiter engine on Github.

public final class DisallowUppercaseLetterAtBeginningTest { 
    @Test
    void testIt() {
        // Warning here: I checked the test container created below will
        // execute on the same thread as used for this test. We should remain
        // careful though, as the map used here is not thread-safe.
        final Map<String, TestExecutionResult> events = new HashMap<>();

        EngineExecutionListener listener = new EngineExecutionListener() {
            @Override
            public void executionFinished(TestDescriptor descriptor, TestExecutionResult result) {
                if (descriptor.isTest()) {
                    events.put(descriptor.getDisplayName(), result);
                }
                // skip class and container reports
            }

            @Override
            public void reportingEntryPublished(TestDescriptor testDescriptor, ReportEntry entry) {}
            @Override
            public void executionStarted(TestDescriptor testDescriptor) {}
            @Override
            public void executionSkipped(TestDescriptor testDescriptor, String reason) {}
            @Override
            public void dynamicTestRegistered(TestDescriptor testDescriptor) {}
        };

        // Build our test container and use Jupiter fluent API to launch our test. The following static imports are assumed:
        //
        // import static org.junit.platform.engine.discovery.DiscoverySelectors.selectClass
        // import static org.junit.platform.launcher.core.LauncherDiscoveryRequestBuilder.request

        JupiterTestEngine engine = new JupiterTestEngine();
        LauncherDiscoveryRequest request = request().selectors(selectClass(MyTest.class)).build();
        TestDescriptor td = engine.discover(request, UniqueId.forEngine(engine.getId())); 

        engine.execute(new ExecutionRequest(td, listener, request.getConfigurationParameters()));

        // Bunch of verbose assertions, should be refactored and simplified in real code.
        assertEquals(new HashSet<>(asList("validTest()", "TestShouldNotBeCalled()")), events.keySet());
        assertEquals(Status.SUCCESSFUL, events.get("validTest()").getStatus());
        assertEquals(Status.FAILED, events.get("TestShouldNotBeCalled()").getStatus());

        Throwable t = events.get("TestShouldNotBeCalled()").getThrowable().get();
        assertEquals(RuntimeException.class, t.getClass());
        assertEquals("test method names should start with lowercase.", t.getMessage());
}

Though a little verbose, one advantage of this approach is it doesn't require mocking and execute the tests in the same JUnit container as will be used later for real unit tests.

With a bit of clean-up, a much more readable code is achievable. Again, JUnit-Jupiter sources can be a great source of inspiration.

Answer 3

If the extension throws an exception then there's not much a @Test method can do since the test runner will never reach the @Test method. In this case, I think, you have to test the extension outside of its use in the normal test flow i.e. let the extension be the SUT. For the extension provided in your question, the test might be something like this:

@Test
public void willRejectATestMethodHavingANameStartingWithAnUpperCaseLetter() throws NoSuchMethodException {
    ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
    Method method = Testable.class.getMethod("MethodNameStartingWithUpperCase");

    Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);

    DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();

    RuntimeException actual =
            assertThrows(RuntimeException.class, () -> sut.beforeEach(extensionContext));
    assertThat(actual.getMessage(), is("test method names should start with lowercase."));
}

@Test
public void willAllowTestMethodHavingANameStartingWithAnLowerCaseLetter() throws NoSuchMethodException {
    ExtensionContext extensionContext = Mockito.mock(ExtensionContext.class);
    Method method = Testable.class.getMethod("methodNameStartingWithLowerCase");

    Mockito.when(extensionContext.getRequiredTestMethod()).thenReturn(method);

    DisallowUppercaseLetterAtBeginning sut = new DisallowUppercaseLetterAtBeginning();

    sut.beforeEach(extensionContext);

    // no exception - good enough
}

public class Testable {
    public void MethodNameStartingWithUpperCase() {

    }
    public void methodNameStartingWithLowerCase() {

    }
}

However, your question suggests that the above extension is only an example so, more generally; if your extension has a side effect (e.g. sets something in an addressable context, populates a System property etc) then your @Test method could assert that this side effect is present. For example:

public class SystemPropertyExtension implements BeforeEachCallback {

    @Override
    public void beforeEach(ExtensionContext context) {
        System.setProperty("foo", "bar");
    }
}

@ExtendWith(SystemPropertyExtension.class)
public class SystemPropertyExtensionTest {

    @Test
    public void willSetTheSystemProperty() {
        assertThat(System.getProperty("foo"), is("bar"));
    }
}

This approach has the benefit of side stepping the potentially awkward setup steps of: creating the ExtensionContext and populating it with the state required by your test but it may come at the cost of limiting the test coverage since you can really only test one outcome. And, of course, it is only feasible if the extension has a side effect which can be evaulated in a test case which uses the extension.

So, in practice, I suspect you might need a combination of these approaches; for some extensions the extension can be the SUT and for others the extension can be tested by asserting against its side effect(s).

Answer 4

JUnit 5.4 introduced the JUnit Platform Test Kit which allows you to execute a test plan and inspect the results.

To take a dependency on it from Gradle, it might look something like this:

testImplementation("org.junit.platform:junit-platform-testkit:1.4.0")

And using your example, your extension test could look something like this:

import org.junit.jupiter.api.extension.ExtendWith
import org.junit.jupiter.api.fail
import org.junit.platform.engine.discovery.DiscoverySelectors
import org.junit.platform.testkit.engine.EngineTestKit
import org.junit.platform.testkit.engine.EventConditions
import org.junit.platform.testkit.engine.TestExecutionResultConditions

internal class DisallowUpperCaseExtensionTest {
  @Test
  internal fun `succeed if starts with lower case`() {
    val results = EngineTestKit
        .engine("junit-jupiter")
        .selectors(
            DiscoverySelectors.selectMethod(ExampleTest::class.java, "validTest")
        )
        .execute()

      results.tests().assertStatistics { stats ->
          stats.finished(1)
        }
  }

  @Test
  internal fun `fail if starts with upper case`() {
    val results = EngineTestKit
        .engine("junit-jupiter")
        .selectors(
            DiscoverySelectors.selectMethod(ExampleTest::class.java, "TestShouldNotBeCalled")
        )
        .execute()

    results.tests().assertThatEvents()
        .haveExactly(
            1,
            EventConditions.finishedWithFailure(
                TestExecutionResultConditions.instanceOf(java.lang.RuntimeException::class.java),
                TestExecutionResultConditions.message("test method names should start with lowercase.")
            )
        )

  }

  @ExtendWith(DisallowUppercaseLetterAtBeginning::class)
  internal class ExampleTest {
    @Test
    fun validTest() {
    }

    @Test
    fun TestShouldNotBeCalled() {
      fail("test should have failed before")
    }
  }
}