How to get the exception that was thrown when a Cu

Question

I can perform actions on test failure by using:

@After
public void afterTest(Scenario scenario) {
    if (scenario.isFailed()) {
        /*Do stuff*/
    }
}

However some of the actions I need to perform depend on the Exception that was thrown and in what context it was thrown. Is there a way to get the Throwable that caused the test to fail? For example in JUnit I would do this by extending TestWatcher and add a rule to my tests:

@Override
protected void failed(Throwable e, Description description) {
    /*Do stuff with e*/
}

However the cucumber-junit iplementation does not allow the use of rules, so this solution would not work with Cucumber.

I don't think I need to explain why accessing a thrown exception on test failure would be useful, however I will still provide an Example:

My test environment is not always stable, so my tests might fail unexpectedly at any moment (there's no specific place I can try to catch the exception since it could occur at any time). When this happens I need the test to reschedule for another attempt, and log the incident so that we can get some good statistical data on the environment instability (when, how frequent, how long etc.)

Answer 1

If you just want to massage the result being sent to the report then you can extend the CucumberJSONFormatter and override the result method like this:

public class CustomReporter extends CucumberJSONFormatter {

    CustomReporter(Appendable out) {
        super(out);
    }

    /**
     * Truncate the error in the output to the testresult.json file.
     * @param result the error result
     */
    @Override
    void result(Result result) {
        String errorMessage = null;
        if (result.error) {
            errorMessage = "Error: " + truncateError(result.error);
        }
        Result myResult = new Result(result.status, result.duration, errorMessage);
        // Log the truncated error to the JSON report
        super.result(myResult);
    }
}

Then set the plugin option to:

plugin = ["com.myimpl.CustomReporter:build/reports/json/testresult.json"]

Answer 2

I've implemented this method using reflections. You can't access directly to steps errors (stack trace). I've created this static method which allows you to access to "stepResults" attribute and then you can iterate and get the error and do whatever you want.

@After
public void afterScenario(Scenario scenario) {
  if (scenario.isFailed())
    logError(scenario);
}


private static void logError(Scenario scenario) {
   Field field = FieldUtils.getField(((ScenarioImpl) scenario).getClass(), "stepResults", true);
   field.setAccessible(true);
   try {
       ArrayList<Result> results = (ArrayList<Result>) field.get(scenario);
       for (Result result : results) {
           if (result.getError() != null)
               LOGGER.error("Error Scenario: {}", scenario.getId(), result.getError());
       }
   } catch (Exception e) {
       LOGGER.error("Error while logging error", e);
   }
}

Answer 3

You can to this by writing your own custom implementation of Formatter & Reporter interface. The empty implementation of Formatter is the NullFormatter.java which you can extend. You will need to provide implementations for the Reporter interface.

The methods which would be of interest will be the result() of the Reporter interface and possibly the done() method of Formatter. The result() has the Result object which has the exceptions.

You can look at RerunFormatter.java for clarity.

Github Formatter source

public void result(Result result) {
      //Code to create logs or store to a database etc...
      result.getError();
      result.getErrorMessage();
}

You will need to add this class(com.myimpl.CustomFormRep) to the plugin option.

plugin={"pretty", "html:report", "json:reports.json","rerun:target/rerun.txt",com.myimpl.CustomFormRep}

More details on custom formatters.

You can use the rerun plugin to get a list of failed scenarios to run again. Not sure about scheduling a run of failed tests, code to create a batch job or schedule one on your CI tool.