How do I get SpecFlow to expect an exception?

Question

I'm using SpecFlow, and I'd like to write a scenario such as the following:

Scenario: Pressing add with an empty stack throws an exception
    Given I have entered nothing into the calculator
    When I press add
    Then it should throw an exception

It's calculator.Add() that's going to throw an exception, so how do I handle this in the method marked [Then]?

Answer 1

My solution involves a couple of items to implement, but at the very end it will look much more elegant:

@CatchException
Scenario: Faulty operation throws exception
    Given Some Context
    When Some faulty operation invoked
    Then Exception thrown with type 'ValidationException' and message 'Validation failed'

To make this work, follow those 3 steps:

Step 1

Mark Scenarios you expect exceptions in with some tag, e.g. @CatchException:

@CatchException
Scenario: ...

Step 2

Define an AfterStep handler to change ScenarioContext.TestStatus to be OK. You may only want ignore errors in for When steps, so you can still fail a test in Then verifying an exception. Had to do this through reflection as TestStatus property is internal:

[AfterStep("CatchException")]
public void CatchException()
{
    if (ScenarioContext.Current.StepContext.StepInfo.StepDefinitionType == StepDefinitionType.When)
    {
        PropertyInfo testStatusProperty = typeof(ScenarioContext).GetProperty("TestStatus", BindingFlags.NonPublic | BindingFlags.Instance);
        testStatusProperty.SetValue(ScenarioContext.Current, TestStatus.OK);
    }
}

Step 3

Validate TestError the same way you would validate anything within ScenarioContext.

[Then(@"Exception thrown with type '(.*)' and message '(.*)'")]
public void ThenExceptionThrown(string type, string message)
{
    Assert.AreEqual(type, ScenarioContext.Current.TestError.GetType().Name);
    Assert.AreEqual(message, ScenarioContext.Current.TestError.Message);
}

Answer 2

Great question. I am neither a bdd or specflow expert, however, my first bit of advice would be to take a step back and assess your scenario.

Do you really want to use the terms "throw" and "exception" in this spec? Keep in mind the idea with bdd is to use a ubiquitous language with the business. Ideally, they should be able to read these scenarios and interpret them.

Consider changing your "then" phrase to include something like this:

Scenario: Pressing add with an empty stack displays an error
    Given I have entered nothing into the calculator
    When I press add
    Then the user is presented with an error message

The exception is still thrown in the background but the end result is a simple error message.

Scott Bellware touches this concept in this Herding Code podcast: http://herdingcode.com/?p=176

Answer 3

Changing the scenario not to have an exception is a probably good way to have the scenario more user oriented. However, if you still need to have it working, please consider the following:

1. Catch an exception (I really recommend catching specific exceptions unless you really need to catch all) in the step that invokes an operation and pass it to the scenario context.

   [When("I press add")]
   public void WhenIPressAdd()
   {
      try
      {
        _calc.Add();
      }
      catch (Exception err)
      {
         ScenarioContext.Current[("Error")] = err;
      }
   }
   

2. Validate that exception is stored in the scenario context

   [Then(@"it should throw an exception")]
   public void ThenItShouldThrowAnException()
   {
         Assert.IsTrue(ScenarioContext.Current.ContainsKey("Error"));
   }
   

P.S. It's very close to one of the existing answers. However, if you try getting value from ScenarioContext using syntax like below:

var err = ScenarioContext.Current["Error"]

it will throw another exception in case if "Error" key doesn't exist (and that will fail all scenarios that perform calculations with correct parameters). So ScenarioContext.Current.ContainsKey may be just more appropriate

Answer 4

As a newbie to SpecFlow I won't tell you that this is the way to do it, but one way to do it would be to use the ScenarioContext for storing the exception thrown in the When;

try
{
    calculator.Add(1,1);
}
catch (Exception e)
{
    ScenarioContext.Current.Add("Exception_CalculatorAdd", e);
}

In your Then you could check the thrown exception and do asserts on it;

var exception = ScenarioContext.Current["Exception_CalculatorAdd"];
Assert.That(exception, Is.Not.Null);

With that said; I agree with scoarescoare when he says that you should formulate the scenario in a bit more 'business-friendly' wordings. However, using SpecFlow to drive the implementation of your domain-model, catching exceptions and doing asserts on them can come in handy.

Btw: Check out Rob Conery's screencast over at TekPub for some really good tips on using SpecFlow: http://tekpub.com/view/concepts/5

Answer 5

BDD can be practiced on feature level behavior or/and on unit level behavior.

SpecFlow is a BDD tool that focuses on feature level behavior. Exceptions are not something that you should specify/observe on feature level behavior. Exceptions should be specified/observed on unit-level behavior.

Think of SpecFlow scenarios as a live specification for the non technical stakeholder. You would also not write in the specification that an exception is thrown, but how the system behaves in such a case.

If you do not have any non technical stakeholders, then SpecFlow is the wrong tool for you! Don't waste energy in creating business readable specifications if there is nobody interested in reading them!

There are BDD tools that focus on unit level behavior. In .NET the most popular one is MSpec (http://github.com/machine/machine.specifications). BDD on unit-level can also easily be practices with standard unit-testing frameworks.

That said, you could still check for an exception in SpecFlow.

Here are some more discussion of bdd on unit-level vs. bdd on feature-level: SpecFlow/BDD vs Unit Testing BDD for Acceptance Tests vs. BDD for Unit Tests (or: ATDD vs. TDD)

Also have look at this blog post: Classifying BDD Tools (Unit-Test-Driven vs. Acceptance Test Driven) and a bit of BDD history

Answer 6

In case you are testing user interactions I will only advice what has already been said about focusing on the user experience: "Then the user is presented with an error message". But, in case you are testing a level below the UI, I'd like to share my experience:

I'm using SpecFlow to develop a business layer. In my case, I don't care about the UI interactions, but I still find extremely useful the BDD approach and SpecFlow.

In the business layer I don't want specs that say "Then the user is presented with an error message", but actually verifying that the service correctly responds to a wrong input. I've done for a while what has already been said of catching the exception at the "When" and verifying it at the "Then", but I find this option not optimal, because if you reuse the "When" step you could swallow an exception where you didn't expect it.

Currently, I'm using explicit "Then" clauses, some times without the "When", this way:

Scenario: Adding with an empty stack causes an error
     Given I have entered nothing into the calculator
     Then adding causes an error X

This allows me to specifically code the action and the exception detection in one step. I can reuse it to test as many error cases as I want and it doesn't make me add unrelated code to the non failing "When" steps.