NUnit - cleanup after test failure

问题:

We have some NUnit tests that access the database. When one of them fails it can leave database in inconsistent state - which is not an issue, since we rebuild database for every test run - but it can cause other tests to fail in the same run.

Is it possible to detect that one of the tests failed and perform some sort of cleanup?

We don't want to write cleanup code in every test, we already do that now. I'd like to perfrom cleanup in Teardown but only if test failed, as cleanup might be expensive.

Update: To clarify - I would like tests to be simple and NOT include any cleanup or error handling logic. I also don't want to perform database reset on every test run - only if test fails. And this code should probably be executed in Teardown method but I am not aware of any way to get info if test we are currently tearing down from failed or was successful.

Update2:

        [Test]
        public void MyFailTest()
        {
            throw new InvalidOperationException();
        }

        [Test]
        public void MySuccessTest()
        {
            Assert.That(true, Is.True);
        }

        [TearDown]
        public void CleanUpOnError()
        {
            if (HasLastTestFailed()) CleanUpDatabase();
        }

I am looking for implementation of HasLastTestFailed()

回答1:

This idea got me interested, so I did a little digging. NUnit doesn't have this ability out of the box, but there is a whole extensibility framework supplied with NUnit. I found this great article about extending NUnit - it was a good starting point. After playing around with it, I came up with the following solution: a method decorated with a custom CleanupOnError attribute will be called if one of the tests in the fixture failed.

Here's how the test looks like:

  [TestFixture]
  public class NUnitAddinTest
  {
    [CleanupOnError]
    public static void CleanupOnError()
    {
      Console.WriteLine("There was an error, cleaning up...");
      // perform cleanup logic
    }

    [Test]
    public void Test1_this_test_passes()
    {
      Console.WriteLine("Hello from Test1");
    }

    [Test]
    public void Test2_this_test_fails()
    {
      throw new Exception("Test2 failed");
    }

    [Test]
    public void Test3_this_test_passes()
    {
      Console.WriteLine("Hello from Test3");
    }
  }

where the attribute is simply:

  [AttributeUsage(AttributeTargets.Method, AllowMultiple = false, Inherited = false)]
  public sealed class CleanupOnErrorAttribute : Attribute
  {
  }

And here is how it's executed from the addin:

public void RunFinished(TestResult result)
{
  if (result.IsFailure)
  {
    if (_CurrentFixture != null)
    {
      MethodInfo[] methods = Reflect.GetMethodsWithAttribute(_CurrentFixture.FixtureType,
                                                             CleanupAttributeFullName, false);
      if (methods == null || methods.Length == 0)
      {
        return;
      }

      Reflect.InvokeMethod(methods[0], _CurrentFixture);
    }
  }
}

But here's the tricky part: the addin must be placed in the addins directory next to the NUnit runner. Mine was placed next to the NUnit runner in TestDriven.NET directory:

C:\Program Files\TestDriven.NET 2.0\NUnit\addins

(I created the addins directory, it wasn't there)

EDIT Another thing is that the cleanup method needs to be static!

I hacked together a simple addin, you can download the source from my SkyDrive. You will have to add references to nunit.framework.dll, nunit.core.dll and nunit.core.interfaces.dll in the appropriate places.

A few notes: The attribute class can be placed anywhere in your code. I didn't want to place it in the same assembly as the addin itself, because it references two Core NUnit assemblies, so I placed it in a different assembly. Just remember to change the line in the CleanAddin.cs, if you decide to put it anywhere else.

Hope that helps.

回答2:

Since version 2.5.7, NUnit allows Teardown to detect if last test failed. A new TestContext class allows tests to access information about themselves including the TestStauts.

For more details, please refer to http://nunit.org/?p=releaseNotes&r=2.5.7

[TearDown]
public void TearDown()
{
    if (TestContext.CurrentContext.Result.Status == TestStatus.Failed)
    {
        PerformCleanUpFromTest();
    }
}

回答3:

Whilst it might be possible to coerce nUnit into doing this it isn't the most sensible design, you could always set a temporary file somewhere and if that file exists, run your cleanup.

I would recommend changing your code so that you have database-transactions enabled and at the end of the test, simply revert the database to the original state (e.g. discard the transaction that represents your unit-tests).

回答4:

Yes, there is. You can use the Teardown attribute which will teardown after each test. You'd want to apply that Database "reset" script that you have and teardown and re-setup before and after each test.

This attribute is used inside a TestFixture to provide a common set of functions that are performed after each test method is run.

Update: Based on the comments and update to the question, I'd say you can use the teardown attribute and use private variables to indicate whether the method contents should fire.

Though, I did also see that you don't want any complex logic or error handling code.

Given that, I'd think that a standard Setup/Teardown would work best for you. It doesn't matter if there is an error and you don't have to have any error handling code.

If you need to special clean up because the next tests depends on successful completion of the current test, I'd suggest to revisit your tests -- they probably shouldn't depend on each other.

回答5:

What about using a Try-Catch block, rethrowing the exception caught?

try
{
//Some assertion
}
catch
{
     CleanUpMethod();
     throw;
}

回答6:

I would do like phsr suggests for now and when you can afford it, refactor the tests so that they never have to rely on the same data that another test needs or even better abstract the data access layer and mock the results coming from that database. It sounds like your tests are rather expensive and you you should do all your query logic on the database and business logic in your assembly you don't really care what the results are that are returned.

You will also be able to test your ExceptionHandling a lot better.

回答7:

Another option is to have a special function that will throw your exceptions, that sets a switch in the testfixture that says an exception occured.

public abstract class CleanOnErrorFixture
{
     protected bool threwException = false;

     protected void ThrowException(Exception someException)
     {
         threwException = true;
         throw someException;
     }

     protected bool HasTestFailed()
     {
          if(threwException)
          {
               threwException = false; //So that this is reset after each teardown
               return true;
          }
          return false;
     }
}

Then using your example:

[TestFixture]
public class SomeFixture : CleanOnErrorFixture
{
    [Test]
    public void MyFailTest()
    {
        ThrowException(new InvalidOperationException());
    }

    [Test]
    public void MySuccessTest()
    {
        Assert.That(true, Is.True);
    }

    [TearDown]
    public void CleanUpOnError()
    {
        if (HasLastTestFailed()) CleanUpDatabase();
    }
}

The only issue here is that the Stack trace will lead to the CleanOnErrorFixture

回答8:

One option not mentioned so far is to wrap the test up in a TransactionScope object, so it doesn't matter what happens as the test never commits anything to the DB.

Here's some details on the technique. You can probably find more if you do a search on unit testing and transactionscope (though you are really doing integration testing if you hit a DB). I've used it successfully in the past.

This approach is simple, does not require any cleanup and ensures that tests are isolated.

Edit- I've just noticed Ray Hayes answer is also similar to mine.

回答9:

How does it fail? Is it possible to put it in a try (do test) / catch (fix broken db) / finally block?

Or you could call a private method to fix it when you've checked your fail condition.

回答10:

I'm not saying this is a great idea, but it should work. Remember that assertion failures are just exceptions. Also don't forget there is also a [TestFixtureTearDown] attribute that runs just once after all tests in the fixture have run.

Using those two facts you can write something like setting a flag if a tests failed and checking the value of the flag in the test fixture tear down.

I don't recommend this, but it would work. You aren't really using NUnit as intended, but you can do it.

[TestFixture]
public class Tests {
     private bool testsFailed = false;

     [Test]
     public void ATest() {
         try {
             DoSomething();
             Assert.AreEqual(....);
         } catch {
            testFailed = true;
         }
     }

     [TestFixtureTearDown]
     public void CleanUp() {
          if (testsFailed) {
              DoCleanup();
          }
     }
}

回答11:

You can add a [TearDown] method with
if (TestContext.CurrentContext.Result.Status != TestStatus.Passed)
some code to be executed if test failed.