unit test via output sanity checks

Question

I have often seen tests where canned inputs are fed into a program, one checks the outputs generated against canned (expected) outputs usually via diff. If the diff is accepted, the code is deemed to pass the test.

Questions:

1) Is this an acceptable unit test?

2) Usually the unit test inputs are read in from the file system and are big xml files (maybe they represent a very large system). Are unit tests supposed to touch the file system? Or would a unit test create a small input on the fly and feed that to the code to be tested?

3) How can one refactor existing code to be unit testable?

Answer 1

Output differences

If your requirement is to produce output with certain degree of accuracy, then such tests are absolutely fine. It's you who makes the final decision - "Is this output good enough, or not?".

Talking to file system

You don't want your tests to talk to file system in terms of relying on some files to exists somewhere in order for your tests to work (for example, reading values from configuration files). It's a bit different with tests input resources - you can usually embed them in your tests (or at least test project), treat them as part of codebase, and on top of that they usually should be loaded before test executes. For example, when testing rather large XMLs it's reasonable to have them stored as separete files, rather than strings in code files (which sometimes can be done instead).

Point is - you want to keep your tests isolated and repeatable. If you can achieve that with file being loaded at runtime - it's probably fine. However it's still better to have them as part of codebase/resources than standard system file lying somewhere.

Refactoring

This question is fairly broad, but to put you in the right direction - you want to introduce more solid design, decouple objects and separate responsibilities. Better design will make testing easier and, what's most important - possible. Like I said, it's broad and complex topic, with entire books dedicated to it.

Answer 2

1. This is an acceptable unit test.

2. The files being read should be part of the test project so anyone that checks out the project from repository will have the same files at the same relative location.

3. Having black box tests is a great start, you can refactor the existing code and use the current tests to verify that it is still working (depending on the quality of the tests). Here is a short blog about refactoring for testability: http://www.beletsky.net/2011/02/refactoring-to-testability.html

Answer 3

A diff test can be acceptable as a Unit Tests, especially when your using test data that is shared between Unit Tests.

If you don't know how many items there are in the SUT you could use the following:

int itemsBeforeTest = SUT.Items.Count;

SUT.AddItem();

Assert.AreEqual(itemsBeforeTest + 1, SUT.Items.Count);

If a Unit Tests requires so much data that it needs to be read from a big XML file, it's not a real Unit Test. A Unit Test should test a class in complete isolation and mock out all dependencies.

Using a pattern like the Builder pattern, can also help in creating test data for your unit test. The biggest problem with having your test data in a separate file, is that it's hard to understand what the test does exactly. If you create your test data in the arrange part of your unit test, it's immediately clear what is important for your test.

For example, let's say you have the following arrange code to test if the price of an invoice is correct:

Address billingAddress = new Address("Stationsweg 9F",
    "Groningen", "Nederland", "9726AE"); shippingAddress = new Address("Aweg 1",
    "Groningen", "Nederland", "9726AB");
Customer customer = new Customer(99, "Piet", "Klaassens",
                        30,
                        billingAddress,
                        shippingAddress);
Product product = new Product(88, "Tafel", 19.99);
Invoice invoice = new Invoice(customer);

Can be changed to the following when using a Builder

Invoice invoice = Builder<Invoice>.CreateNew()
                    .With(i => i.Product = Builder<Product>.CreateNew()
                        .With(p => p.Price = 19.99)
                        .Build())
                    .Build();

When using a Builder its much easier to see what is important and your code is also more maintainable.

Refactoring code to become more testable is a broad topic. It comes down to thinking about 'how would I test this code?' while you are writing the code.

Take the following example:

public class AlarmClock
{
    public AlarmClock()
    {
        SatelliteSyncService = new SatelliteSyncService();
        HardwareClient = new HardwareClient();
    }
}

This is hard to test. You need to make sure that both the SatteliteSyncService and the HardwareClient are functional when testing the AlarmClock.

This change tot the constructor makes it much easier to test:

public AlarmClock(IHardwareClient hardwareClient, ISatelliteSyncService satelliteSyncService)
{
    SatelliteSyncService = satelliteSyncService;
    HardwareClient = hardwareClient;
}

Techniques like Dependency Injection help with refactoring your code to be more testable. Also watch out for static values like DateTime.Now or the use of a Singleton because they are hard to test.

A really good introduction to writing testable code can be found here.

Answer 4

You should not require the code to be refactored to be able to perform unit tests. Unit tests, as the name implies, are testing a unit of code for the system. The best unit tests are small, quick to execute and exercise only a very small subset of the piece of code being tested (e.g. class).

The reason for having small, compact unit tests that only exercise one part of the code is that the objective of unit tests is to find bugs in that unit of code. If the unit test takes a long time to execute and tests lots of things it makes the identification of a bug in the code that much harder.

As to accessing the file system, I see no problem. Some unit tests may require a database to be constructed before the test is carried out, the output to be checked that would be difficult or time expensive to write the checks in code.

The files for unit testing should be treated like the rest of the code - put under version control. If you are paranoid you could implement a check within the unit test such as perform a MD5 on it and check against a hard coded value so future reruns of the test can verify that the test data has not inadvertenly changed.

Just my humble thoughts.

Answer 5

1) is this an acceptable unit test?

This is not a unit test by definition. A unit test focuses on the smallest possible amount of code. Your test can still be a useful test, regression test, self-documenting test, TDD test, etc. It is just not a unit test, although it may be equally useful.

2) Are unit tests supposed to touch the file system?

Typically not, unless you need to unit test something explicitly related to the filesystem. One reason is, if you have several hundred unit tests it is nice to have them run in a couple seconds rather than minutes.

3) How can one refactor existing code to be unit testable?

A better question is why do you want the code to be unit testable? If you are trying to learn TDD it is better to start with a new project. If you have bugs then try to write tests for the bugs. If the design is slowing you down then you can refactor towards testability over time.

Answer 6

Addressing only the 3rd question. It is extremely difficult. You really need to write tests at the same time you write the code, or before. It is a nightmare to try to slap tests onto an existing code base, and it is often more productive to throw away the code and start over.