How should I unit test threaded code?

Question

I have thus far avoided the nightmare that is testing multi-threaded code since it just seems like too much of a minefield. I'd like to ask how people have gone about testing code that relies on threads for successful execution, or just how people have gone about testing those kinds of issues that only show up when two threads interact in a given manner?

This seems like a really key problem for programmers today, it would be useful to pool our knowledge on this one imho.

Answer 1

Have a look at my related answer at

Designing a Test class for a custom Barrier

It's biased towards Java but has a reasonable summary of the options.

In summary though (IMO) its not the use of some fancy framework that will ensure correctness but how you go about designing you multithreaded code. Splitting the concerns (concurrency and functionality) goes a huge way towards raising confidence. Growing Object Orientated Software Guided By Tests explains some options better than I can.

Static analysis and formal methods (see, Concurrency: State Models and Java Programs) is an option but I've found them to be of limited use in commercial development.

Don't forget that any load/soak style tests are rarely guaranteed to highlight problems.

Good luck!

Answer 2

I spent most of last week at a university library studying debugging of concurrent code. The central problem is concurrent code is non-deterministic. Typically, academic debugging has fallen into one of three camps here:

1. Event-trace/replay. This requires an event monitor and then reviewing the events that were sent. In a UT framework, this would involve manually sending the events as part of a test, and then doing post-mortem reviews.
2. Scriptable. This is where you interact with the running code with a set of triggers. "On x > foo, baz()". This could be interpreted into a UT framework where you have a run-time system triggering a given test on a certain condition.
3. Interactive. This obviously won't work in an automatic testing situation. ;)

Now, as above commentators have noticed, you can design your concurrent system into a more deterministic state. However, if you don't do that properly, you're just back to designing a sequential system again.

My suggestion would be to focus on having a very strict design protocol about what gets threaded and what doesn't get threaded. If you constrain your interface so that there is minimal dependancies between elements, it is much easier.

Good luck, and keep working on the problem.

Answer 3

I have had the unfortunate task of testing threaded code and they are definitely the hardest tests I have ever written.

When writing my tests, I used a combination of delegates and events. Basically it is all about using PropertyNotifyChanged events with a WaitCallback or some kind of ConditionalWaiter that polls.

I am not sure if this was the best approach, but it has worked out for me.

Answer 4

Pete Goodliffe has a series on the unit testing of threaded code.

It's hard. I take the easier way out and try to keep the threading code abstracted from the actual test. Pete does mention that the way I do it is wrong but I've either got the separation right or I've just been lucky.

Answer 5

For Java, check out chapter 12 of JCIP. There are some concrete examples of writing deterministic, multi-threaded unit tests to at least test the correctness and invariants of concurrent code.

"Proving" thread-safety with unit tests is much dicier. My belief is that this is better served by automated integration testing on a variety of platforms/configurations.

Answer 6

I handle unit tests of threaded components the same way I handle any unit test, that is with inversion of control and isolation frameworks. I develop in the .Net-arena and out of the box the threading (among other things) is very hard (I'd say nearly impossible) to fully isolate.

Therefore I've written wrappers that looks something like this (simplified):

public interface IThread
{
    void Start();
    ...
}

public class ThreadWrapper : IThread
{
    private readonly Thread _thread;

    public ThreadWrapper(ThreadStart threadStart)
    {
        _thread = new Thread(threadStart);
    }

    public Start()
    {
        _thread.Start();
    }
}

public interface IThreadingManager
{
    IThread CreateThread(ThreadStart threadStart);
}

public class ThreadingManager : IThreadingManager
{
    public IThread CreateThread(ThreadStart threadStart)
    {
         return new ThreadWrapper(threadStart)
    }
}

From there I can easily inject the IThreadingManager into my components and use my isolation framework of choice to make the thread behave as I expect during the test.

That has so far worked great for me, and I use the same approach for the thread pool, things in System.Environment, Sleep etc. etc.