How do you reproduce bugs that occur sporadically?

Question

We have a bug in our application that does not occur every time and therefore we don't know its "logic". I don't even get it reproduced in 100 times today.

Disclaimer: This bug exists and I've seen it. It's not a pebkac or something similar.

What are common hints to reproduce this kind of bug?

Answer 1

Add some sort of logging or tracing. For example log the last X actions the user committed before causing the bug (only if you can set a condition to match bug).

Answer 2

all the above, plus throw some brute force soft-robot at it that is semi random, and scater a lot of assert/verify (c/c++, probably similar in other langs) through the code

Answer 3

Tons of logging and careful code review are your only options.

These can be especially painful if the app is deployed and you can't adjust the logging. At that point, your only choice is going through the code with a fine-tooth comb and trying to reason about how the program could enter into the bad state (scientific method to the rescue!)

Answer 4

It's quite common for programmers not to be able to reiterate a user-experienced crash simply because you have developed a certain workflow and habits in using the application that obviously goes around the bug.

At this frequency of 1/100, I'd say that the first thing to do is to handle exceptions and log anything anywhere or you could be spending another week hunting this bug. Also make a priority list of potentially sensitive articulations and features in your project. For example : 1 - Multithreading 2 - Wild pointers/ loose arrays 3 - Reliance on input devices etc. This will help you segment areas that you can brute-force-until-break-again as suggested by other posters.

Answer 5

There's a good chance your application is MTWIDNTBMT (Multi Threaded When It Doesn't Need To Be Multi Threaded), or maybe just multi-threaded (to be polite). A good way to reproduce sporadic errors in multi-threaded applications is to sprinkle code like this around (C#):

Random rnd = new Random();
System.Threading.Thread.Sleep(rnd.Next(2000));

and/or this:

for (int i = 0; i < 4000000000; i++)
{
    // tight loop
}

to simulate threads completing their tasks at different times than usual or tying up the processor for long stretches.

I've inherited many buggy, multi-threaded apps over the years, and code like the above examples usually makes the sporadic errors occur much more frequently.

Answer 6

This has worked for really weird heisenbugs. (I'd also recommend getting a copy of "Debugging" by Dave Argans, these ideas are partly derived form using his ideas!)

(0) Check the ram of the system using something like Memtest86!

The whole system exhibits the problem, so make a test jig that exercises the whole thing. Say it's a server side thing with a GUI, you run the whole thing with a GUI test framework doing the necessary input to provoke the problem.

It doesn't fail 100% of the time, so you have to make it fail more often.

Start by cutting the system in half ( binary chop) worse case, you have to remove sub-systems one at a time. stub them out if they can't be commented out.

See if it still fails. Does it fail more often ?

Keep proper test records, and only change one variable at a time!

Worst case you use the jig and you test for weeks to get meaningful statistics. This is HARD; but remember, the jig is doing the work.

I've got No threads and only one process, and I don't talk to hardware

If the system has no threads, no communicating processes and contacts no hardware; it's tricky; heisenbugs are generally synchronization, but in the no-thread no processes case it's more likely to be uninitialized data, or data used after being released, either on the heap or the stack. Try to use a checker like valgrind.

For threaded/multi-process problems:

Try running it on a different number of CPU's. If it's running on 1, try on 4! Try forcing a 4-computer system onto 1. It'll mostly ensure things happen one at a time.

If there are threads or communicating processes this can shake out bugs.

If this is not helping but you suspect it's synchronization or threading, try changing the OS time-slice size. Make it as fine as your OS vendor allows! Sometimes this has made race conditions happen almost every time!

Obversely, try going slower on the timeslices.

Then you set the test jig running with debugger(s) attached all over the place and wait for the test jig to stop on a fault.

If all else fails, put the hardware in the freezer and run it there. The timing of everything will be shifted.