What is the role of asserts in C++ programs that h

Question

I've been adding unit tests to some legacy C++ code, and I've run into many scenarios where an assert inside a function will get tripped during a unit test run. A common idiom that I've run across is functions that take pointer arguments and immediately assert if the argument is NULL.

I could easily get around this by disabling asserts when I'm unit testing. But I'm starting to wonder if unit tests are supposed to alleviate the need for runtime asserts. Is this a correct assessment? Are unit tests supposed to replace runtime asserts by happening sooner in the pipeline (ie: the error is caught in a failing test instead of when the program is running).

On the other hand, I don't like adding soft fails to code (e.g. if (param == NULL) return false;). A runtime assert at least makes it easier to debug a problem in case a unit test missed a bug.

Answer 1

Personally I don't tend to use asserts as, as you've discovered, they often don't play nicely with unit tests. I tend to prefer throwing exceptions in situations where others would often use asserts. These checks, and the exceptions that are thrown on failure, are present in both debug and release builds and I find that they often catch things that 'can't possibly happen' even in release builds (which asserts often don't as they're often compiled out). I find it works better for me and means that I can write unit tests that expect the exception to be thrown on invalid input rather than expecting an assertion to fire.

Lots of people don't agree, see 1, 2, etc but I don't care. Avoiding asserts and using exceptions instead works well for me and helps me to produce robust code for clients...

Answer 2

First, for a unit test to hit an assert (or an ASSERT or _ASSERT or _ASSERTE on Windows builds), the unit test would need to run the code under test with the debug build.

I guess this can easily happen on a developer's machine. For our nightly builds, we only run the unit tests in the release configuration, so there's no worries about asserts there.

Second, one can take the normative approach with asserts --

Asserts are meant to ensure that certain conditions / invariants are always valid during the lifetime of the program. Or to be more precise, to ensure that if such a condition gets broken, we get to know about it ASAP, as close to the root cause of the problem as possible.

In this case, no unit test should raise an assertion, because calling the code in a way that an assertion is raised should not be possible.

or one can take the "pragmatic" approach with assertions:

Let developers sprinkle ASSERT all over the place for "don't do this" and "not implemented" scenarios. (And we can argue all day whether that's wrong™ or right™, but that won't get the features delivered.)

If you take the pragmatic approach, then a unit test hitting an assertion means the unit test called the code in a way that is not quite supported by the code. It just may mean that the code "does nothing" in a release build or it may mean that the code crashes in a release build or it may mean that the code does "something interesting".

Here's the options that I have been known to use:

• If the assert is accompanied by an additional check to make the call "harmless", make the unit test test for the assertion (in debug) and for the "harmless" condition in release.
• For crashes or "something interesting", either there's no unit test that makes sense, or you can make a "debug only" unit test that tests that you really get an assertion (though I'm not so sure that's helpful).

Answer 3

Two possibilities here:

1) The behaviour of the function is defined (by its own interface explicitly, or by general rules for the project) when the input is null. The unit test therefore needs to test this behaviour. So you need a handler to run a process that runs the test case, and the handler validates that the code tripped the assertion and aborted, or you need to mock assert somehow.

2) The behaviour of the function is not defined when the input is null. The unit test therefore needs to not pass in null - a test is a client of the code too. You can't test something if there's nothing in particular that it's supposed to do.

There is no third option, "the function has undefined behaviour when passed a null input, but the tests pass in null anyway, just in case something interesting happens". So I don't see how, "I could easily get around this by disabling asserts when I'm unit testing" helps at all. Surely the unit tests will cause the function under test to dereference a null pointer, which isn't any better than tripping an assert. The whole reason the asserts are there is to stop something even worse from happening.

In your case, perhaps (1) applies in DEBUG builds, and (2) applies in NDEBUG builds. So perhaps you could run the null-input tests only on debug builds, and skip them when testing the release build.

Answer 4

If your unit test code is correct, then the assert is a bug that the unit test has uncovered.

But it is far more likely that your unit test code is violating the constraints of the units it tests - your unit test code is buggy!

Commentators have raised the point that:

Consider a unit test that validates the function handles invalid input properly.

The assert is the programmer's way of handling invalid input, by aborting the program. By aborting, the program is functioning properly.

Asserts are only in the debug builds (they are not compiled if the NDEBUG macro is defined) and it's important to test that the program does something sensible in release builds. Consider running your invalid parameters unit-test on release builds.

If you want both worlds - checking asserts fire in debug builds - then you want your in-thread unit test harness to capture these asserts. You can do this by providing your own assert.h rather than using the system one; a macro (you'd want the __LINE__ and __FILE__ and ##expr) would call a function you wrote, which can throw a custom AssertionFailed when run in a unit test harness. This obviously does not capture asserts compiled into other binaries you link against but did not compile from source with your custom asserter. (I'd recommend this over providing your own abort(), but that's another approach you might consider to achieve the same end.)

Answer 5

I only use asserts to check for things that "can never happen". If an assert fires, then a programming mistake has been made somewhere.

Let's say a method takes the name of an input file, and a unit test feeds it the name of a non-existent file to see if a "file not found" exception is thrown. That's not something that "can never happen". It is possible to not find a file at runtime. I would not use an assert in the method to verify that the file was found.

However, the string length of the file name argument must never be negative. If it is, then there is a bug somewhere. So, I might use an assert to say "this length can never be negative". (It's just an artificial example.)

In the case of your question, if the function asserts != NULL, either the unit test is wrong and should not be sending in NULL, because this will never happen, or, the unit test is valid and NULL might possibly be sent in, and the function is wrong and should not be asserting != NULL and must instead handle that condition.

Answer 6

A runtime assert at least makes it easier to debug a problem in case a unit test missed a bug.

This is a pretty fundamental point. Unit tests are not meant to replace assertions (which IMHO are a standard part of producing good quality code), they're meant to complement them.

Secondly, lets say you have a function Foo which asserts that it's parameters are valid.
In your unit test for Foo you can make sure you only supply valid parameters, so you think you're alright.
6 months down the track some other developer is going to call Foo from some new piece of code (which may or may not have unit tests), and at that point you'll be very grateful you put those asserts in there.