How can I use cmake to test processes that are exp

Question

I've got some tests that test that clang's address sanitizer catch particular errors. (I want to ensure my understanding of the types of error it can catch is correct, and that future versions continue to catch the type of errors I'm expecting them to.) This means I have several tests that fail by crapping out with an OTHER_FAULT, which appears to be the fixed way that clang's runtime reports an error.

I've set the WILL_FAIL flag to TRUE for these tests, but this only seems to check the return value from a successful, exception-free failure. If the process terminates with an exception, cmake still classes it as a failure.

I've also tried using PASS_REGULAR_EXPRESSION to watch for the distinguishing messages that are printed out when this error occurs, but again, cmake seems to class the test as a failure if it terminates with an exception.

Is there anything I can do to get around this?

(clang-specific answers are also an option! - but I doubt this will be the last time I need to test something like this, so I'd prefer to know how to do it with cmake generally, if it's possible)

Answer 1

CTest provides only basic, commonly used interpretators for result of test programs. For implement other interpretators you can write simple program/script, which wraps the test and interpret its result as needed. E.g. C program (for Linux):

test_that_crash.c:

#include <unistd.h>
#include <stdlib.h>
#include <sys/types.h>

int main(int argc, char** argv)
{
    pid_t pid = fork();
    if(pid == -1)
    {
        // fork fails
        return 1;
    }
    else if(pid)
    {
        // Parent - wait child and interpret its result
        int status = 0;
        wait(&status);
        if(WIFSIGNALED(status)) return 0; // Signal-terminated means success
        else return 1;
    }
    else
    {
        // Child - execute wrapped command
        execvp(argv[1], argv + 1);
        exit(1);
    }
}

This program can be used in CMake as follows:

CMakeLists.txt:

# Compile our wrapper
add_executable(test_that_crash test_that_crash.c)
# Similar to add_test(name command), but test is assumed successfull only if it is crashed(signalled)
macro(add_test_crashed name command)
    # Use generic flow of add_test() command for automatically recognize our executable target
    add_test(NAME ${name} COMMAND test_that_crash ${command} ${ARGN})
endmacro(add_test_crashed)
# ...

# Add some test, which should crash
add_test_crashed(clang.crash.1 <clang-executable> <clang-args>)

Answer 2

There is also a clang-specific solution: configure its manner of exit using the ASAN_OPTIONS environment variable. (See https://github.com/google/sanitizers/wiki/AddressSanitizerFlags.) To do this, set the ASAN_OPTIONS environment variable to abort_on_error=0. When the address sanitizer detects a problem, the process will then do _exit(1) rather than (presumably) abort(), and will thus appear to have terminated cleanly. You can then pick this up using cmake's WILL_FAIL mechanism. (It's still not clear why OS X and Linux differ in this respect - but there you go.)

As a bonus, the test fails much more quickly.

(Another handy option that can improve turnaround time when running through cmake is to set ASAN_SYMBOLIZER_PATH to an empty value, which stops the address sanitizer symbolizing the stack traces. Symbolizing takes a moment, but there's no point doing it when running through cmake, since you can't see the output.)

Rather than do this by hand, I made a Python script that sets the environment appropriately on OS X (doing nothing on Linux), and invokes the test. I then add each asan test using a macro, along the lines of Tsyvarev's answer.

macro(add_asan_test basename)
  add_executable(${basename} ${basename}.c)
  add_test(NAME test/${basename} COMMAND ${CMAKE_CURRENT_SOURCE_DIR}/wrap_clang_sanitizer_test.py -a $<TARGET_FILE:${basename}>)
  set_tests_properties(test/${basename} PROPERTIES WILL_FAIL TRUE)
endmacro()

This gives a simple pass/fail as quickly as possible. I'm in the habit of investigating failures by running the test in question from the shell by hand and examining the output, in which case I get the stack trace as normal (and the fact exiting by abort is a bit slow is less of a problem).

(There are similar options for the other sanitizers, but I haven't investigated them.)