How to use redirection in cmake add_test

Question

I have a console application called "foo", which takes a reference text file as input (in.txt) and generates text at standard output (I want to keep this behaviour).

In make (not cmake), I use a test target, which calls foo and redirects the output to a file (out.txt) as follows. Then, I use diff to compare the file out.txt with the expected refernece (ref.txt)

test:
    ./foo -a test/in.txt > test/out.txt
    diff test/out.txt test/ref.txt

This works fine using make. Now my question is; how can I use cmake to create a similar Makefile?

From within a subdrectory called build, I tried

project(foo)
...
add_test(NAME test1 COMMAND ./foo ../test/in.txt > ../test/out.txt)
enable_testing()

Using cmake version 3.5, I get a Makefile without errors, but when I call make test, the test itself fails. It seems the cmake command add_test supports command line arguments, but not the redirection. I tried quotes and escaping witout success. Since I could not pass this part, I didn't try to use diff. I just imagine that I could pack foo and diff in one line using & as you can do with bash. That would be the second step.

Answer 1

Turning my comment into an answer

As @Tsyvarev has stated, CTest commands are not run in a shell's context. But you could just add the shell needed yourself and use e.g. sh as the command to be called with add_test().

I've run some tests with your example code and the following did work successfully:

add_test(NAME test1 COMMAND sh -c "$<TARGET_FILE:foo> ../test/in.txt > ../test/out.txt")

This solution is not platform independent (it depends on sh to be available in the search paths).

---

So if you want to be more flexible you could do something like:

include(FindUnixCommands)

file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/in.txt" _in)
file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/test/out.txt" _out)
if (BASH)
    add_test(
        NAME test1 
        COMMAND ${BASH} -c "$<TARGET_FILE:foo> ${_in} > ${_out}"
    )
else()
    if (WIN32)
        add_test(
            NAME test1 
            COMMAND ${CMAKE_COMMAND} -E chdir $<TARGET_FILE_DIR:foo> $ENV{ComSpec} /c "$<TARGET_FILE_NAME:foo> ${_in} > ${_out}"
        )
    else()
        message(FATAL_ERROR "Unknown shell command for ${CMAKE_HOST_SYSTEM_NAME}")
    endif()
endif()

---

Additionally there is the possibility to execute a more platform independent diff with ${CMAKE_COMMAND} -E compare_files <file1> <file2>. So you could simplify your complete makefile based example in CMake with:

add_custom_command(
    TARGET foo
    POST_BUILD
    COMMAND ${CMAKE_COMMAND} -E echo "Running $<TARGET_FILE_NAME:foo> ..."
    COMMAND foo in.txt > out.txt
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)

add_test(
    NAME test1 
    COMMAND ${CMAKE_COMMAND} -E compare_files in.txt out.txt
    WORKING_DIRECTORY ${CMAKE_CURRENT_SOURCE_DIR}/test
)

References

• Integrate bash test scripts in cmake
• CMake: piping commands to executable
• cmake: make tests successfully passing part of the build process

Answer 2

They say you cannot:

There is no redirection of output using add_test arguments.

Unlike to commands in add_custom_command, which are executed as a part of makefile receipts (that is, in the context of some shell), tests are executed directly by CTest, without any shell involved. So, shell mechanisms don't work for tests.

You may create wrapper script, which calls program, given as parameter, and performs redirection, futher diff and so on. Then use this script (with appropriate arguments) as a COMMAND for add_test.