CMake: How to Unit-Test your own CMake Script Macr

Question

I've written some convenience wrappers around standard CMake commands and want to unit-test this CMake script code to ensure its functionality.

I've made some progress, but there are two things I hope to get help with:

1. Is there some "official" way of unit-testing your own CMake script code? Something like a special mode to run CMake in? My goal is "white-box testing" (as much as possible).
2. How do I handle the global variables and the variable scopes issues? Inject Global variables into the test via loading the a project's cache, configure the test CMake file or pushing it via -D command line option? Simulation/Testing of variable scopes (cached vs. non-cached, macros/functions/includes, parameters passed by references)?

To start with I've looked into the CMake source code (I'm using CMake version 2.8.10) under /Tests and especially under Tests/CMakeTests. There is a huge number of varieties to be found and it looks like a lot of them are specialized on a single test case.

So I looked also into some available CMake script libraries like CMake++ to see their solution, but those - when they have unit tests - are heavily depending on their own library functions.

Answer 1

How do I handle the global variables and the variable scopes issues? Inject Global variables into the test via loading the a project's cache, configure the test CMake file or pushing it via -D command line option?

Generally speaking all currently existing methods (via cache, via environment variables and via -D command line) are a bad choice in one or another case as involving unpredictable behaviour.

This is a least list of issues i can recall:

• Which one variable could intersect/overlap another one and when?
• Variables load or set can not be applied out of cmake detection stage (for example, in cmake script mode).
• The same unique variable can not hold different values for different OS/compilers/configurations/architectures and so on.
• Variables can not be attached to a package (not scope) term represented by system functions like Find* or add_subdirectory.

I've used variables inside cmake lists a long time and decided to write my own solution to cut off them all at once out of the cmake lists.

The idea is to write a standalone parser through a cmake script to load variables from a file or set of files and define a set of rules to enable variables set in predefined or strict order with check on collisions and overlapping.

Here a list of several features:

• bool A=ON is equal to bool A=TRUE is equal to bool A=1
• path B="c:\abc" is equal on the Windows to path B="C:\ABC" (explicit path variable instead of a string which is by default)
• B_ROOT="c:\abc" is equal on the Windows to B_ROOT="C:\ABC" (variable's type detection by the ending of a variable name)
• LIB1:WIN="c:\lib1" sets only in the Windows, when LIB1:UNIX="/lib/lib1" sets only in the Unix (a variable specialization).
• LIB1:WIN=c:\lib1, LIB1:WIN:MSVC:RELEASE=$/{LIB1}\msvc_release - variables reusing via expansion and specialization

I can not said everything here, but you can take as an example the tacklelib library (https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/) to research the implementation on your own.

The example of described configuration files is stored here: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/config/

The implementation: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake/tacklelib/SetVarsFromFiles.cmake

As a mandatory the cmake list must be initialized through the configure_environment(...) macro: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/CMakeLists.txt

Read the readme file for the details around the tacklelib project: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/README_EN.txt

The entire project currently is an experimental.

PS: Write a parser script on the cmake is a tough task, read at least these issues for the start:

• ;-escape list implicit unescaping: https://gitlab.kitware.com/cmake/cmake/issues/18946
• Not paired]or[characters breaks "file(STRINGS": https://gitlab.kitware.com/cmake/cmake/issues/19156

Is there some "official" way of unit-testing your own CMake script code? Something like a special mode to run CMake in? My goal is "white-box testing" (as much as possible).

I did my own "white-box" or a way of testing my own scripts. I have write a set of modules (which itself dependent to the library) to run test in a separate cmake process: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake/tacklelib/testlib/

My tests built on it: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake_tests/

The idea is to put into the tests directory a hierarchy of directories and files with tests and the runner code would just search for the tests in predefined order to execute each test in a separate cmake process:

function(tkl_testlib_enter_dir test_dir)
  # Algorithm:
  #   1. Iterate not recursively over all `*.include.cmake` files and
  #      call to `tkl_testlib_include` function on each file, then
  #      if at least one is iterated then
  #      exit the algorithm.
  #   2. Iterate non recursively over all subdirectories and
  #      call to the algorithm recursively on each subdirectory, then
  #      continue.
  #   3. Iterate not recursively over all `*.test.cmake` files and
  #      call to `tkl_testlib_test` function on each file, then
  #      exit the algorithm.
  #

, where set of functions can be used both from a runner cmake script or *.include.cmake file:

Where TestLib.cmake is designed to run cycle over creation external cmake processes with a test module - *.test.cmake and these functions should be called from a runner script or from an include module (groups other include modules - *.include.cmake or test modules - *.test.cmake):

tkl_testlib_enter_dir test_dir
tkl_testlib_include test_dir test_file_name
tkl_testlib_test test_dir test_file_name

Where TestModule.cmake automatically includes in all *.test.cmake modules in which you have to put your testing code.

After that you just use tkl_test_assert_true inside a *.test.cmake module to mark a test as succeeded or failed.

Additionally, you can use filter parameters in the runner scripts in the _scripts subdirectory to filter tests out:

--path_match_filter <[+]regex_include_match_expression> | <-regex_exclude_match_expression>[;...]
--test_case_match_filter <[+]regex_include_match_expression> | <-regex_exclude_match_expression>[;...]

Pros:

• The TestModule.cmake does traverse the entire directory with tests by predefined rules, you just need to make sure the correct hierarchy and naming to order the testing.
• Usage of a per directory basis separate inclusion file *.include.cmake to exclusive inclusion or to reorder the tests in the directory and its descedants.
• Existence of a *.test.cmake file is the only requirement to put the test to running by default. To exclusively include or exclude the test you can start use command line flags --path_match_filter ... and --test_case_match_filter ....

Cons:

• Mostly all test functions has implemented through the function keyword, which a bit reduces the functionality of several functions. For example, the tkl_test_assert_true can only mark the test is succeeded or failed. To explicitly interrupt the test you have make the branching through the call to tkl_return_if_failed macro.
• All files in the directory with the tests must have the suffix, .test.cmake - for a test, and .include.cmake - for inclusion commands. All builtin search logic depends on it.
• You have write you own runner to call the script RunTestLib.cmake. The example of a run all on the unix shell could be found here: https://sourceforge.net/p/tacklelib/tacklelib/HEAD/tree/trunk/cmake_tests/_scripts/test_all.sh

The entire project currently is an experimental.

Answer 2

Here is my current solution for unit-testing my own CMake script code.

By the assumption that using CMake Script processing mode is my best catch and that I have to mock the CMake commands that are not usable in script mode I - so far - came up with the following.

The Helper Functions

Utilizing my own global properties, I have written helper functions to store and compare function calls:

function(cmakemocks_clearlists _message_type)
    _get_property(_list_names GLOBAL PROPERTY MockLists)
    if (NOT "${_list_names}" STREQUAL "")
        foreach(_name IN ITEMS ${_list_names})
            _get_property(_list_values GLOBAL PROPERTY ${_name})
            if (NOT "${_list_values}" STREQUAL "")
                foreach(_value IN ITEMS ${_list_values})
                    _message(${_message_type} "cmakemocks_clearlists(${_name}): \"${_value}\"")
                endforeach()
            endif()
            _set_property(GLOBAL PROPERTY ${_name} "")
        endforeach()
    endif()
endfunction()

---

function(cmakemocks_pushcall _name _str)
    _message("cmakemocks_pushcall(${_name}): \"${_str}\"")
    _set_property(GLOBAL APPEND PROPERTY MockLists "${_name}")
    _set_property(GLOBAL APPEND PROPERTY ${_name} "${_str}")
endfunction()

function(cmakemocks_popcall _name _str)
    _get_property(_list GLOBAL PROPERTY ${_name})
    set(_idx -1)
    list(FIND _list "${_str}" _idx)
    if ((NOT "${_list}" STREQUAL "") AND (NOT ${_idx} EQUAL -1))
        _message("cmakemocks_popcall(${_name}): \"${_str}\"")
        list(REMOVE_AT _list ${_idx})
        _set_property(GLOBAL PROPERTY ${_name} ${_list})
    else()
        _message(FATAL_ERROR "cmakemocks_popcall(${_name}): No \"${_str}\"")
    endif()
endfunction()

---

function(cmakemocks_expectcall _name _str)
    _message("cmakemocks_expectcall(${_name}): \"${_str}\" -> \"${ARGN}\"")
    _set_property(GLOBAL APPEND PROPERTY MockLists "${_name}")
    string(REPLACE ";" "|" _value_str "${ARGN}")
    _set_property(GLOBAL APPEND PROPERTY ${_name} "${_str} <<<${_value_str}>>>")
endfunction()

function(cmakemocks_getexpect _name _str _ret)
    if(NOT DEFINED ${_ret})
        _message(SEND_ERROR "cmakemocks_getexpect: ${_ret} given as _ret parameter in not a defined variable. Please specify a proper variable name as parameter.")
    endif()

    _message("cmakemocks_getexpect(${_name}): \"${_str}\"")
    _get_property(_list_values GLOBAL PROPERTY ${_name})

    set(_value_str "")

    foreach(_value IN ITEMS ${_list_values})
        set(_idx -1)
        string(FIND "${_value}" "${_str}" _idx)
        if ((NOT "${_value}" STREQUAL "") AND (NOT ${_idx} EQUAL -1))
            list(REMOVE_ITEM _list_values "${_value}")
            _set_property(GLOBAL PROPERTY ${_name} ${_list_values})

            string(FIND "${_value}" "<<<" _start)
            string(FIND "${_value}" ">>>" _end)
            math(EXPR _start "${_start} + 3")
            math(EXPR _len "${_end} - ${_start}")
            string(SUBSTRING "${_value}" ${_start} ${_len} _value_str)
            string(REPLACE "|" ";" _value_list "${_value_str}")
            set(${_ret} "${_value_list}" PARENT_SCOPE)
            break()
        endif()
    endforeach()
endfunction()

The Mockups

By adding mockups like:

macro(add_library)
    string(REPLACE ";" " " _str "${ARGN}")
    cmakemocks_pushcall(MockLibraries "${_str}")
endmacro()

macro(get_target_property _var)
    string(REPLACE ";" " " _str "${ARGN}")
    set(${_var} "[NULL]")
    cmakemocks_getexpect(MockGetTargetProperties "${_str}" ${_var})
endmacro()

The Tests

I can write a test like this:

MyUnitTests.cmake

cmakemocks_expectcall(MockGetTargetProperties "MyLib TYPE" "STATIC_LIBRARY")
my_add_library(MyLib "src/Test1.cc")
cmakemocks_popcall(MockLibraries "MyLib src/Test1.cc")
...
cmakemocks_clearlists(STATUS)

---

And include it into my CMake projects with:

CMakeLists.txt

add_test(
    NAME TestMyCMake 
    COMMAND ${CMAKE_COMMAND} -P "MyUnitTests.cmake"
)