Producing CTest results in Jenkins (xUnit >= 1.58)

问题:

It seems like this should be easy to integrate CMake+CTest in jenkins. The cmakebuilder plugin is extremely easy to configure (just set the source tree and the build tree, done!). However I fail to understand how to call the CTest steps.

According to the main xUnit page, since version 1.58 the XML output from CTest is supported, see bug report.

That's about all the documentation I could find. When I search on google or on stackoverflow, I can only find very old documentation requiring manual steps.

I would like to know how to setup a recent jenkins (1.532.1) with xUnit (1.81). Should I create a 'Add build-step' ? Should I create a 'post-build action' ? What do I need to fill in to get CTest to run and to produce proper XML files, so that jenkins can integrate them ?

回答1:

Here is a small example that demonstrates how to have xUnit pick up a CTest generated XML test result file. The example consists of a single C++ source file main.cpp

#include <cstdlib>
int main() {
    std::exit(-1);
}

and an accompanying CMakeLists.txt:

cmake_minimum_required(VERSION 2.8)
project(JenkinsExample)
enable_testing()
add_executable(main main.cpp)
add_test(test1 main)
add_test(test2 main)
set_tests_properties(test2 PROPERTIES WILL_FAIL ON)

The CMake list file generates an executable main and runs this executable from two CMake tests, where for demonstration purposes one will always fail and the other one will always succeed.

Using Jenkins, set up a new job and a add new cmakebuilder build step. Point the CMake source directory to the folder that contains the example project on your local machine. The CMake build directory should be set to build. This will make Jenkins create a CMake build folder in the job's workspace directory. It's a good idea to set the Clean Build flag to make the job always start with a clean state.

Then, assuming you are running Unix, add an Execute shell job step and enter the following shell script in the Command box:

cd build
/usr/local/bin/ctest --no-compress-output -T Test || /usr/bin/true

Running ctest with the option -T Test will make CTest generate an XML output file in a sub-folder Testing inside the build folder, which can be picked up by the xUnit plug-in in a post-build action then. The || /usr/bin/true is necessary to prevent Jenkins from aborting the build prematurely (without running the xUnit plug-in) if some tests fail.

If you are using Windows, set up a similar Execute Windows batch command job step instead:

cd build
"C:\Program Files (x86)\CMake 2.8\bin\ctest.exe" --no-compress-output -T Test || verify > NUL

Finally the xUnit plug-in must be configured in the following way:

Add a Publish xUnit test result report post-build action and then use the plug-in's Add button to create a CTest-Version test result report. In the CTest-Version (default) Pattern enter the file pattern build/Testing/**/Test.xml.

回答2:

For those wanting to parse CTest output in a Jenkins Declarative Pipeline, you can now do this quite easily using the xUnit plugin, since it can parse CTest output directly.

Add a stage in your Jenkinsfile to run ctest in the build directory, and add a post stage to process the output with xUnit. Here's a skeleton example:

pipeline {
  agent any
  stages {
    stage('Configure') {
      steps {
        dir('build') {
          sh 'cmake ../'
        }
      }
    }
    stage('Build') {
      steps {
        dir('build') {
          sh 'cmake --build .'
        }
      }
    }  
    stage('Test') {
      steps {
        dir('build') {
          sh 'ctest -T test --no-compress-output'
        }
      }
    }
  }
  post {
    always {
      // Archive the CTest xml output
      archiveArtifacts (
        artifacts: 'build/Testing/**/*.xml',
        fingerprint: true
      )

      // Process the CTest xml output with the xUnit plugin
      xunit (
        testTimeMargin: '3000',
        thresholdMode: 1,
        thresholds: [
          skipped(failureThreshold: '0'),
          failed(failureThreshold: '0')
        ],
      tools: [CTest(
          pattern: 'build/Testing/**/*.xml',
          deleteOutputFiles: true,
          failIfNotNew: false,
          skipNoTestFiles: true,
          stopProcessingIfError: true
        )]
      )

      // Clear the source and build dirs before next run
      deleteDir()
    }
  }
}

For examples of how to configure the xUnit plugin in a Declarative Pipeline, the Jenkins Snippets Generator is a very helpful resource.