Jenkins integration for dotnet test

Question

I'm running a unit test of a dotnet core library using dotnet test. I run the test on my Jenkins slave like this.

dotnet test test/Turbine.Domain.UnitTest -xml mstest-reports/Turbine.Domain.UnitTest.xml

The test report looks like this.

<?xml version="1.0" encoding="utf-8"?>
<assemblies>
  <assembly name="Turbine.Domain.UnitTest.dll" environment="64-bit .NET (unknown version) [collection-per-class, parallel (8 threads)]" test-framework="xUnit.net 2.1.0.3179" run-date="2017-04-07" run-time="13:34:31" total="31" passed="31" failed="0" skipped="0" time="0.170" errors="0">
    <errors />
    <collection total="3" passed="3" failed="0" skipped="0" name="Test collection for Turbine.Domain.Tests.AccumulatePositionsTests" time="0.052">
      <test name="Turbine.Domain.Tests.AccumulatePositionsTests.CanAccumulatePositionsByPortfolioIndex" type="Turbine.Domain.Tests.AccumulatePositionsTests" method="CanAccumulatePositionsByPortfolioIndex" time="0.0402475" result="Pass" />
      <test name="Turbine.Domain.Tests.AccumulatePositionsTests.LotEventsTriggerPositionEventsImmediately" type="Turbine.Domain.Tests.AccumulatePositionsTests" method="LotEventsTriggerPositionEventsImmediately" time="0.0102925" result="Pass" />
      <test name="Turbine.Domain.Tests.AccumulatePositionsTests.CanAccumulatePositionsByDefaultIndex" type="Turbine.Domain.Tests.AccumulatePositionsTests" method="CanAccumulatePositionsByDefaultIndex" time="0.0012357" result="Pass" />
    </collection>
    <collection total="4" passed="4" failed="0" skipped="0" name="Test collection for Turbine.Domain.Tests.Queries.AnalyticsSummaryTests" time="0.087">
      <test name="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests.MarketValueHandlesNegativeAmounts" type="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests" method="MarketValueHandlesNegativeAmounts" time="0.0826806" result="Pass" />
      <test name="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests.CanProduceFirmSummaryFromSnapshot" type="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests" method="CanProduceFirmSummaryFromSnapshot" time="0.0012097" result="Pass" />
      <test name="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests.GrossMarketValueHandlesNegativeAmounts" type="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests" method="GrossMarketValueHandlesNegativeAmounts" time="0.0020873" result="Pass" />
      <test name="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests.FirmSummaryProducesOutputOnQuote" type="Turbine.Domain.Tests.Queries.AnalyticsSummaryTests" method="FirmSummaryProducesOutputOnQuote" time="0.0010767" result="Pass" />
    </collection>
etc...

I use an archiveXUnit block in my Jenkins jobs DSL to try and read in the report.

archiveXUnit {
  msTest {
    pattern('**/mstest-reports/*.xml')
  }
}

Jenkins appears to see the report.

Recording test results

[xUnit] [INFO] - Starting to record.
[xUnit] [INFO] - Processing MSTest-Version N/A (default)
[xUnit] [INFO] - [MSTest-Version N/A (default)] - 1 test report file(s) were found with the pattern '**/mstest-reports/*.xml' relative to '/home/jenkins/workspace/routing/Turbine/build_Turbine' for the testing framework 'MSTest-Version N/A (default)'.
[xUnit] [INFO] - Check 'Failed Tests' threshold.
[xUnit] [INFO] - Check 'Skipped Tests' threshold.
[xUnit] [INFO] - Setting the build status to SUCCESS
[xUnit] [INFO] - Stopping recording.

But it isn't parsing and incorporating the results in its report. I don't see the test report on my Jenkins build dashboard.

Any ideas?

Answer 1

You can use the following Pipeline code to run and publish the dotnet core test results:

node {
stage 'Checkout'
    cleanWs()
    checkout scm

stage 'Build'
    bat "\"C:/Program Files/dotnet/dotnet.exe\" restore \"${workspace}/YourProject.sln\""
    bat "\"C:/Program Files/dotnet/dotnet.exe\" build \"${workspace}/YourProject.sln\""

stage 'UnitTests'
    bat returnStatus: true, script: "\"C:/Program Files/dotnet/dotnet.exe\" test \"${workspace}/YourProject.sln\" --logger \"trx;LogFileName=unit_tests.xml\" --no-build"
    step([$class: 'MSTestPublisher', testResultsFile:"**/unit_tests.xml", failOnError: true, keepLongStdio: true])
}

I have uploaded some examples that I made to my GitHub for everyone to use and contribute, feel free to take a look:

https://github.com/avrum/JenkinsFileFor.NETCore

Those pipline jenkinsfile will add this pipline template to your build:

Answer 2

So I'm using xUnit and the trx format which works happily:

Run Tests:

dotnet test test_dir\test_project.csproj --logger "trx;LogFileName=results\unit_tests.xml"

However using this gives me the following issue:

No test discoverer is registered to perform discovery of test cases.
Register a test discoverer and try again.

And so copying the xunit runner manually to the bin folder of the test project fixes this (yes this is hacky):

copy packages\xunit.runner.visualstudio.2.2.0\build\_common\*.dll test_dir\bin\Release /Y

I then add a Publish xUnit tests step as shown:

Tests then get reported correctly on the project and build pages.

Answer 3

Thanks Matt and kiml42!

I'm using dotnet core 2.2 with MSpec as my test framework, and don't need the hacky copying, and can use the trx format (https://docs.microsoft.com/en-us/dotnet/core/tools/dotnet-test?tabs=netcore21#examples).

The test command I use is: dotnet test --logger "trx;LogFileName=UnitTests.trx", which runs tests in each of the test projects, and writes the results to {ProjectFolder}/TestResults/UnitTests.trx

I do have the MSTest plugin installed, which converts from trx to junit format (https://wiki.jenkins.io/display/JENKINS/MSTest+Plugin).

I also link to the test results using a post/always/step block as follows:

  post {
    always {
      step ([$class: 'MSTestPublisher', testResultsFile:"**/TestResults/UnitTests.trx", failOnError: true, keepLongStdio: true])
    }
  }

I have the following declarative Jenkinsfile pipeline:

pipeline {
  agent any
  stages {
    stage('Restore') {
      steps {
        sh 'dotnet restore'
      }
    }
    stage('Test') {
      steps {
        sh 'dotnet test --logger "trx;LogFileName=UnitTests.trx"'
      }
    }
    stage('Build') {
      steps {
        sh 'dotnet build'
      }
    }
    stage('Stop') {
      steps {
        sh 'sudo systemctl stop core-app.service'
      }
    }
    stage('Deploy') {
      steps {
        sh 'rm -rf /var/www/core-app'
        sh 'cp -R . /var/www/core-app'
      }
    }
    stage('Start') {
      steps {
        sh 'sudo systemctl start core-app.service'
      }
    }
  }
  post {
    always {
      step ([$class: 'MSTestPublisher', testResultsFile:"**/TestResults/UnitTests.trx", failOnError: true, keepLongStdio: true])
    }
  }
  tools {
    msbuild '.NET Core 2.2.103'
  }
  environment {
    ASPNETCORE_ENVIRONMENT = 'Production'
  }
}