How do I use NUnit in VSTS?

Question

I am trying to use NUnit 3 within Visual Studio Team System which they seem to have made really hard. I have added in a test assemblies step to the build process and in Advanced execution options -> Path to Custom Test Adapters, I have put in the directory path to NUnitVisualStudioTestAdapter-2.0.0.vsix plugin as advised in the help.

No matter what I do it doesn't seem to find or attempt to run this. It just runs Visual Studio Test and reports that no tests were found. I have also tried putting the test adapter in as a NuGet package and the path to the tools in that, but that doesn't work either.

Has anyone managed to get NUnit working in VSTS and reporting back test results?

The log I get is:

2017-02-07T10:59:26.2778725Z ##[section]Starting: Test Assemblies **\*.Unit.Tests.dll;-:**\obj\**
2017-02-07T10:59:26.2798723Z ==============================================================================
2017-02-07T10:59:26.2798723Z Task         : Visual Studio Test
2017-02-07T10:59:26.2798723Z Description  : Run tests with Visual Studio test runner
2017-02-07T10:59:26.2798723Z Version      : 2.0.10
2017-02-07T10:59:26.2798723Z Author       : Microsoft Corporation
2017-02-07T10:59:26.2798723Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=835764)
2017-02-07T10:59:26.2798723Z ==============================================================================
2017-02-07T10:59:28.6268742Z ##[warning]No test assemblies found matching the pattern: **\*.Unit.Tests.dll;-:**\obj\**.
2017-02-07T10:59:28.6998724Z ##[section]Finishing: Test Assemblies **\*.Unit.Tests.dll;-:**\obj\**

The unit test DLL file is called App.Web.Unit.Tests.dll.

With system.debug = true set, the log I get is as follows (with some import statements stripped out to fit in 30,000 characters):

2017-02-08T10:44:46.2171630Z ##[section]Starting: Test Assemblies **\*.Unit.Tests.dll;-:**\obj\**
2017-02-08T10:44:46.2251636Z ==============================================================================
2017-02-08T10:44:46.2251636Z Task         : Visual Studio Test
2017-02-08T10:44:46.2251636Z Description  : Run tests with Visual Studio test runner
2017-02-08T10:44:46.2251636Z Version      : 1.0.84
2017-02-08T10:44:46.2251636Z Author       : Microsoft Corporation
2017-02-08T10:44:46.2251636Z Help         : [More Information](https://go.microsoft.com/fwlink/?LinkId=624539)
2017-02-08T10:44:46.2251636Z ==============================================================================
2017-02-08T10:44:46.2381636Z Preparing task execution handler.
2017-02-08T10:44:52.1286673Z Executing the powershell script: D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\1.0.84\VSTest.ps1
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddCommand(D:\a\_tasks\VSTest_ef087383-ee5e-42c7-9a53-ab56c98420f9\1.0.84\VSTest.ps1)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - Add inputParameters
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(testAssembly=**\*Tests.dll;-:**\obj\**)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(testFiltercriteria=)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(runSettingsFile=D:\a\1\s)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(overrideTestrunParameters=)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(codeCoverageEnabled=true)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(runInParallel=false)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(vstestLocationMethod=version)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(vsTestVersion=latest)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(vstestLocation=)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(pathtoCustomTestAdapters=D:\a\1\s\packages)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(otherConsoleOptions=)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(testRunTitle=)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(platform=any cpu)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(configuration=release)
2017-02-08T10:44:52.1296976Z ##[debug]PowerShellHandler.Execute - AddParameter(publishRunAttachments=true)
2017-02-08T10:44:52.4586977Z ##[debug]PowerShellHandler.Execute - Invoke
2017-02-08T10:44:52.5076979Z ##[debug]Entering script VSTest.ps1
2017-02-08T10:44:52.5096982Z ##[debug]vsTestVersion = latest
2017-02-08T10:44:52.5096982Z ##[debug]testAssembly = **\*Tests.dll;-:**\obj\**
2017-02-08T10:44:52.5096982Z ##[debug]testFiltercriteria = 
2017-02-08T10:44:52.5096982Z ##[debug]runSettingsFile = D:\a\1\s
2017-02-08T10:44:52.5096982Z ##[debug]codeCoverageEnabled = true
2017-02-08T10:44:52.5106990Z ##[debug]pathtoCustomTestAdapters = D:\a\1\s\packages
2017-02-08T10:44:52.5106990Z ##[debug]overrideTestrunParameters = 
2017-02-08T10:44:52.5106990Z ##[debug]otherConsoleOptions = 
2017-02-08T10:44:52.5106990Z ##[debug]testRunTitle = 
2017-02-08T10:44:52.5106990Z ##[debug]platform = any cpu
2017-02-08T10:44:52.5106990Z ##[debug]configuration = release
2017-02-08T10:44:52.5106990Z ##[debug]publishRunAttachments = true
2017-02-08T10:44:52.5106990Z ##[debug]vstestLocation = 
2017-02-08T10:44:52.5476990Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Internal\Microsoft.TeamFoundation.DistributedTask.Task.Internal.psd1'.
2017-02-08T10:44:52.5566992Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Internal\Microsoft.TeamFoundation.DistributedTask.Task.Internal.psm1'.
2017-02-08T10:44:52.5656993Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.LegacySDK.dll'.
2017-02-08T10:44:52.6656999Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Common\Microsoft.TeamFoundation.DistributedTask.Task.Common.psd1'.
2017-02-08T10:44:52.6727018Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.Common\Microsoft.TeamFoundation.DistributedTask.Task.Common.psm1'.
2017-02-08T10:44:52.7086999Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.TestResults\Microsoft.TeamFoundation.DistributedTask.Task.TestResults.psm1'.
2017-02-08T10:44:52.7336993Z ##[debug]Loading module from path 'C:\LR\MMS\Services\Mms\TaskAgentProvisioner\Tools\agents\2.111.1\externals\vstshost\Microsoft.TeamFoundation.DistributedTask.Task.CodeCoverage\Microsoft.TeamFoundation.DistributedTask.Task.CodeCoverage.psm1'.
2017-02-08T10:44:52.7406994Z ##[debug]Importing cmdlet 'Invoke-ResultPublisher'.
2017-02-08T10:44:52.7416986Z ##[debug]Importing cmdlet 'Publish-TestResults'.
2017-02-08T10:44:52.7416986Z ##[debug]Importing cmdlet 'Add-BuildArtifactLink'.
2017-02-08T10:44:52.7416986Z ##[debug]Importing cmdlet 'Get-MSBuildLocation'.
2017-02-08T10:44:52.7416986Z ##[debug]Importing cmdlet 'Get-ToolPath'.
2017-02-08T10:44:52.7416986Z ##[debug]Importing cmdlet 'Get-VisualStudioPath'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Get-VssConnection'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Invoke-Ant'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Register-XamarinLicense'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Remove-Environment'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Remove-EnvironmentResources'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Complete-EnvironmentOperation'.
2017-02-08T10:44:52.7427007Z ##[debug]Importing cmdlet 'Complete-EnvironmentResourceOperation'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Complete-ResourceOperation'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Get-Environment'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Get-EnvironmentProperty'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Get-EnvironmentResources'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Get-ExternalIpAddress'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Get-ParsedSessionVariables'.
2017-02-08T10:44:52.7436999Z ##[debug]Importing cmdlet 'Add-TaskIssue'.
2017-02-08T10:44:52.7447001Z ##[debug]Importing cmdlet 'Publish-CodeCoverage'.
2017-02-08T10:44:52.7447001Z ##[debug]Exporting cmdlet 'Enable-CodeCoverage'.
2017-02-08T10:44:52.7447001Z ##[debug]Exporting cmdlet 'Publish-CodeCoverage'.
2017-02-08T10:44:52.7447001Z ##[debug]Importing cmdlet 'Enable-CodeCoverage'.
2017-02-08T10:44:52.7447001Z ##[debug]Importing cmdlet 'Publish-CodeCoverage'.
2017-02-08T10:44:52.8017010Z ##[debug]Invoke - Get-TaskVariable cmdlet
2017-02-08T10:44:52.8107020Z ##[debug]Pattern found in solution parameter. Calling Find-Files.
2017-02-08T10:44:52.8107020Z ##[debug]Calling Find-Files with pattern: **\*Tests.dll;-:**\obj\**
2017-02-08T10:44:52.8147008Z ##[debug]FindFiles.FindMatchingFiles(rootFolder = D:\a\1\s, matchPattern = **\*Tests.dll;-:**\obj\**, includeFiles = True, includeFolders = False
2017-02-08T10:44:52.8177007Z ##[debug]FindFiles.GetMatchingItems(includePatterns.Count = 1, excludePatterns.Count = 1, includeFiles = True, includeFolders = False
2017-02-08T10:44:53.4177077Z ##[debug]FindFiles.FindMatchingFiles - Found 0 matches
2017-02-08T10:44:53.4237058Z ##[debug]Found files: 
2017-02-08T10:44:53.4467065Z ##[debug]Processed: ##vso[task.logissue type=warning;code=002004;]
2017-02-08T10:44:53.4467065Z 
2017-02-08T10:44:53.4467065Z 
2017-02-08T10:44:53.4587060Z ##[warning]No test assemblies found matching the pattern: '**\*Tests.dll;-:**\obj\**'.
2017-02-08T10:44:53.4587060Z ##[debug]Processed: ##vso[task.logissue type=warning]No test assemblies found matching the pattern: '**\*Tests.dll;-:**\obj\**'.
2017-02-08T10:44:53.4607060Z ##[debug]Leaving script VSTest.ps1
2017-02-08T10:44:53.4914646Z ##[section]Finishing: Test Assemblies **\*.Unit.Tests.dll;-:**\obj\**

UPDATE:

I have installed the NUnit3TestAdapter package to the unit test project. I have amended the build step in VSTS:

It is version 2.*(preview)

The search pattern is: *test*.dll !\obj** and in Advanced execution options I have Path to Custom Test Adapters set to $(Build.SourcesDirectory)\

I then get this debug information so at least it has found the test adapters if not the tests now:

[command]"C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe" D:\a\1\s\packages\NUnit3TestAdapter.3.7.0\tools\NUnit3.TestAdapter.dll /EnableCodeCoverage /logger:trx "/TestAdapterPath:\"D:\a\1\s\\\""
Microsoft (R) Test Execution Command Line Tool Version 14.0.25420.1
Copyright (c) Microsoft Corporation.  All rights reserved.

Starting test execution, please wait...
Warning: Using Isolation mode to run tests as required by effective Platform:X86 and .Net Framework:Framework35 settings for test run. Use the /inIsolation parameter to suppress this warning.
Information: NUnit Adapter 3.7.0.0: Test execution started

Information: Running all tests in D:\a\1\s\packages\NUnit3TestAdapter.3.7.0\tools\NUnit3.TestAdapter.dll

Information: NUnit failed to load D:\a\1\s\packages\NUnit3TestAdapter.3.7.0\tools\NUnit3.TestAdapter.dll

Information: NUnit Adapter 3.7.0.0: Test execution complete

Warning: No test is available in D:\a\1\s\packages\NUnit3TestAdapter.3.7.0\tools\NUnit3.TestAdapter.dll. Make sure that installed test discoverers & executors, platform & framework version settings are appropriate and try again.

Answer 1

The path to custom test adapters must point to the extracted vsix or the extracted NuGet package.

The easiest way to make this work is to add the NUnit Test Adapter NuGet package 2.x/3.x to your unit test project. The VsTest task should automatically detect test runners that are restored to the NuGet packages directory:

Make sure the nunit.framework version and the NUnit adapters versions match. Make sure that both packages are added to your test projects.

Answer 2

The issue in your original post is that the Visual Studio Test task cannot find the test assembly (App.Web.Unit.Tests.dll) you'd like to test. And in your update, after you change the search pattern to "*test.dll", it still cannot find the test assembly. It just find the "NUnit3.TestAdapter.dll" which is the binary file for NUnit test adapter and the binary file does not contains any test method. So you need to check the build log to see if the test assembly is generated correctly in build step and what's the path to it and then set the search pattern according to the path.

Update:

According to the logs you provided, the output files are placed in "D:\a\1\a" folder since you specified argument "/p:OutDir=D:\a\1\a". Try changing the search pattern to: D:\a\1\a\*.Unit.Tests.dll or $(build.stagingDirectory)\*.Unit.Tests.dll