VSTS Build fails running NUnit tests

Question

When building a visual studio solution using visual studio team services it runs the unit tests, written in nunit 3 using the nunit 3 NUnit3TestAdapter it starts running the tests.

The error it gives is: Error: Illegal characters in path.

The complete output as follows

2016-02-11T17:02:40.8514411Z Executing the powershell script: C:\BuildAgent\tasks\VSTest\1.0.29\VSTest.ps1
2016-02-11T17:02:41.1483135Z ##[debug]Calling Invoke-VSTest for all test assemblies
2016-02-11T17:02:41.2108131Z Working folder: C:\BuildAgent\_work\5
2016-02-11T17:02:41.2108131Z Executing C:\Program Files (x86)\Microsoft Visual Studio 14.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow\vstest.console.exe "C:\BuildAgent\_work\5\s\xxxxxxxxxxTests\bin\Release\xxxxxxxxxxTests.dll"  /logger:trx /TestAdapterPath:"C:\BuildAgent\_work\5\s"
2016-02-11T17:02:41.5240692Z Microsoft (R) Test Execution Command Line Tool Version 14.0.24720.0
2016-02-11T17:02:41.5240692Z Copyright (c) Microsoft Corporation.  All rights reserved.
2016-02-11T17:02:42.1326930Z Starting test execution, please wait...
2016-02-11T17:02:46.4921857Z Information: NUnit Adapter 3.0.8.0 executing tests is started
2016-02-11T17:02:46.5077303Z Information: Running all tests in C:\BuildAgent\_work\5\s\xxxxxxxxxxTests\bin\Release\xxxxxxxxxxTests.dll
2016-02-11T17:02:46.6327336Z Information: NUnit Adapter 3.0.8.0 executing tests is finished
2016-02-11T17:02:46.6327336Z ##[error]Error: Exception System.ArgumentException, Exception thrown executing tests
2016-02-11T17:02:46.6327336Z ##[error]
2016-02-11T17:02:46.6327336Z ##[error]Error: Illegal characters in path.
2016-02-11T17:02:46.6327336Z ##[error]
2016-02-11T17:02:46.7108474Z Information: Additionally, you can try specifying '/UseVsixExtensions' command if the test discoverer & executor is installed on the machine as vsix extensions and your installation supports vsix extensions. Example: vstest.console.exe myTests.dll /UseVsixExtensions:true
2016-02-11T17:02:46.8358887Z ##[error]VSTest Test Run failed with exit code: 1
2016-02-11T17:02:46.8514965Z ##[warning]No results found to publish.

We have an on premise build agent but I don't think this should make an difference.

Any advice you can give it appreciated.

Answer 1

FWIW, I switched on verbose logging by following the Tips and Tricks and got this:

2016-02-23T14:45:49.5620657Z Information: Running all tests in C:\TFS2015\_work\3\s\BlaBlaTests\bin\Debug\BlaBlaTests.dll
2016-02-23T14:45:49.6088663Z ##[error]Error: Exception thrown executing tests
2016-02-23T14:45:49.6088663Z ##[error]
2016-02-23T14:45:49.6088663Z ##[error]Error: System.ArgumentException: Illegal characters in path.
2016-02-23T14:45:49.6088663Z ##[error]   at System.IO.Path.CheckInvalidPathChars(String path)
2016-02-23T14:45:49.6088663Z ##[error]   at System.IO.Path.Combine(String path1, String path2)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.Services.ExtensionService.ProcessAddinsFile(DirectoryInfo baseDir, String fileName)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.Services.ExtensionService.FindExtensionsInDirectory(DirectoryInfo startDir)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.Services.ExtensionService.StartService()
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.Services.ServiceManager.StartServices()
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.TestEngine.Initialize()
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.Engine.TestEngine.GetRunner(TestPackage package)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.VisualStudio.TestAdapter.NUnitTestAdapter.GetRunnerFor(String assemblyName)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunAssembly(String assemblyName, IFrameworkHandle frameworkHandle)
2016-02-23T14:45:49.6088663Z ##[error]   at NUnit.VisualStudio.TestAdapter.NUnit3TestExecutor.RunTests(IEnumerable`1 sources, IRunContext runContext, IFrameworkHandle frameworkHandle)
2016-02-23T14:45:49.6088663Z ##[error]
2016-02-23T14:45:49.6088663Z Information: NUnit Adapter 3.0.8.0 executing tests is finished

which correlates with %LOCALAPPDATA%\NUnit\Nunit30Settings.xml being accessed but might not be related. ExtensionService is here.

Issues #117 and #118 are related to this with a fix as provided by RobertK66:

If you install the Version 3.0.8.0 as a Nuget package then there is a corrupted (not empty!) “ignore.addins” file installed in the packages\NUnit3TestAdapter.3.0.8-ctp-8\lib folder. Earlier Versions did install a 0byte empty file!

After removing the content of this file in the /packages/NUnit3TestAdapter.3.0.8-ctp-8\lib folder and deleting the cached version of the adapter under \AppData\Local\Temp\VisualStudioTestExplorerExtensions... everything works as expected!

Answer 2

I met this issue when I use NUnit3 Test Adapter Nuget Package during the build. And I resolved it by installing NUnit3 Test Adapter and upload all the installed files to Version Control and use it as the Test Adapter.

Since you are using on premise build agent. You can easily install the NUnit3 Test Adapter on the build agent and use "/UseVsixExtensions:true" argument as the error message indicates.