VS2013 CodeCoverage.exe runsettings file never par

Question

Environment: Visual Studio 2013 Premium, Win7Ultimate, CodeCoverage.exe

Goal: Code Coverage Report that excludes test project code to later be converted to a .coveragexml for reporting to SonarQube 5.1.

Annoyance I wouldn't even know of this parse error without adding the /verbose switch to the command. My only indication of a failure was the .coverage file was no longer being generated when I added the /config switch.

File Works in VS2013 IDE: MyProject.runsettings file provides the expected output using "Analyze Code Coverage" in the IDE.

Menu: Test | Test Settings | Select Test Settings File... MyProject.runsettings

Menu: Test | Analyze Code Coverage | All Tests

Attempting to run the CodeCoverage.exe file to generate code coverage for my tests I can't seem to use ANY *.runsettings files without getting an error:

"Error: Failed to parse configuration file <configfile>.runsettings"

Path Definitions:

codeCoveragePath = C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Dynamic Code Coverage Tools

vstestpath = C:\Program Files (x86)\Microsoft Visual Studio 12.0\Common7\IDE\CommonExtensions\Microsoft\TestWindow

myProjectOutputPath = assume correct since I get results when not using /config switch

Run Command receiving Error (assume paths are correct): Note: I'm not showing with /verbose switch since I shouldn't be using it under working conditions

%codeCoveragePath%\CodeCoverage.exe collect /config:MyProject.runsettings /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe %myProjectOutputPath%\MyClass.Tests.Unit.dll

Exe Works if I DON'T use the /config option If I remove the /config:MyProject.runsettings from the run command, I get a full report that includes the test project, but that let's me know the rest of the command is correct, it just doesn't like the runsettings file.

I've tried using the following examples:

Visual Studio 2013 runsettings Template file WITHOUT modification

MSDN's sample file

Completed blank file, no content: error

File with only the xml declaration: error

File with only RunSettings Node declared: error

I've even used the Troubleshooting tips from MSDN, too: no help.

MyProject.runsettings file:

<?xml version="1.0" encoding="utf-8"?>
<RunSettings>
  <DataCollectionRunSettings>
    <DataCollectors>
      <DataCollector friendlyName="Code Coverage" uri="datacollector://Microsoft/CodeCoverage/2.0" assemblyQualifiedName="Microsoft.VisualStudio.Coverage.DynamicCoverageDataCollector, Microsoft.VisualStudio.TraceCollector, Version=11.0.0.0, Culture=neutral, PublicKeyToken=b03f5f7f11d50a3a">
        <Configuration>
          <CodeCoverage>
            <ModulePaths>
              <Exclude>
                <ModulePath>.*\.Tests\.Unit\.dll$</ModulePath>
              </Exclude>
            </ModulePaths>
          </CodeCoverage>
        </Configuration>
      </DataCollector>
    </DataCollectors>
  </DataCollectionRunSettings>
</RunSettings>

The file seems to be correct based on the fact that the IDE will use it and generate the correct output in the "Code Coverage Results" window by only reporting the MyClass code and not any MyClass.Tests.Unit code.

I'm at the point that I think it is the CodeCoverage.exe command line doesn't like the /config option or it is using a different xml schema.

Update

Works gives the output I want, just can't specify file location for next step

%vstestpath%\vstest.console.exe /Settings:MySettings.runsettings %myProjectOutputPath%\MyClass.Tests.Unit.dll

Doesn't Work Gives exact opposite output I want (only test.dll coverage in the report).

%codeCoveragePath%\CodeCoverage.exe collect /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe /Settings:MySettings.runsettings %myProjectOutputPath%\MyClass.Tests.Unit.dll

Still looking for an answer.

Answer 1

I believe you need to specify the runsettings file at the back of the vstest.console.exe using the /Settings: flag (as opposed to the config flag of the CodeCoverage.exe).

So that your command becomes:

%codeCoveragePath%\CodeCoverage.exe collect /output:CoverageOutput.coverage %vstestpath%\vstest.console.exe %myProjectOutputPath%\MyClass.Tests.Unit.dll /Settings:MyProject.runsettings

Answer 2

Okay, Here comes the HACK!!!!

Basic steps:

• Find and delete all *.coverage files
• Run vstest command WITHOUT [codecoverage.exe collect] wrapper
• Find the new *.coverage files and send to [codecoverage.exe analyze] command

Details

I updated the build.proj file I've been using to execute all this to do the Basic steps:

<PropertyGroup>
  <SqCodeCoverageResultsFile>VisualStudio.coveragexml</SqCodeCoverageResultsFile>
</PropertyGroup>
<Target Name="BuildTestAssemblyList"
      BeforeTargets="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <CreateItem Include="**\*.Tests.Unit.dll">
    <Output TaskParameter="Include" ItemName="TestAssemblies" />
  </CreateItem>
</Target>
<Target Name="BuildCoverageFileList"
      BeforeTargets="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <CreateItem Include="**\*.coverage">
    <Output TaskParameter="Include" ItemName="CoverageFiles" />
  </CreateItem>    
</Target>
<Target Name="RunAllTestsWithCodeCoverageAndConvertToXmlOutput">
  <Delete Condition="Exists($(SqCodeCoverageResultsFile))" Files="$(SqCodeCoverageResultsFile)" />
  <Delete Files="@(CoverageFiles)" />
  <Exec Command="&quot;$(VsTestExecutable)&quot; /EnableCodeCoverage /Settings:MyProject.runsettings /inIsolation /logger:trx  @(TestAssemblies->'&quot;%(FullPath)&quot;',' ') " /> 
  <CreateItem Include="**\*.coverage">
    <Output TaskParameter="Include" ItemName="NewCoverageFiles" />
  </CreateItem>    
  <Exec Command="&quot;$(VsCodeCoverageExecutable)&quot; analyze /output:&quot;$(SqCodeCoverageResultsFile)&quot; @(NewCoverageFiles->'&quot;%(FullPath)&quot;',' ') " />
</Target>`

Now running the CodeCoverage.exe analyze command with those found *.coverage files will now output to the same filename I was trying to achieve before and get the results I wanted.

MSBuild.SonarQube.Runner.exe gets what it wants, I have the results I want, and the world can start to revolve again =)

Improvement: I could use a CustomTask and search for the perfect or most recent or whatever logic you can think of for that correct single file so I wouldn't have to delete all my other *.coverage files. I could, but I didn't because this is supposed to be run on a build server that shouldn't have that kind of history laying around anyways in my opinion.

Answer 3

I was having the same problem and found your question when searching for some info on my error. My assumption had been also that the config file format was the same as the .runsettings used by vstest.console.exe but from the parse error after adding /verbose I then suspected it was a different format so had a look to see if there was a default config for CodeCoverage.exe to see what it looked like, and I did find one at:

C:\Program Files (x86)\Microsoft Visual Studio 12.0\Team Tools\Dynamic Code Coverage Tools\CodeCoverage.config

And the format appears to just be the inner <CodeCoverage> part of the .runsettings format

I have now got filtering to work but needed to copy over all the filter items from the default config as they now don't get loaded, so I ended up using a config like the following:

<CodeCoverage>

  <ModulePaths>
    <Exclude>
    <ModulePath>.*\\unittests.dll</ModulePath>
    </Exclude>
  </ModulePaths>

  <Sources>
    <Exclude>
      <!--extracted from default CodeCoverage.config -->
      <Source>.*\\atlmfc\\.*</Source>
      <Source>.*\\vctools\\.*</Source>
      <Source>.*\\public\\sdk\\.*</Source>
      <Source>.*\\externalapis\\.*</Source>
      <Source>.*\\microsoft sdks\\.*</Source>
      <Source>.*\\vc\\include\\.*</Source>
    </Exclude>
  </Sources>

  <Functions>
    <Exclude>
      <!--extracted from default CodeCoverage.config -->
      <Function>^std::.*</Function>
      <Function>^ATL::.*</Function>
      <Function>.*::__GetTestMethodInfo.*</Function>
      <Function>.*__CxxPureMSILEntry.*</Function>
      <Function>^Microsoft::VisualStudio::CppCodeCoverageFramework::.*</Function>
      <Function>^Microsoft::VisualStudio::CppUnitTestFramework::.*</Function>
      <Function>.*::YOU_CAN_ONLY_DESIGNATE_ONE_.*</Function>
    </Exclude>
  </Functions>

</CodeCoverage>

and command line:

CodeCoverage collect /output:coverage.dat /config:coverage.settings vstest.console unitTests.dll /Logger:trx /Settings:test.runsettings