Excluding tests from tfs build

Question

I want to exclude some tests from my continuous integration build but I haven't found a way to do so.

One of the things I've tried was to set up the priority of those tests to -2 and then on the build I specified Minimum Test Priority = -1 but it still run those tests.

Any help would be greatly appreciated.

Answer 1

If you are using MSTest you can create a Test List for the tests that you need in you continuous integration.

Answer 2

With MSTest, you can simply create two test projects (assemblies) and only specify one in the build config to use for testing. In MSBuild, this was the way to go. For the new WF-Based build definitions, I currently don't have a sample at hand:

<ItemGroup>
    <!--  TEST ARGUMENTS
     If the RunTest property is set to true then the following test arguments will be used to run 
     tests. Tests can be run by specifying one or more test lists and/or one or more test containers.

     To run tests using test lists, add MetaDataFile items and associated TestLists here.  Paths can 
     be server paths or local paths, but server paths relative to the location of this file are highly 
     recommended:

        <MetaDataFile Include="$(BuildProjectFolderPath)/HelloWorld/HelloWorld.vsmdi">
            <TestList>BVT1;BVT2</TestList>
        </MetaDataFile>

     To run tests using test containers, add TestContainer items here:

        <TestContainer Include="$(OutDir)\AutomatedBuildTests.dll" />
        <TestContainer Include="$(SolutionRoot)\TestProject\WebTest1.webtest" />
        <TestContainer Include="$(SolutionRoot)\TestProject\LoadTest1.loadtest" />

     Use %2a instead of * and %3f instead of ? to prevent expansion before test assemblies are built
    -->
</ItemGroup>

<PropertyGroup>
    <RunConfigFile>$(SolutionRoot)\LocalTestRun.testrunconfig</RunConfigFile>
</PropertyGroup>

Tip: To use a generic build definition, we name all our Test projects "AutomatedBuildTests", i.e. there is no solution difference. So the build definition can be included in any existing build definition (or even be a common one) that always executes the right set of tests. It would be an easy task to prepend an "if exists" check in order to allow a build definition to only run tests when a Test assembly is present. We do not use this in order to get build errors when no test assembly is found as we absolutely want test with all those builds that use this definition.

Answer 3

Instead of using "Test Lists" that have been described, you should use the "Test Category" method. The test lists & VSMDI functionality have actually been deprecated in Visual Studio 2010 and Microsoft may remove the feature completely in a future version of Visual Studio.

If you'd like some more information about how to use test categories especially with your automated build process, check out this blog post: http://www.edsquared.com/2009/09/25/Test+Categories+And+Running+A+Subset+Of+Tests+In+Team+Foundation+Server+2010.aspx

You can also exclude test categories from running by specifying the ! (exclamation point) character in front of the category name to further define your filter.

Answer 4

In Visual Studio 2012 and later you can configure your build definition using the Test case filter setting.

This setting is part of your build definition. Open the build definition and navigate to the Process tab. In the section 3. Test you can define mutiple test sources. For each test source your can specify a Test case filter.

You can find the details in this MSDN article: Running selective unit tests in VS 2012 RC using TestCaseFilter

I have copied the supported operators and some examples from this article:

Operators supported in RC are:

1.= (equals)

2.!= (not equals)

3.~ (contains or substring only for string values)

4.& (and)

5.| (or)

6.( ) (paranthesis for grouping)

Expresssion can be created using these operators as any valid logical condition. & (and) has higher precedence over | (or) while evaluating expression.

E.g.
"TestCategory=NAR|Priority=1" "Owner=vikram&TestCategory!=UI" "FullyQualifiedName~NameSpace.Class"
"(TestCategory!=UI&(Priority=1|Priority=2))|(TestCategory=UI&Priority=1)"

Another possibility would be to have some test sources in one build definition in some (i.e. more or fewer) test sources in other build definitions.

Answer 5

My preference would be as above using a Test List, but some people have issued merging/editing the vsmdi files... We end up with separate solutions and use a pattern match to execute all tests in the appropriate DLL.