How to stop MsTest tests execution after *n* faile

Question

I want to run unit tests via MS Test (from windows console) in a way that I can stop/kill the test execution whenever the failed tests count exceeds certain threshold value.

For my use case there is no point to keep running tests when certain percentage of the tests already failed.

I can only think in creating a new console app to wrap the mstest.exe execution, so I can parse the standard output in real-time, and eventually kill the process, for example:

var pi = new ProcessStartInfo()
{
    FileName = MS_TEST,
    UseShellExecute = false,
    RedirectStandardOutput = true,
    Arguments = MS_TEST_ARGS
};
int passed = 0; int failed = 0;
using (var process = Process.Start(pi))
{
    while (!process.StandardOutput.EndOfStream)
    {
        string line = process.StandardOutput.ReadLine();
        if (line.Contains("Passed"))
            passed++;
        if (line.Contains("Failed"))
            failed++;
        if (failed >= THRESHOLD)
        {
            process.Kill();
            break;
        }
    }
}

Can anyone suggest a better way for doing this? I don't think this is natively supported by MsTest.

PowerShell seems to be an option, but the stdout redirect is not trivial.

Update

As a note, I cannot modify the test code, I need this to be done without modifying the tests code in any way.

Answer 1

Create a BaseTestClass which contains a method responsible for killing the process that runs the tests.

using System.Diagnostics;

namespace UnitTestProject1
{
    public class BaseTestClass
    {
        private readonly int _threshold = 1;
        private static int _failedTests;

        protected void IncrementFailedTests()
        {
            if (++_failedTests >= _threshold)
                Process.GetCurrentProcess().Kill();
        }
    }
}

Your must inherit all your test classes from BaseTestClass and use the [TestCleanup] attribute. The TestCleanup() method is evaluated when a test defined in the DemoTests class has finished running. Is in that method where we evaluate the output of the test that has just finished. If it failed, we kill the process responsible for running the tests.

In the following example we have defined three tests. The second test, Test_Substracting_Operation(), is intended to fail intentionally, so the third test will never be run.

using Microsoft.VisualStudio.TestTools.UnitTesting;

namespace UnitTestProject1
{
    [TestClass]
    public class DemoTests : BaseTestClass
    {
        public TestContext TestContext { get; set; }

        [TestCleanup]
        public void TestCleanup()
        {
            if (TestContext.CurrentTestOutcome == UnitTestOutcome.Failed)
            {
                IncrementFailedTests();
            }
        }
        [TestMethod]
        public void Test_Adding_Operation()
        {
            // Arrange
            int x = 1;
            int y = 2;

            // Act
            int result = x + y;

            // Assert
            Assert.AreEqual(3, result);
        }

        [TestMethod]
        public void Test_Substracting_Operation()
        {
            // Arrange
            int x = 1;
            int y = 2;

            // Act
            int result = x - y;

            // Assert
            Assert.AreEqual(100, result);
        }

        [TestMethod]
        public void Test_Multiplication_Operation()
        {
            // Arrange
            int x = 1;
            int y = 2;

            // Act
            int result = x * y;

            // Assert
            Assert.AreEqual(2, result);
        }
    }
}