Ordering Coded UI tests and Results in Test Manage

Question

I have a series of CodedUI test methods that make up a single test case. The test methods need to run in order (IE testmethoda runs then testmethodb and then testmethodc) and I want the results to show up in Microsoft Test Manager to look like testmethoda passed, testmethodb passed, testmethodc failed. Is there a way to do this while being able to run multiple iterations of the overall test case?

I have tried are putting the test methods into a single test method and calling that. This gives me the desired test order and the ability to make multiple test runs, but test manager shows a single pass/fail on the entire test case.

I have also tried attaching a datasource to the individual test methods and ordering them in test manager which gives me the desired test results in test manager but has the side effect that if i want to run more than one data row the order gets messed up. For example 3 data rows would run:

testmethoda
testmethoda
testmethoda

testmethodb
testmethodb
testmethodb

testmethodc
testmethodc
testmethodc

I want them to run:
testmethoda
testmethodb
testmeothdc

testmethoda
testmethodb
testmethodc etc..

I have thought about using an ordered test as well but that still shows up as a single test in MTM and there isn't a method I am aware of to data drive it anyways so it would have it's own problems.

Is there a feature that I am missing in VS or MTM to get these results? Maybe a method that would allow me to define a test run in the results file? Would writing/editing the trx file get my results into MTM? I have a feeling I would also have to make changes to the TFS database which isn't an option.

Answer 1

I don't think there is a way to do this through VS or MTM. The option to add all your test methods on a single one sounds good but when one of them fails then the 'parent' test method stops and throws the 'AssertFailedException' that one of the inner tests had thrown.

However, if your test methods (a, b, c...) are completely unaffected one by the other (this means that if the testMethodA fails the other tests can run without problems), I would try to catch all the internal exceptions and at the end print which methods passed and which not.

[TestClass]
public class TestClass
{
    Dictionary<string, string> testMethods;
    bool testResult;

    [TestInitialize]
    public void TestInitialize()
    {
        testMethods = new Dictionary<string, string>();
        testResult = true;
    }

    [TestMethod]
    public void TestMethod()
    {
        //Run TestMethodA
        try
        {
            TestMethodA();
            testMethods.Add("TestMethodA", "Passed");
        }
        catch (AssertFailedException exception) //Edit: better catch a generic Exception because CodedUI tests often fail when searcing for UI Controls 
        {
            testResult = false;
            testMethods.Add("TestMethodA", "Failed: " + exception.Message);
        }

        //Run TestMethodB
        try
        {
            TestMethodB();
            testMethods.Add("TestMethodB", "Passed");
        }
        catch (AssertFailedException exception)
        {
            testResult = false;
            testMethods.Add("TestMethodB", "Failed: " + exception.Message);
        }
    }

    [TestCleanup]
    public void TestCleanup()
    {
        foreach (KeyValuePair<string, string> testMethod in testMethods)
        {
            //Print the result for each test method
            TestContext.WriteLine(testMethod.Key.ToString() + " --> " + testMethod.Value.ToString());
        }

        //Assert if the parent test was passed or not.
        Assert.IsTrue(testResult, "One or more inner tests were failed.");
    }
}

You could also create a different class that will manage all this behaviour to avoid all these 'try-catch'.