Execute unit tests serially (rather than in parall

Question

I am attempting to unit test a WCF host management engine that I have written. The engine basically creates ServiceHost instances on the fly based on configuration. This allows us to dynamically reconfigure which services are available without having to bring all of them down and restart them whenever a new service is added or an old one is removed.

I have run into a difficulty in unit testing this host management engine, however, due to the way ServiceHost works. If a ServiceHost has already been created, opened, and not yet closed for a particular endpoint, another ServiceHost for the same endpoint can not be created, resulting in an exception. Because of the fact that modern unit testing platforms parallelize their test execution, I have no effective way to unit test this piece of code.

I have used xUnit.NET, hoping that because of its extensibility, I could find a way to force it to run the tests serially. However, I have not had any luck. I am hoping that someone here on SO has encountered a similar issue and knows how to get unit tests to run serially.

NOTE: ServiceHost is a WCF class, written by Microsoft. I don't have the ability to change it's behavior. Hosting each service endpoint only once is also the proper behavior...however, it is not particularly conducive to unit testing.

Answer 1

you can Use Playlist

right click on the test method -> Add to playlist -> New playlist

then you can specify the execution order, the default is, as you add them to the play list but you can change the playlist file as you want

Answer 2

I don't know the details, but it sounds like you might be trying to do integration testing rather than unit testing. If you could isolate the dependency on ServiceHost, that would likely make your testing easier (and faster). So (for instance) you might test the following independently:

• Configuration reading class
• ServiceHost factory (possibly as an integration test)
• Engine class that takes an IServiceHostFactory and an IConfiguration

Tools that would help include isolation (mocking) frameworks and (optionally) IoC container frameworks. See:

• http://www.mockobjects.com/
• http://www.hanselman.com/blog/ListOfNETDependencyInjectionContainersIOC.aspx

Answer 3

Maybe you can use Advanced Unit Testing. It allows you to define the sequence in which you run the test. So you may have to create a new cs file to host those tests.

Here's how you can bend the test methods to work in the sequence you want.

[Test]
[Sequence(16)]
[Requires("POConstructor")]
[Requires("WorkOrderConstructor")]
public void ClosePO()
{
  po.Close();

  // one charge slip should be added to both work orders

  Assertion.Assert(wo1.ChargeSlipCount==1,
    "First work order: ChargeSlipCount not 1.");
  Assertion.Assert(wo2.ChargeSlipCount==1,
    "Second work order: ChargeSlipCount not 1.");
  ...
}

Do let me know whether it works.

Answer 4

As stated above, all good unit tests should be 100% isolated. Using shared state (e.g. depending on a static property that is modified by each test) is regarded as bad practice.

Having said that, your question about running xUnit tests in sequence does have an answer! I encountered exactly the same issue because my system uses a static service locator (which is less than ideal).

By default xUnit 2.x runs all tests in parallel. This can be modified per-assembly by defining the CollectionBehavior in your AssemblyInfo.cs in your test project.

For per-assembly separation use:

using Xunit;
[assembly: CollectionBehavior(CollectionBehavior.CollectionPerAssembly)]

or for no parallelization at all use:

[assembly: CollectionBehavior(DisableTestParallelization = true)]

The latter is probably the one you want. More information about parallelisation and configuration can be found on the xUnit documentation.

Answer 5

Each test class is a unique test collection and tests under it will run in sequence, so if you put all of your tests in same collection then it will run sequentially.

In xUnit you can make following changes to achieve this:

Following will run in parallel:

namespace IntegrationTests
{
    public class Class1
    {
        [Fact]
        public void Test1()
        {
            Console.WriteLine("Test1 called");
        }

        [Fact]
        public void Test2()
        {
            Console.WriteLine("Test2 called");
        }
    }

    public class Class2
    {
        [Fact]
        public void Test3()
        {
            Console.WriteLine("Test3 called");
        }

        [Fact]
        public void Test4()
        {
            Console.WriteLine("Test4 called");
        }
    }
}

To make it sequential you just need to put both the test classes under same collection:

namespace IntegrationTests
{
    [Collection("Sequential")]
    public class Class1
    {
        [Fact]
        public void Test1()
        {
            Console.WriteLine("Test1 called");
        }

        [Fact]
        public void Test2()
        {
            Console.WriteLine("Test2 called");
        }
    }

    [Collection("Sequential")]
    public class Class2
    {
        [Fact]
        public void Test3()
        {
            Console.WriteLine("Test3 called");
        }

        [Fact]
        public void Test4()
        {
            Console.WriteLine("Test4 called");
        }
    }
}

For more info you can refer to this link

Answer 6

For .NET Core projects, create xunit.runner.json with:

{
  "parallelizeAssembly": false,
  "parallelizeTestCollections": false
}

Also, your project.json should contain

"buildOptions": {
  "copyToOutput": {
    "include": [ "xunit.runner.json" ]
  }
}