We have the following setup:
- Team City v8.1.2
- .NET 2013 Solution with several different C# projects (ASP.NET MVC,
libraries, testing projects, etc.)
- 95% of our tests are MSpec, but we also have some NUnit tests.
- We have 14 test projects and growing...
Currently the testing phase is taking, roughly, 9 minutes for MSpec tests and 1 minute for NUnit. We would like to start parallelizing this, as we expect the number of tests to grow quickly. What would be the best solution, provided that:
- MSpec doesn't seem to have any parallel runner.
- The tests should be parallelized per assembly, which tests running sequentially within the same assembly.
- The parallelization shouldn't affect how we display the build results. We still want everything aggregated as it is right now.
- There shouldn't be any friction with the local execution in developers' machine. It would be required that, whatever the setup is, doesn't make running our tests locally more complicated. If the solution, whatever it is, allows us running the tests locally in parallel, that would be great.
- We prefer to scale making use of the parallel capabilities of each build agent, instead of running on multiple agents, because we pay per agent.
To run MSpec tests in parallel with TeamCity you can use mspec-teamcity-prunner.exe, which is a drop-in replacement for the standard MSpec runner.
Create/Edit a standard MSpec Build Step in TeamCity and then replace the path to mspec.exe with mspec-teamcity-prunner.exe
By default the tool will run up to 2 test assemblies in parallel, but this can be controlled by using --threads N) (e.g. --threads 4 for 4 threads) in the Additional command line parameters: field.
I'm not sure how you've configured the test part of your build: an MSBuild exec task, an Albacore mspec command, or the built-in TeamCity MSpec runner. No matter, they all run the mspec command line at some point. The command-line usage of mspec is to pass all the test assemblies to the test runner.
> mspec.exe [options] <assemblies>
The general solution then is to construct multiple mspec runs, each configured to run a single assembly.
> mspec.exe [options] <assmebly1>
> mspec.exe [options] <assembly2>
> mspec.exe [options] <assembly3>
Then you would run each of those tasks in parallel. I don't think TeamCity has the notion of parallel steps. MSBuild does not seem have general parallel tasks either, but you can fake it with an extension.
Other build systems do provide parallel tasks, like Rake's multitask. Provided you configure each individual task, you can mark the parent task to run all dependent tasks in parallel.
multitask :test => [:test_assmebly1, :test_assembly2, :test_assembly3]
