State of unit testing for Windows Phone

Question

I've been pushing my Google Fu to the limits trying to find the most recommended / stable setup for doing TDD + CI for Windows Phone applications. Can anyone who has successfully been doing this point me in the right direction?

Here's what I want to be able to do (if it's possible):

• Write unit tests for view models and application services that don't require phone functionality
• Execute tests directly in Visual Studio via Resharper or TD.NET
• Execute the unit tests from the command line with XML out, without launching the emulator
• Preferably be resiliant (as far as third party libraries go) to SDK updates

Since I'd like to keep this question as a resource to others looking for the same thing, here's what I'd prefer answers to avoid:

• Open source ports of projects that are either incomplete or abandoned
• Projects that are only available as an attachment on someone's blog

I'd also like to get full BDD-style acceptance tests going, but that's another issue entirely.

Answer 1

Update: Unit testing for Windows Phone 8 is officially supported now.

http://blogs.msdn.com/b/visualstudioalm/archive/2013/01/31/windows-phone-unit-tests-in-visual-studio-2012-update-2.aspx

Answer 2

I'm adding this answer as Community Wiki so that others may modify it to keep it up to date.

Unit Testing

Intent: To run isolated (no phone functionality), fast-executing tests often from both the IDE and Continuous Integration server without requiring the emulator (eg. TDD View Models)

The method I've seen recommended in a number of presentations involves referencing your source files in a .NET 4 project and running the tests against those (referencing the desktop equivalents of the assemblies). If your code doesn't use any APIs that are different to the desktop BCL and you can deal with keeping the reference project up to date (new files aren't added automatically) than that should be sufficient.

Otherwise, you can follow the steps below to execute code that references WP7 assemblies in the desktop CLR:

1. Add a tests Silverlight 4 Class Library (not a Silverlight for Windows Phone project) to your solution
2. In the class library, set Copy Local for all the framework references except mscorelib (basically System.* and Microsoft.*) to true
3. Add a reference from this test libary to NUnit.Silverlight.Framework.dll and NUnit.Silverlight.Compatibility.dll from the NUnit-Silverlight project
4. Add a reference from the test libary to your main Windows Phone project, ignoring the version warning.
5. Run the tests from in the IDE using ReSharper (tested) or TestDriven.NET
6. Run the tests from the command line using the standard .NET 2.0 nunit-console.exe from the latest NUnit release, passing in /framework=v4.0.

(Many of the above workarounds are required because WP7 uses SL3. Once Mango is released with the SL4 runtime, it should be a cleaner setup)

Integration Testing

Intent: To run longer-running tests that interact with resources external to the code (like phone features and web services) on the emulator, both on demand and on the CI server

UPDATE Unit Testing Windows Phone 8 applications will be official supported in Visual Studio 2012 Update 2, including VS integration and command line support. These tests run in the emulator, so I've included it under Integration Tests.

This is not currently supported by the WP7 port of the Silverlight test framework (and that only ships as a download from a blog).

In the meantime, I have created a codeplex project that adds an MSBuild task that launches the emulator and collates the results into an XML file. The simplest method installation is to add the wp7-ci NuGet package.

NOTE: Installing the WP7 SDK on Windows Server requires modifying the installer configuration and is not supported, but works well.

Acceptance/System Testing

Intent: To run end-to-end automated tests that interact with the phone's UI on the emulator, both on demand or on the CI server

Expensify's (poorly named for SEO) Windows Phone Test Framework supports writing UI automation tests from a host PC using SpecFlow.

Answer 3

I just did a 16 min screencast on this very topic. I show how to get started with NUnit and resharper on WP7. I also show a few of the issues I ran across (nothing CI related btw)

http://toranbillups.com/blog/archive/2011/07/24/Test-Driving-My-Windows-Phone-7-App

Answer 4

Here is a link to someone that has managed to automate WP7 tests on the 'phone, to automatically deploy the app, run tests, and read results: http://justinangel.net/WindowsPhone7EmulatorAutomation

It makes use of the CoreCon API, which looked super interesting until I started playing with it, and discovered that most of the functions threw not-implemented exceptions - but there is enough there do run automated tests.

Answer 5

I think a matter of this has to due with how you write your tests.

• Eventually some code will have to touch some phone specific things. Those should be isolated as dependencies and faked out. I haven't figured out a proper way to get the unit tests to actually run within the phone itself. Unfortunately, those dependencies remain untested for me.
• Using NUnit for Silverlight allows you to get assertions: http://code.google.com/p/nunit-silverlight/
• I've used Resharper to run the unit tests without issue. Similarly, you can use the nunit-console to run the tests and get XML output.
• Ayende's Rhino Mocks for Silverlight work fine for mocking / stubbing dependencies.
• Continuous integration was a bit tricky. The WP7 SDK is not available for on the Server platform, so I built a new one on Windows 7 for my CI. There may be ways around that limitation, but I didn't bother.

The other tool you will want is MVVMLight. This will allow you to use EventTrigger and ICommand instead of events, since testing the events is significantly more work and can't be bound through the DataContext.

As far as how I designed my application:

The ViewModel can take in any number of dependencies, which get resolved using MicroIoC.

The actual code behind of the XAML resolved the ViewModel and sets it to the data context. This is unfortunate because that means you can't set the DataContext in XAML, but was a trade off I was willing to accept for dependency injection, like this:

public partial class SignUpPage
{
    public SignUpPage()
    {
        InitializeComponent();
        DataContext = IoC.Resolve<SignUpViewModel>();
    }
}

Fortunately, that's the only C# code that actually appears in my XAML code behind. From there, it's fairly regular MVVM with using binding and the DataContext.

Now you can test your ViewModel, inject the required dependencies (or fake them out) and it'll run fine without being in the emulator, so long as you don't try to use something WP7 specific.

Answer 6

Well, there are at least 3 different ways to do TDD for Windows Phone apps.

1. Silverlight Unit Test Framework - written by Jeff Wilcox. Here you can find its latest update. It seems it is most popular way, there is a lot of information about this approach:

   • First of all Jeff's screencast Unit Testing Silverlight and Windows Phone Applications
   • Unit Testing with Silverlight for Phone7
   • Windows Phone 7 – MVVM Light and Unit Testing Example
   • Good TDD kata - Building a Windows Phone 7 app with MVVM pattern, using TDD and mock objects
   • Unit Testing the Windows Phone 7 applications

   Also please check Visual Studio test project template so that you don't need to create test project manually. However, this solution requires running your tests on phone emulator what sometimes can be very annoying.

2. Portable Library Tools - a new Visual Studio add-in from Microsoft which enables you to create C# and Visual Basic libraries that run on a variety of .NET-based platforms without recompilation, including Windows Phone.

   Check this post to see how to do TDD for Windows Phone with Portable Library Tools. Here you can find a Visual Studio extension.

   The drawback of this approach is that this library has a limited assemblies support, thus most probably you cannot use everything that you have used to do. For example you cannot use commands (since ICommand lives in PresentationCore.dll), you cannot use MVVMLight as well, etc. From the other hand, it gives you more flexibility in your test project (you can use different mock frameworks,IoC containers run tests from VS, run tests with Resharper and so forth).

3. Silverlight NUnit Project VS template. Check this blog post to see how to use this template in your project. Also check the Toran Billups screencast that appeared in this thread. Be aware that you might have problems using Silverlight NUnit Project with some other frameworks or programs like Moq or Resharper.