WatiN or Selenium? [closed]

Question

I'm going to start coding some automated tests of our presentation soon. It seems that everyone recommends WatiN and Selenium. Which do you prefer for automated testing of ASP.NET web forms? Which of these products work better for you?

As a side note, I noticed that WatiN 2.0 has been in CTP since March 2008, is that something to be concerned about?

Answer 1

Until now we are a pure Microsoft Shop for delivering solutions for the enterprise and went with WatiN. This may change in the future.

As a more recent source:

Microsoft printed in MSDN Magazine 12/2010 a BDD-Primer with the combination of SpecFlow with WatiN (cool BDD-Behavior Driven Development). Its author Brandon Satrom (msft Developer Evangelist) has also posted in December 2010 a Video Webcast teaching in detail 1:1 his above findings.

There is a Whitepaper from 04/2011 on Supporting ATDD/BDD with SpecLog, SpecFlow and Team Foundation Server (Acceptance Test Driven Development/Behavior Driven Development) from Christian Hassa, whose team built SpecFlow.

Answer 2

I've been trying 'em both out and here are my initial thoughts...

---

WatiN

The Good

• Fast execution.
• Script creation tools are independent projects; there are 2 that I know of: Wax (Excel based, hosted on CodePlex) and WatiN Test Record (hosted on SourceForge). Neither is as robust as Selenium IDE.
• Very good IE support. Can attach and detach to/from running instances. Can access native window handles etc. (See script example below).
• NuGet packaged, easy to get running in .NET, Visual Studio style environments and keep updated.

The Bad

• Googling WatiN (watin xyz) often causes Google to recommend "watir xyz" instead. Not that much documentation out there.
• What little there is (documentation), it is confusing; for example: at first blush it would appear that there is no native support for CSS selectors. Especially since there are extensions libraries like 'WatiNCssSelectorExtensions' and many blog articles about alternative techniques (such as injecting jQuery/sizzle into the page). On Stack Overflow, I found a comment by Jeroen van Menen which suggests that there is native support. At least the lead-developer spends time on Stack Overflow :)
• No native XPath support.
• No out-of-the-box remote execution/grid based execution.

Script Example (C#). You can't do this with Selenium (not that I know off, at least):

class IEManager
{
    IE _ie = null;
    object _lock = new object();

    IE GetInstance(string UrlFragment)
    {
        lock (_lock)
        {
            if (_ie == null)
            {
                var instances = new IECollection(true);  //Find all existing IE instances
                var match = instances.FirstOrDefault(ie=>ie.Url.Contains(UrlFragment));
                _ie = match ?? new IE();
                if (match==null)  //we created a new instance, so we should clean it up when done!
                    _ie.AutoClose = true;
            }
        }

        return _ie;
    }
}

---

Selenium

• Slower than WatiN (especially since a new process has to be created).
• Built-in CSS selectors/XPath support.
• Selenium IDE is good (can't say great, but it’s the best in class!).
• Feels more Java-ish than .NET-ish...but really, it's programming language agnostic; all commands are sent to an out-of-process 'Driver'. The driver is really a 'host' process for the browser instance. All communication must be serialised in/out across process boundaries, which might explain the speed issues relative to WatiN.
• Decoupled processes - "Driver" and "Control" mean more robustness, more complexity, etc., but also easier to create grids/distributed test environments. Would have really liked it if the "distribution" mechanism (i.e. the communication between Driver & Control) were across WebSphere or other existing, robust, message queue manager.
• Support chrome and other browsers out of the box.

Despite everything, I went with WatiN in the end; I mainly intend to write small screen-scraping applications and want to use LINQPad for development. Attaching to a remote IE instance (one that I did not spawn myself) is a big plus. I can fiddle around in an existing instance...then run a bit of script...then fiddle again etc. This is harder to do with Selenium, though I suppose "pauses" could be embedded in the script during which time I could fiddle directly with the browser.

Answer 3

I generally use Selenium, mainly because I like the Selenium IDE plugin for FireFox for recording starting points for my tests.

Answer 4

Neither. Use Coypu. It wraps Selenium. Much more durable. https://github.com/featurist/coypu

Update Ye Oliver you're right. Ok why's it better? Personally I've found the Selenium driver for IE in particular to be very fragile - there's a number of 'standard' driver exceptions that I've time again found when driving Selenium for Unit Tests on ajax heavy websites.

Did I mention I want to write my scripts in c# as a Test Project ? Yes Acceptance Tests within a continous build deployment.

Well Coypu deals with the above. It's a wrapper for Selenium that allows test fixtures such as,

browser.Visit("file:///C:/users/adiel/localstuff.htm")
browser.Select("toyota").From("make");
browser.ClickButton("Search");

... which will spin up a (configurable brand of) browser and run the script. It works great with scoped regions and is VERY extendable.

There's more examples at GitHub and as Olvier below mentions, Adrian's video is excellent. I think it's the best way to drive browser based tests in the .Net world and tries to follow it's Ruby namesake capybara

Answer 5

The biggest difference is that Selenium has support for different browsers (not just IE or FF, see http://seleniumhq.org/about/platforms.html#browsers.

Also, Selenium has a remote control server (http://seleniumhq.org/projects/remote-control/), which means that you don't need to run the browser on the same machine the test code is running. You can therefore test your Web app. on different OS platforms.

In general I would recommend using Selenium. I've used WatiN few years ago, but I wasn't satisfied with its stability (it has probably improved by now). The biggest plus for Selenium for me is the fact that you can test the Web app. on different browsers.

Answer 6

Just want to say that I'm currently working hard on a beta release of WatiN 2.0 somewhere in Q1 of 2009. It will be a major upgrade to the current CTP 2.0 versions and will basically give you the same functionality to automate FireFox and IE as version 1.3.0 offers for automating IE.

So no concerns there.

Hope this helps in making your choice Jeroen van Menen Lead dev WatiN