.NET HttpWebRequest Speed versus Browser

Question

I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).

If I use HttpWebRequest to request an html page (in this case news.bbc.co.uk) and analyse the speed (using HttpAnalyzer) at which the response is read by the application, this is significantly slower than a browser (Firefox, Chrome, IE) requesting the same resource (all caches cleared etc). The .Net application takes approximately 1.7 seconds versus 0.2 - 0.3 seconds for a browser.

Is this purely down to the speed and efficiency of the code / application or are there any other factors to consider?

Code as follows:

HttpWebRequest request = null;

Uri uriTest = new Uri("http://news.bbc.co.uk");

request = (HttpWebRequest)WebRequest.Create(uriTest);

request.Method = "GET";
request.KeepAlive = true;
request.Headers["Accept-Encoding"] = "gzip, deflate";

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

response.Close();

Answer 1

If you make two requests does the second one happen more quickly?

I have also notice speed disparities between browsers and WebClient or WebRequest. Even the raw speed of the response can be drastically different - but not all the time!

There are a few things this could be caused by:

• It could be all the .Net bootstrapping that happens. .Net assemblies aren't loaded and JITted until they are used, therefore you can see significant speed degradation on the initial call to a piece of code even if the application itself has been running for ages. Okay - so the .Net framework itself is nGen'd - but there's still the bridge between your code and the .Net framework to build on the fly.

• Just checking that you're running without the debugger attached and that you definitely don't have symbol server switched on - symbol server and VS interrupts programs as the symbols are downloaded, slowing them down bucket-loads. Sorry if this is an insult ;)

• Browsers are coded to make efficient use of only a few underlying sockets; and they will be opened and primed as soon as the browser is there. 'Our' code that uses .Net WebClient/WebRequest is totally inefficient in comparison, as everything is initialised anew each time.

• There are a lot of platform resources associated with networking, and whilst .Net makes it much easier to code with networks, it's still bound to the same platform resource issues. Ergo, the closer you are to the platform you are, the faster some code will be. IE and Firefox et al are native and therefore can thrown around system resources natively; .Net isn't and therefore some marshalling(=slow) is required to set things up. Obviously, once a port is opened and being used, however, .Net is still no slouch; but it almost would never be as fast as well-written non-marshalled native code.

Answer 2

Run the application with Ctrl+F5 instead of F5 (Debug mode). You will see a difference:

class Program
{
    static void Main()
    {
        using (var client = new WebClient())
        {
            Stopwatch watch = Stopwatch.StartNew();
            var data = client.DownloadData("http://news.bbc.co.uk");
            watch.Start();
            Console.WriteLine("{0} ms", watch.ElapsedMilliseconds);
        }
    }
}

Prints 880 ms on my PC.

Answer 3

It maybe that bbc.co.uk checks the User-Agent header that is being passed to it and handles the response based on that. So if it sees automated clients then it responds slowly, where as if it believes that there is a real person at the end of the line then it speeds up. If you really want to try it out just tell the HttpWebRequest to pass a different header.