.NET HttpWebRequest Speed versus Browser

Question

I have a question regarding the performance of the .Net HttpWebRequest client (or WebClient, gives similar results).

If I use HttpWebRequest to request an html page (in this case news.bbc.co.uk) and analyse the speed (using HttpAnalyzer) at which the response is read by the application, this is significantly slower than a browser (Firefox, Chrome, IE) requesting the same resource (all caches cleared etc). The .Net application takes approximately 1.7 seconds versus 0.2 - 0.3 seconds for a browser.

Is this purely down to the speed and efficiency of the code / application or are there any other factors to consider?

Code as follows:

HttpWebRequest request = null;

Uri uriTest = new Uri("http://news.bbc.co.uk");

request = (HttpWebRequest)WebRequest.Create(uriTest);

request.Method = "GET";
request.KeepAlive = true;
request.Headers["Accept-Encoding"] = "gzip, deflate";

HttpWebResponse response = (HttpWebResponse)request.GetResponse();

response.Close();

Answer 1

The first time you request a page, .net tries to detect proxy settings. The solution is to pass in an empty WebProxy object. This way it just connects to remote server instead of autodetecting the proxy server.

HttpWebRequest request = (HttpWebRequest)WebRequest.Create(uriTest);
request.Proxy = new WebProxy();

Answer 2

What's the breakdown of that 1.7s? I suspect you are measuring the entire process?

Using this piece of code I get about 200ms in average:

var request = (HttpWebRequest)WebRequest.Create("http://www.bbc.co.uk/news/");

var stopwatch = new Stopwatch();
stopwatch.Start();

using (var response = (HttpWebResponse)request.GetResponse())
{
    stopwatch.Stop();
    Console.WriteLine("Elapsed: {0}ms", stopwatch.ElapsedMilliseconds);

    var responseStream = response.GetResponseStream();
    if (responseStream != null)
        using (var sr = new StreamReader(responseStream))
            Console.WriteLine("Title: {0}", Regex.Match(sr.ReadToEnd(), @"title>(.*)</title").Groups[1].Value);
}

Edit changed the code just to measure the actual HTTP request and tried again using Fiddler as well:

Program above: Elapsed: 78ms

Fiddler: Overall Elapsed: 00:00:00.0620000

Answer 3

Markos' answer worked perfectly for me for the same issue:

request.Proxy = new WebProxy();

Reduced a 16-second request to less than a second. Thanks!

Answer 4

I'd jack Fiddler in the middle, run the browser request and the .NET request one after the other and make sure you're really getting what you think. It's possible there's redirection or something else hinky going on (maybe browser is pre-appending the '/' while .NET waits for the redir, etc) that isn't immediately visible. I've built huge apps on the .NET HTTP client with nothing like what you describe- something else must be going on.

What happens if you stick '/' on the end of the URL?

Answer 5

Have you watched the network while using the browser? Perhaps the browser is using cached resources?

Answer 6

Whenever you measure anything, you have to account for the startup costs. If your .net code is in a single process,and you are only measuring the single request, then your measurement will be tainted by first time costs of initializing assemblies, types, etc.

As Darin and others have suggested, you should make sure that:

1) You are not running the process under debuggger. 2) You account for startup costs.

One way you can do #2, is to make two requests and only measure the second one. Or you can make N requests, discard the 1st one, and get the average of last N-1 requests. Also make sure that you read the entity stream.