HttpWebRequest gets slower when adding an Interval

Question

Testing different possibilities to download the source of a webpage I got the following results (Average time in ms to google.com, 9gag.com):

• Plain HttpWebRequest: 169, 360
• Gzip HttpWebRequest: 143, 260
• WebClient GetStream : 132, 295
• WebClient DownloadString: 143, 389

So for my 9gag client I decided to take the gzip HttpWebRequest. The problem is, after implementing in my actual program, the request takes more than twice the time.
The Problem also occurs when just adding a Thread.Sleep between two requests.

EDIT:
Just improved the code a bit, still the same problem: When running in a loop the requests takes longer when I add an Delay between to requests

for(int i = 0; i < 100; i++)
{
    getWebsite("http://9gag.com/");
}

Takes about 250ms per request.

for(int i = 0; i < 100; i++)
{
    getWebsite("http://9gag.com/");
    Thread.Sleep(1000);
}

Takes about 610ms per request.

    private string getWebsite(string Url)
    {
        Stopwatch stopwatch = Stopwatch.StartNew();

        HttpWebRequest http = (HttpWebRequest)WebRequest.Create(Url);
        http.AutomaticDecompression = DecompressionMethods.GZip | DecompressionMethods.Deflate;
        string html = string.Empty;

        using (HttpWebResponse webResponse = (HttpWebResponse)http.GetResponse())
        using (Stream responseStream = webResponse.GetResponseStream())
        using (StreamReader reader = new StreamReader(responseStream))
        {

            html = reader.ReadToEnd();
        }

        Debug.WriteLine(stopwatch.ElapsedMilliseconds);
        return html;
    }

Any ideas to fix this problem?

Answer 1

Maybe give this a try, although it might only help your case of a single request and actually make things worse when doing a multithreaded version.

ServicePointManager.UseNagleAlgorithm = false;

Here's a quote from MSDN docs for the HttpWebRequest Class

Another option that can have an impact on performance is the use of the UseNagleAlgorithm property. When this property is set to true, TCP/IP will try to use the TCP Nagle algorithm for HTTP connections. The Nagle algorithm aggregates data when sending TCP packets. It accumulates sequences of small messages into larger TCP packets before the data is sent over the network. Using the Nagle algorithm can optimize the use of network resources, although in some situations performance can also be degraded. Generally for constant high-volume throughput, a performance improvement is realized using the Nagle algorithm. But for smaller throughput applications, degradation in performance may be seen.

An application doesn't normally need to change the default value for the UseNagleAlgorithm property which is set to true. However, if an application is using low-latency connections, it may help to set this property to false.

Answer 2

I think you might be leaking resources as you aren't disposing of all of your IDisposable object with each method call.

Give this version and try and see if it gives you a more consistent execution time.

public string getWebsite( string Url )
  {
     Stopwatch stopwatch = Stopwatch.StartNew();

     HttpWebRequest http = (HttpWebRequest) WebRequest.Create( Url );
     http.Headers.Add( HttpRequestHeader.AcceptEncoding, "gzip,deflate" );

     string html = string.Empty;
     using ( HttpWebResponse webResponse = (HttpWebResponse) http.GetResponse() )
     {
        using ( Stream responseStream = webResponse.GetResponseStream() )
        {
           Stream decompressedStream = null;

           if ( webResponse.ContentEncoding.ToLower().Contains( "gzip" ) )
              decompressedStream = new GZipStream( responseStream, CompressionMode.Decompress );
           else if ( webResponse.ContentEncoding.ToLower().Contains( "deflate" ) )
              decompressedStream = new DeflateStream( responseStream, CompressionMode.Decompress );

           if ( decompressedStream != null )
           {
              using ( StreamReader reader = new StreamReader( decompressedStream, Encoding.Default ) )
              {
                 html = reader.ReadToEnd();
              }

              decompressedStream.Dispose();
           }
        }
     }

     Debug.WriteLine( stopwatch.ElapsedMilliseconds );

     return html;
  }