WebClient is very slow

Question

I have problem with Webclient.

It is very slow. It takes about 3-5 seconds to downloadString from one website. I don't have any network problems.

This is my Modifed WebClient.

using System;
using System.Collections.Generic;
using System.Linq;
using System.Text;
using System.Net;

namespace StatusChecker
{
    class WebClientEx: WebClient
    {
        public CookieContainer CookieContainer { get; private set; }

        public WebClientEx()
        {
            CookieContainer = new CookieContainer();

            ServicePointManager.Expect100Continue = false;
            Encoding = System.Text.Encoding.UTF8;

            WebRequest.DefaultWebProxy = null;
            Proxy = null;
        }

        public void ClearCookies()
        {
            CookieContainer = new CookieContainer();
        }

        protected override WebRequest GetWebRequest(Uri address)
        {

            var request = base.GetWebRequest(address);
            if (request is HttpWebRequest)
            {
                (request as HttpWebRequest).CookieContainer = CookieContainer;
            }
            return request;
        }
    }
}

UPDATE: In wireshark I saw that single DownladString is sending and receiving few thousands packets.

Answer 1

There are a few options if it is related to the initial proxy settings being checked:

1. Disable the automatic proxy detection settings in Internet Explorer

2. Set the proxy to null:

   WebClient.Proxy = null

3. On application startup set the default webproxy to null:

   WebRequest.DefaultWebProxy = null;

In older .NET code instead of setting to null, you used to write this (but null is now preferred):

webclient.Proxy = GlobalProxySelection.GetEmptyWebProxy();

Answer 2

There may be two issues at hand here (that I've also noticed in my own programs previously):

• The first request takes an abnormally long time: This occurs because WebRequest by default detects and loads proxy settings the first time it starts, which can take quite a while. To stop this, simply set the proxy property (WebRequest.Proxy) to null and it'll bypass the check (provided you can directly access the internet)
• You can't download more than 2 items at once: By default, you can only have 2 simultaneous HTTP connections open. To change this, set ServicePointManager.DefaultConnectionLimit to something larger. I usually set this to int.MaxValue (just make sure you don't spam the host with 1,000,000 connections).

Answer 3

Maybe it will help somebody. Some web services support compression (gzip or other). So you can add Accept-Encoding header for your requests and then enable automatic decompression for web client instance. Chrome works in that way.