WebRequest.Create - The operation has timed out

问题:

I'm trying to crawl a couple of pages on my own site, but I'm getting a time-out webException("The operation has timed out") on my live environment but not on my test environment. The time-out does not occur on the same page twice, but randomly and often after some requests. After the first time-out, the frequency of the time-outs rises.

The requestUristring on test enviroment: http://localhost/Opgaver/Flytning/Haarde-hvidevarer/Bortkoersel-amerikaner-koeleskab-paa.aspx

The requestUristring on live enviroment: http://www.servicebyen.dk/Opgaver/Flytning/Haarde-hvidevarer/Bortkoersel-amerikaner-koeleskab-paa.aspx

var webRequest = (HttpWebRequest)WebRequest.Create(requestUriString);
webRequest.KeepAlive = false;
webRequest.Timeout = 3 * 30 * 1000;
webRequest.PreAuthenticate = false;

using (WebResponse webResponse = webRequest.GetResponse()) //ERROR OCCURS HERE
{

    using (Stream responseStream = webResponse.GetResponseStream())
    {
         if (responseStream != null)
         {
              using (var reader = new StreamReader(responseStream))
              {
                   string readToEnd = reader.ReadToEnd();
                   .....          
              }

         }                
     }
 }

回答1:

I would try setting webRequest.ServicePoint.ConnectionLimit = 100.

回答2:

Try this (but only for your test environment).

Keep-alive: true if the request to the Internet resource should contain a Connection HTTP header with the value

webRequest.KeepAlive = true;
webRequest.Timeout = 95 * 95 * 100000;