How to get a Fast .Net Http Request

问题:

I need an Http request that I can use in .Net which takes under 100 ms. I'm able to achieve this in my browser so I really don't see why this is such a problem in code.

I've tried WinHTTP as well as WebRequest.Create and both of them are over 500ms which isn't acceptable for my use case.

Here are examples of the simple test I'm trying to pass. (WinHttpFetcher is a simple wrapper I wrote but it does the most trivial example of a Get Request that I'm not sure it's worth pasting.)

I'm getting acceptable results with LibCurlNet but if there are simultaneous usages of the class I get an access violation. Also since it's not managed code and has to be copied to bin directory, it's not ideal to deploy with my open source project.

Any ideas of another implementation to try?

    [Test]
    public void WinHttp_Should_Get_Html_Quickly()
    {
        var fetcher = new WinHttpFetcher();
        var startTime = DateTime.Now;          
        var result = fetcher.Fetch(new Uri("http://localhost"));
        var endTime = DateTime.Now;
        Assert.Less((endTime - startTime).TotalMilliseconds, 100);
    }
    [Test]
    public void WebRequest_Should_Get_Html_Quickly()
    {
        var startTime = DateTime.Now;
        var req = (HttpWebRequest) WebRequest.Create("http://localhost");
        var response = req.GetResponse();
        var endTime = DateTime.Now;
        Assert.Less((endTime - startTime).TotalMilliseconds, 100);
    }

回答1:

When benchmarking, it is best to discard at least the first two timings as they are likely to skew the results:

• Timing 1: Dominated by JIT overhead i.e. the process of turning byte code into native code.
• Timing 2: A possible optimization pass for the JIT'd code.

Timings after this will reflect repeat performance much better.

The following is an example of a test harness that will automatically disregard JIT and optimization passes, and run a test a set number of iterations before taking an average to assert performance. As you can see the JIT pass takes a substantial amount of time.

JIT:410.79ms

Optimize:0.98ms.

Average over 10 iterations:0.38ms

Code:

[Test]
public void WebRequest_Should_Get_Html_Quickly()
{
    private const int TestIterations = 10;
    private const int MaxMilliseconds = 100;

    Action test = () =>
    {
       WebRequest.Create("http://localhost/iisstart.htm").GetResponse();
    };

    AssertTimedTest(TestIterations, MaxMilliseconds, test);
}

private static void AssertTimedTest(int iterations, int maxMs, Action test)
{
    double jit = Execute(test); //disregard jit pass
    Console.WriteLine("JIT:{0:F2}ms.", jit);

    double optimize = Execute(test); //disregard optimize pass
    Console.WriteLine("Optimize:{0:F2}ms.", optimize);

    double totalElapsed = 0;
    for (int i = 0; i < iterations; i++) totalElapsed += Execute(test);

    double averageMs = (totalElapsed / iterations);
    Console.WriteLine("Average:{0:F2}ms.", averageMs);
    Assert.Less(averageMs, maxMs, "Average elapsed test time.");
}

private static double Execute(Action action)
{
    Stopwatch stopwatch = Stopwatch.StartNew();
    action();
    return stopwatch.Elapsed.TotalMilliseconds;
}

回答2:

Use the StopWatch class to get accurate timings.

Then, make sure you're not seeing the results of un-optimized code or JIT compilation by running your timing test several times in Release code. Discard the first few calls to remove he impact of JIT and then take the mean tidings of the rest.

VS.NET has the ability to measure performance, and you might also want to use something like Fiddler to see how much time you're spending "on the wire" and sanity check that it's not your IIS/web server causing the delays.

500ms is a very long time, and it's possible to be in the 10s of ms with these classes, so don't give up hope (yet).

Update #1:

This is a great article that talks about micro benchmarking and what's needed to avoid seeing things like JIT:

http://blogs.msdn.com/b/vancem/archive/2009/02/06/measureit-update-tool-for-doing-microbenchmarks.aspx

You're not quite micro-benchmarking, but there are lots of best practices in here.

Update #2:

So, I wrote this console app (using VS.NET 2010)...

class Program
{
    static void Main(string[] args)
    {
        var stopwatch = Stopwatch.StartNew();
        var req = (HttpWebRequest)WebRequest.Create("http://localhost");
        var response = req.GetResponse();
        Console.WriteLine(stopwatch.ElapsedMilliseconds);            
    }
}

... and Ctrl-F5'd it. It was compiled as debug, but I ran it without debugging, and I got 63ms. I'm running this on my Windows 7 laptop, and so http://localhost brings back the default IIS7 home page. Running it again I get similar times.

Running a Release build gives times in the 50ms to 55ms range.

This is the order of magnitude I'd expect. Clearly, if your website is performing an ASP.NET recompile, or recycling the app pool, or doing lots of back end processing, then your timings will differ. If your markup is massive then it will also differ, but none of the classes you're using client side should be the rate limiting steps here. It'll be the network hope and/or the remote app processing.

回答3:

Try setting the Proxy property of the HttpWebRequest instance to null. If that works, then try setting it to GlobalProxySelection.GetEmptyWebProxy(), which seems to be more correct.

You can read mor about it here: - WebRequest slow?: http://holyhoehle.wordpress.com/2010/01/12/webrequest-slow/

---

Update 2018: Pulling this up from the comments.

System.Net.GlobalProxySelection is obsolete.This class has been deprecated. Please use WebRequest.DefaultWebProxy instead to access and set the global default proxy. Use null instead of GetEmptyWebProxy(). – jirarium Jul 22 '17 at 5:44