Throttle WebRequests

问题:

I want to execute a bunch of WebRequests, but set a threshold on how many can be started simultaneously.

I came across this LimitedConcurrencyTaskScheduler example and tried to utilize it like so

scheduler = new LimitedConcurrencyLevelTaskScheduler(1);
taskFactory = new TaskFactory(scheduler);

...

    private Task<WebResponse> GetThrottledWebResponse(WebRequest request)
    {
        return taskFactory.FromAsync<WebResponse>(request.BeginGetResponse, request.EndGetResponse, null);
    }

However I noticed that even with a max concurrency of 1, my tasks seemed to be completing in a non-FIFO order. When I put breakpoints in LimitedConcurrencyLevelTaskScheduler, it became apparent that it's not being used at all. I guess the way I'm using TaskFactory.FromAsync is not doing what I had expected.

Is there a proper way to throttle simultaneous WebRequests?

回答1:

When I put breakpoints in LimitedConcurrencyLevelTaskScheduler, it became apparent that it's not being used at all

That is correct. FromAsync doesn't use the TaskFactory at all. In fact, I don't really understand why this method isn't static.

You have multiple ways to implement the throttling. You could use the ActionBlock From Microsoft.Tpl.Dataflow. Or you could make your own using SemaphoreSlim:

private static readonly SemaphoreSlim Semaphore = new SemaphoreSlim(1);

private static async Task<WebResponse> GetThrottledWebResponse(WebRequest request)
{
    await Semaphore.WaitAsync().ConfigureAwait(false);

    try
    {
        return await request.GetResponseAsync().ConfigureAwait(false);
    }
    finally
    {
        Semaphore.Release();
    }
}