Run x number of web requests simultaneously

问题:

Our company has a web service which I want to send XML files (stored on my drive) via my own HTTPWebRequest client in C#. This already works. The web service supports 5 synchronuous requests at the same time (I get a response from the web service once the processing on the server is completed). Processing takes about 5 minutes for each request.

Throwing too many requests (> 5) results in timeouts for my client. Also, this can lead to errors on the server side and incoherent data. Making changes on the server side is not an option (from different vendor).

Right now, my Webrequest client will send the XML and wait for the response using result.AsyncWaitHandle.WaitOne();

However, this way, only one request can be processed at a time although the web service supports 5. I tried using a Backgroundworker and Threadpool but they create too many requests at same, which make them useless to me. Any suggestion, how one could solve this problem? Create my own Threadpool with exactly 5 threads? Any suggestions, how to implement this?

回答1:

The easy way is to create 5 threads ( aside: that's an odd number! ) that consume the xml files from a BlockingCollection.

Something like:

var bc = new BlockingCollection<string>();

for ( int i = 0 ; i < 5 ; i++ )
{
    new Thread( () =>
        {
            foreach ( var xml in bc.GetConsumingEnumerable() )
            {
                // do work
            }
        }
    ).Start();
}

bc.Add( xml_1 );
bc.Add( xml_2 );
...
bc.CompleteAdding(); // threads will end when queue is exhausted

回答2:

If you're on .Net 4, this looks like a perfect fit for Parallel.ForEach(). You can set its MaxDegreeOfParallelism, which means you are guaranteed that no more items are processed at one time.

Parallel.ForEach(items,
                 new ParallelOptions { MaxDegreeOfParallelism = 5 },
                 ProcessItem);

Here, ProcessItem is a method that processes one item by accessing your server and blocking until the processing is done. You could use a lambda instead, if you wanted.

回答3:

Creating your own threadpool of five threads isn't tricky - Just create a concurrent queue of objects describing the request to make, and have five threads that loop through performing the task as needed. Add in an AutoResetEvent and you can make sure they don't spin furiously while there are no requests that need handling.

It can though be tricky to return the response to the correct calling thread. If this is the case for how the rest of your code works, I'd take a different approach and create a limiter that acts a bit like a monitor but allowing 5 simultaneous threads rather than only one:

private static class RequestLimiter
{
  private static AutoResetEvent _are = new AutoResetEvent(false);
  private static int _reqCnt = 0;
  public ResponseObject DoRequest(RequestObject req)
  {
    for(;;)
    {
      if(Interlocked.Increment(ref _reqCnt) <= 5)
      {
        //code to create response object "resp".
        Interlocked.Decrement(ref _reqCnt);
        _are.Set();
        return resp;
      }
      else
      {
          if(Interlocked.Decrement(ref _reqCnt) >= 5)//test so we don't end up waiting due to race on decrementing from finished thread.
           _are.WaitOne();
      }
    }
  }
}

回答4:

You could write a little helper method, that would block the current thread until all the threads have finished executing the given action delegate.

static void SpawnThreads(int count, Action action)
{
    var countdown = new CountdownEvent(count);

    for (int i = 0; i < count; i++)
    {
        new Thread(() =>
        {
            action();
            countdown.Signal();
        }).Start();
    }

    countdown.Wait();
}

And then use a BlockingCollection<string> (thread-safe collection), to keep track of your xml files. By using the helper method above, you could write something like:

static void Main(string[] args)
{
    var xmlFiles = new BlockingCollection<string>();

    // Add some xml files....

    SpawnThreads(5, () =>
    {
        using (var web = new WebClient())
        {
            web.UploadFile(xmlFiles.Take());
        }
    });

    Console.WriteLine("Done");
    Console.ReadKey();
}

Update

An even better approach would be to upload the files async, so that you don't waste resources on using threads for an IO task.

Again you could write a helper method:

static void SpawnAsyncs(int count, Action<CountdownEvent> action)
{
    var countdown = new CountdownEvent(count);

    for (int i = 0; i < count; i++)
    {
        action(countdown);
    }

    countdown.Wait();
}

And use it like:

static void Main(string[] args)
{
    var urlXML = new BlockingCollection<Tuple<string, string>>();
    urlXML.Add(Tuple.Create("http://someurl.com", "filename"));

    // Add some more to collection...

    SpawnAsyncs(5, c =>
    {
        using (var web = new WebClient())
        {
            var current = urlXML.Take();

            web.UploadFileCompleted += (s, e) =>
            {
                // some code to mess with e.Result (response)
                c.Signal();
            };

            web.UploadFileAsyncAsync(new Uri(current.Item1), current.Item2);
        }
    });

    Console.WriteLine("Done");
    Console.ReadKey();
}