Patterns for asynchronous but sequential requests

Question

I have been writing a lot of NodeJS recently and that has forced me to attack some problems from a different perspective. I was wondering what patterns had developed for the problem of processing chunks of data sequentially (rather than in parallel) in an asynchronous request-environment, but I haven't been able to find anything directly relevant.

So to summarize the problem:

1. I have a list of data stored in an array format that I need to process.
2. I have to send this data to a service asynchronously, but the service will only accept a few at a time.
3. The data must be processed sequentially to meet the restrictions on the service, meaning making a number of parallel asynchronous requests is not allowed

Working in this domain, the simplest pattern I've come up with is a recursive one. Something like

function processData(data, start, step, callback){
  if(start < data.length){
    var chunk = data.split(start, step);
    queryService(chunk, start, step, function(e, d){
      //Assume no errors
      //Could possibly do some matching between d and 'data' here to
      //Update data with anything that the service may have returned
      processData(data, start+step, step, callback);
    });
  }
  else{
    callback(data);
  }
}

Conceptually, this should step through each item, but it's intuitively complex. I feel like there should be a simpler way of doing this. Does anyone have a pattern they tend to follow when approaching this kind of problem?

Answer 1

My first thought process would be to rely on object encapsulation. Create an object that contains all of the information about what needs to be processed and all of the relevant data about what has been processed and is being processed and the callback function will just call the 'next' function for the object, which will in turn start processing on the next piece of data and update the object. Essentially working like a n asynchronous for-loop.