NodeJS and asynchronous hell

Question

I just came to this awful situation where I have an array of strings each representing a possibly existing file (e.g. var files = ['file1', 'file2', 'file3']. I need to loop through these file names and try to see if it exists in the current directory, and if it does, stop looping and forget the rest of the remaining files. So basically I want to find the first existing file of those, and fallback to a hard-coded message if nothing was found.

This is what I currently have:

var found = false;
files.forEach(function(file) {
  if (found) return false;

  fs.readFileSync(path + file, function(err, data) {
    if (err) return;

    found = true;
    continueWithStuff();
  });
});

if (found === false) {
  // Handle this scenario.
}

This is bad. It's blocking (readFileSync) thus it's slow.

I can't just supply callback methods for fs.readFile, it's not that simple because I need to take the first found item... and the callbacks may be called at any random order. I think one way would be to have a callback that increases a counter and keeps a list of found/not found information and when it reaches the files.length count, then it checks through the found/not found info and decides what to do next.

This is painful. I do see the performance greatness in evented IO, but this is unacceptable. What choices do I have?

Answer 1

Use async.waterfall for controlling the async call in node.js for example: by including async-library and use waterfall call in async:

 var async = require('async'); 
 async.waterfall( 
   [function(callback) 
     { 
       callback(null, taskFirst(rootRequest,rootRequestFrom,rootRequestTo, callback, res));
     },
     function(arg1, callback) 
     {
       if(arg1!==undefined )
       {
         callback(null, taskSecond(arg1,rootRequest,rootRequestFrom,rootRequestTo,callback, res));  
       }      
     }
   ])

Answer 2

Don't use sync stuff in a normal server environment -- things are single threaded and this will completely lock things up while it waits for the results of this io bound loop. CLI utility = probably fine, server = only okay on startup.

A common library for asynchronous flow control is https://github.com/caolan/async

async.filter(['file1','file2','file3'], path.exists, function(results){
    // results now equals an array of the existing files
});

And if you want to say, avoid the extra calls to path.exists, then you could pretty easily write a function 'first' that did the operations until some test succeeded. Similar to https://github.com/caolan/async#until - but you're interested in the output.

Answer 3

You can do this without third-party libraries by using a recursive function. Pass it the array of filenames and a pointer, initially set to zero. The function should check for the existence of the indicated (by the pointer) file name in the array, and in its callback it should either do the other stuff (if the file exists) or increment the pointer and call itself (if the file doesn't exist).

Answer 4

The async library is absolutely what you are looking for. It provides pretty much all the types of iteration that you'd want in a nice asynchronous way. You don't have to write your own 'first' function though. Async already provides a 'some' function that does exactly that.

https://github.com/caolan/async#some

async.some(files, path.exists, function(result) {
  if (result) {
    continueWithStuff();
  }
  else {
    // Handle this scenario
  }
});

If you or someone reading this in the future doesn't want to use Async, you can also do your own basic version of 'some.'

function some(arr, func, cb) {
  var count = arr.length-1;

  (function loop() {
    if (count == -1) {
      return cb(false);
    }
    func(arr[count--], function(result) {
      if (result) cb(true);
      else loop();
    });
  })();
}

some(files, path.exists, function(found) {
  if (found) { 
    continueWithStuff();   
  }
  else {
    // Handle this scenario
  }
});

Answer 5

(Edit: removed sync suggestion because it's not a good idea, and we wouldn't want anyone to copy/paste it and use it in production code, would we?)

If you insist on using async stuff, I think a simpler way to implement this than what you described is to do the following:

var path = require('path'), fileCounter = 0;

function existCB(fileExists) {
    if (fileExists) {
        global.fileExists = fileCounter;
        continueWithStuff();
        return;
    }
    fileCounter++;
    if (fileCounter >= files.length) {
        // none of the files exist, handle stuff
        return;
    }
    path.exists(files[fileCounter], existCB);
}

path.exists(files[0], existCB);