Nodejs connect EMFILE - How to reuse connections?

Question

I am implementing NodeJS based scripts for communicating to couchbase and another service. It is a long running script and after a while I get "connect EMFILE" for the service. My code sample is given below:

function createContainer(chunkName,recordingID,chunkData)
{
  var swiftHTTPPath='http://'+swiftIPAddr+'/swift/v1/'+recordingID;
  var path = '/swift/v1/'+recordingID;
  var swiftOptions = {
    hostname : swiftIPAddr,
    port     : swiftPort,
    path     : path,
    method   : 'PUT',
  };

  http.get(swiftHTTPPath, function(res) {
    if(res.statusCode == '404')
    {
      var req = http.request(swiftOptions, function(res)
      {
        res.setEncoding('utf8');
        res.on('data', function (chunk)
        {
          console.log('Container created');
        });
        res.on('end', function() {
        });
      });
      req.on('error', function(e)
      {
        console.log(e.message);
      });
      req.end();
    }
    else
    {
      console.log('Container already exists');
    }
  }).on('error', function(e)
  {
    console.log('e.message);
  });
}

The following command shows there are more than 1024 open connections

lsof -i -n -P > mylog.log

........
node    14135 ubuntu 1020u  IPv4 5249347      0t0  TCP 10.1.1.1:53623->10.1.1.2:8091 (ESTABLISHED)
node    14135 ubuntu 1021u  IPv4 5249350      0t0  TCP 10.1.1.1:42021->10.1.1.2:11210 (ESTABLISHED)
node    14135 ubuntu 1022u  IPv4 5249351      0t0  TCP 10.1.1.1:53627->10.1.1.2:8091 (ESTABLISHED)
........

I can increase ulimit values but I would like to know am I doing the HTTP GET,POST request in the correct way. Is there a way to reuse/ terminate the connection ?

Answer 1

Since it looks like all (or most) of your requests go to a single host, you can set the maximum number of concurrent sockets per host to a limited number, such as 100. Do this with the agent.maxSockets option in Node's http module.

Regarding simply reusing connections, this should happen automatically. The fact that it isn't leads me to believe you are issuing many requests concurrently...so you could also solve this problem by issuing requests more gradually.