Reliably reconnect to MongoDB

Question

UPDATE: I am using the 2.1 version on the driver, against 3.2

I have a node application that uses MongoDB. The problem I have is that if the MongoDB server goes down for any reason, the application doesn't reconnect. To get this right, I based my tests on the code in this official tutorial.

var MongoClient = require('mongodb').MongoClient
  , f = require('util').format;

MongoClient.connect('mongodb://localhost:27017/test', 

// Optional: uncomment if necessary
// { db: { bufferMaxEntries: 3 } },


function(err, db) {
  var col = db.collection('t');

  setInterval(function() {
    col.insert({a:1}, function(err, r) {
      console.log("insert")
      console.log(err)

      col.findOne({}, function(err, doc) {
        console.log("findOne")
        console.log(err)
      });
    })
  }, 1000)
});

The idea is to run this script, and then stop mongod, and then restart it. So, here we go:

TEST 1: stopping mongod for 10 seconds

Stopping MongoDb for 10 seconds does the desired result: it will stop running the queries for those 10 seconds, and then will run all of them once the server is back ip

TEST 2: stopping mongod for 30 seconds

After exactly 30 seconds, I start getting:

{ [MongoError: topology was destroyed] name: 'MongoError', message: 'topology was destroyed' }
insert

{ [MongoError: topology was destroyed] name: 'MongoError', message: 'topology was destroyed' }

The trouble is that from this on, when I restart mongod, the connection is not re-establised.

Solutions?

Does this problem have a solution? If so, do you know what it is? Once my app starts puking "topology was destroyed", the only way to get everything to work again is by restarting the whole app...

Answer 1

By default the Mongo driver will try to reconnect 30 times, one every second. After that it will not try to reconnect again.

You can set the number of retries to Number.MAX_VALUE to keep it reconnecting "almost forever":

    var connection = "mongodb://127.0.0.1:27017/db";
    MongoClient.connect(connection, {
      server : {
        reconnectTries : Number.MAX_VALUE,
        autoReconnect : true
      }
    }, function (err, db) {

    });

Answer 2

package.json: "mongodb": "3.1.3"

Reconnect existing connections

To fine-tune the reconnect configuration for pre-established connections, you can modify the reconnectTries/reconnectInterval options (default values and further documentation here).

Reconnect initial connection

For the initial connection, the mongo client does not reconnect if it encounters an error (see below). I believe it should, but in the meantime, I've created the following workaround using the promise-retry library (which uses an exponential backoff strategy).

const promiseRetry = require('promise-retry')
const MongoClient = require('mongodb').MongoClient

const options = {
  useNewUrlParser: true,
  reconnectTries: 60,
  reconnectInterval: 1000,
  poolSize: 10,
  bufferMaxEntries: 0
}

const promiseRetryOptions = {
  retries: options.reconnectTries,
  factor: 1.5,
  minTimeout: options.reconnectInterval,
  maxTimeout: 5000
}

const connect = (url) => {
  return promiseRetry((retry, number) => {
    console.log(`MongoClient connecting to ${url} - retry number: ${number}`)
    return MongoClient.connect(url, options).catch(retry)
  }, promiseRetryOptions)
}

module.exports = { connect }

Mongo Initial Connect Error: failed to connect to server [db:27017] on first connect

Answer 3

Behavior may differ with different versions of driver. You should mention your driver version.

driver version : 2.2.10 (latest) mongo db version : 3.0.7

Below code will extend the time mongod can take to come back up.

var MongoClient = require('mongodb').MongoClient
  , f = require('util').format;

function connectCallback(err, db) {
  var col = db.collection('t');

  setInterval(function() {
    col.insert({a:1}, function(err, r) {
      console.log("insert")
      console.log(err)

      col.findOne({}, function(err, doc) {
        console.log("findOne")
        console.log(err)
      });
    })
  }, 1000)
}
var options = { server: { reconnectTries: 2000,reconnectInterval: 1000 }} 
MongoClient.connect('mongodb://localhost:27017/test',options,connectCallback);

2nd argument can be used to pass server options.

Answer 4

It's happening because it might have crossed the retry connection limit. After number of retries it destroy the TCP connection and become idle. So for it increase the number of retries and it would be better if you increase the gap between connection retry.

Use below options:

retryMiliSeconds {Number, default:5000}, number of milliseconds between retries.
numberOfRetries {Number, default:5}, number of retries off connection.

For more details refer to this link https://mongodb.github.io/node-mongodb-native/driver-articles/mongoclient.html

Solution:

MongoClient.connect("mongodb://localhost:27017/integration_test_?", {
    db: {
      native_parser: false,
retryMiliSeconds: 100000,
numberOfRetries: 100
    },
    server: {
      socketOptions: {
        connectTimeoutMS: 500
      }
    }
  }, callback)

Answer 5

If you was using Mongoose for your Schemas, it would be worth considering my option below since mongoose was never retrying to reconnect to mongoDB implicitly after first attempt failed.

Kindly note I am connecting to Azure CosmosDB for MongoDB API. On yours maybe on the local machine.

Below is my code.

const mongoose = require('mongoose');

// set the global useNewUrlParser option to turn on useNewUrlParser for every connection by default.
mongoose.set('useNewUrlParser', true);
// In order to use `findOneAndUpdate()` and `findOneAndDelete()`
mongoose.set('useFindAndModify', false);

async function mongoDbPool() {
// Closure.
return function connectWithRetry() {
    // All the variables and functions in here will Persist in Scope.
    const COSMODDBUSER = process.env.COSMODDBUSER;
    const COSMOSDBPASSWORD = process.env.COSMOSDBPASSWORD;
    const COSMOSDBCONNSTR = process.env.COSMOSDBCONNSTR;

    var dbAuth = {
        auth: {
            user: COSMODDBUSER,
            password: COSMOSDBPASSWORD
        }
    };
    const mongoUrl = COSMOSDBCONNSTR + '?ssl=true&replicaSet=globaldb';

    return mongoose.connect(mongoUrl, dbAuth, (err) => {
        if (err) {
            console.error('Failed to connect to mongo - retrying in 5 sec');
            console.error(err);
            setTimeout(connectWithRetry, 5000);
        } else {
            console.log(`Connected to Azure CosmosDB for MongoDB API.`);
        }
    });
};}

You may decide to export and reuse this module everywhere you need to connect to db via Dependency Injection. But instead I will only show how to access the database connection for now.

(async () => {
    var dbPools = await Promise.all([mongoDbPool()]);
    var mongoDbInstance = await dbPools[0]();

    // Now use "mongoDbInstance" to do what you need.
})();

Answer 6

There are 2 connection options that control how mongo nodejs driver reconnects after connection fails

• reconnectTries: attempt to reconnect #times (default 30 times)
• reconnectInterval: Server will wait # milliseconds between retries (default 1000 ms)

reference on mongo driver docs

Which means that mongo will keep trying to connect 30 times by default and wait 1 second before every retry. Which is why you start seeing errors after 30 seconds.

You should tweak these 2 parameters based on you needs like this sample.

var MongoClient = require('mongodb').MongoClient,
    f = require('util').format;

MongoClient.connect('mongodb://localhost:27017/test', 
    {
        // retry to connect for 60 times
        reconnectTries: 60,
        // wait 1 second before retrying
        reconnectInterval: 1000
    },

    function(err, db) {
        var col = db.collection('t');

        setInterval(function() {
            col.insert({
                a: 1
            }, function(err, r) {
                console.log("insert")
                console.log(err)

                col.findOne({}, function(err, doc) {
                    console.log("findOne")
                    console.log(err)
                });
            })
        }, 1000)
    });

This will try 60 times instead of the default 30, which means that you'll start seeing errors after 60 seconds when it stops trying to reconnect.

Sidenote: if you want to prevent the app/request from waiting until the expiration of the reconnection period you have to pass the option bufferMaxEntries: 0. The price for this is that requests are also aborted during short network interruptions.