Doctrine ODM / MongoDB not retrying queries?

Question

I'm using Doctrine ODM to connect to MongoDB. I have a three node replica set: two fulls and one arbiter. Replication is solely for higher availability, I don't seek to distribute reads across nodes. My application is logging MongoCursorExceptions every so often with the message not master and slaveok=false. I don't see any evidence that a failover event occurred in the mongodb logs and the primary didn't change. CORRECTION: failover had indeed occurred, but the exception with not master and slaveok=false was appearing often, even showing up 6 hours after a new primary was successfully elected.

What to do? I see our version of doctrine-mongodb includes (experimental?) retry functionality, but I don't see an easy way to enable that.

Not sure if it matters, but this is a Symfony2 (v2.0) app.

https://groups.google.com/d/topic/mongodb-user/6p710Rdycpg/discussion implies that we need retries (emphasis mine):

Your application must be written to reconnect/retry since there are any number of transient (network) errors which could come up much like the rolling upgrade process during normal operation.

The Mongo PHP extension docs seem to account for this:

The driver will automatically retry "plain" queries (not commands) a couple of times if the first attempt failed for certain reasons. This is to cause fewer exceptions during replica set failover (although you will probably still have to deal with some) and gloss over transient network issues.

And I thought doctrine-mongodb just used the PHP extension to actually talk to mongod. So I'm left a bit confused whether or not I should have to worry about retry configuration.

I think I solved part of the problem: I removed the arbiter from the connection string following this advice from Kristina Chodorow. I'm no longer seeing any MongoCursorExceptions with the message not master and slaveok=false. I might have been hitting https://jira.mongodb.org/browse/PHP-392.

However, I'm still getting a few MongoCursorExceptions with the message couldn't determine master during failover. For example, I just did a failover; based on the mongod logs, a new primary was elected after a few seconds, but the web application was throwing that exception even 5 minutes later.

Answer 1

Have you configured the replicaSet option to point to your designated replica set configured within MongoDB?

Symfony DoctrineMongoDBBundle Docs

Look at the full configuration at the bottom. As far as I know you need to name all the servers in the server config line AND name the replicaSet in the options section. The documentation on that page doesn't make that very clear.

Looking through the code. It looks like the retry system should be configured by setting.

retryConnect
retryQuery

At the same level as the database name.

This is an example of my Zend Framework config which essentially does the same thing.

; Database details
resources.odm.configuration.defaultDB = "test"
resources.odm.connection.options.replicaSet = "test-replica-set"
resources.odm.connection.server = "mongodb://rs1a.testing.com:27017,rs1b.testing.com:27017,rs1c.testing.com:27017"
resources.odm.configuration.retryConnect = 3
resources.odm.configuration.retryQuery = 3

I was getting similar errors when I didn't have the replicaSet configured.

Answer 2

I assume you're using one of the tags (up to 2.2.1 at this time) corresponding to the 2.0 branch on DoctrineMongoDBBundle. Symfony 2.1+ compatibility is in tag versions 3+ of the bundle.

The configuration class exposes retry_query and retry_connect options, which default to zero, at the document manager level. In YML, the configuration would look like:

doctrine_mongodb:
    # Other proxy, hydrator and connection configuration options

    default_document_manager: dm1

    document_managers:
        dm1:
            retry_query: 1
            retry_connect: 1
            # Other mapping, metadata and DM options follow

I realize this wasn't mentioned in the bundle documentation, so I just added a section on it in this commit.