Concurrent queries to MongoDB using mgo ends in cl

Question

I have been checking several questions here in stackoverflow + some posts where people exposes examples of of how to manage sessions with mgo golang lib .

The point is that all the examples I've seen doesn't run too much concurrent queries at the same time. Increasing the number of concurrent operations ends in closed sockets. Here you can find the code I ran in order to reproduce this behaviour.

Concurrent queries to MongoDB using mgo ends in closed sockets.

Note that I'm just running 200 of concurrent queries, opening a new socket for each one. 200 is not a big number.

The errors I see are:

read tcp 127.0.0.1:59583->127.0.0.1:27018: read: connection reset by peer

Closed explicitly

How should I deal with this? I want to keep this concurrency level, even increase it at some point.

Answer 1

The result is the same with a Mongo database which is not in a docker container if I raise the number of insert to more than 5 000. The inserts with no session copy work ok and the inserts with session copy end up with mongo errors "Closed explicitly".

According to William Kennedy blog, the recommended way is indeed to make session copy. Otherwise, the operations are serialized with other go routines in the same socket.

If you count the number of sockets opened with mongoldb : while true ; do lsof -i | grep 27017 | wc -l ; sleep 0.5 ; done you will see that we have one socket open when we have no session copy. When session copy is used, the number of sockets can raise high number.

I think that an open socket limit is reached in mongo or in the system. ulimit can be used to check the limits of the shell environment.

Note that, by design, mgo doesn't allow any limit on session number. The idea is avoid having limits on the db access side and push the developer to introduce limitations at the entry point of his code: limit the number of http requests treated simultaneously, for instance.

Answer 2

I believe the error you're getting here is not because your code but the image docker you're running.

You can see the same issue right here. One solution is to search another linux image or try to reinstall it.

And Some notice on your code : clear(masterSession) this will panic for the first time since there are no collection and data at the first run. see at your code line 26.

I have tried your code in my local machine using docker-compose only running the mongodb image All works fine. so I can assume the problem is within your docker linux image.

Hope it helps