MongoDB as a queue service?

Question

I would love to hear more about real application experience witn MongoDB as a queue service, if you used MongoDB for this purpose could you share your thoughts, as well as the environment in which it was used?

Answer 1

Here is my Python implementation of PubSub / queue It works by either a tailing cursor on capped collection or polling a normal collection. Used it a few projects where I wanted to simplify my stack with quite good results. Of course as somebody mentioned already until you reached the limits of the atomic findAndModify, but that can be taken care of by various technics

Answer 2

Here is a great article explaining how someone used mongoDB's replication oplog as a queue.

You can do the same with a different collection. The main piece of advice seems to be to use a capped collection — mongo drivers have efficient means of waiting on a capped collection so that the client isn't constantly polling.

Answer 3

I am using mongodb as a queue service for email sending. Soon it will work in following way:

1. When a new message comes I store it in the mongodb.
2. A background job then loads the message from mongodb via the atomic operation findAndModify and sets the flag Processing to true, so it does not process same message twice (because my background job runs multiple threads in parallel).
3. Once the email has been sent I remove the document from mongodb.
4. You can also keep count of the failures for each message and remove it after 3 failed attempts.

In general I use mongodb as a queue service only for one reason: because I need to send emails by specified schedule (each message contains information on what time it should be sent).

If you do not have any schedule and need to process message immediately, I suggest that you look into existing queue services, because they probably handle all cases that you may not see without a deeper understanding of message queues.

Update

When background job crashes during message processing you could do following:

1. Move this message to another, message queue errors collection or..

2. Increase processing attempts counter in a message and again assign status "New", to try process it again. Just make sure that background job is idempotent (can process same message multiple times and not corrupt data) and transactional (when job fails you must undone changes that was made. if any). When job fails after 5 attempts (config value) perform #1.

3. Once bug with message processing was fixed you could process it again once more by assigning "New" status and moving to the message queue, or just delete this message. It depends on business processes actually.

Answer 4

Here is a simple message queue implementation.

It is a part of article that evaluates performance of variety of message queue systems.

A single-thread, single-node setup achieves 7 900 msgs/s sent and 1 900 msgs/s received.

Answer 5

I know that this question is back from 2012, but during my own research i found this article and just want to inform any other user that the devs from serverdensity replaced rabbitmq in favor of a simple queueing system with mongodb.

A detailed article is given here:

https://blog.serverdensity.com/replacing-rabbitmq-with-mongodb/

Answer 6

I've searched a lot and found the JavaScript version https://github.com/chilts/mongodb-queue. But I want a go version, so a simple implementation in Go, including a manager to poll messages was made: https://github.com/justmao945/mongomq