Background processes in Node.js

Question

What is a good aproach to handle background processes in a NodeJS application?

Scenario: After a user posts something to an app I want to crunch the data, request additional data from external resources, etc. All of this is quite time consuming, so I want it out of the req/res loop. Ideal would be to just have a queue of jobs where you can quickly dump a job on and a daemon or task runner will always take the oldest one and process it.

In RoR I would have done it with something like Delayed Job. What is the Node equivalent of this API?

Answer 1

I'd like to suggest using Redis for scheduling jobs. It has plenty of different data structures, you can always pick one that suits better to your use case.

You mentioned RoR and DJ, so I assume you're familiar with sidekiq. You can use node-sidekiq for job scheduling if you want to, but its suboptimal imo, since it's main purpose is to integrate nodejs with RoR.

For worker daemonising I'd recommend using PM2. It's widely used and actively-maintained. It solves a lot of problems (e.g. deployment, monitoring, clustering) so make sure it won't be an overkill for you.

Answer 2

If you're using MongoDB, I recommend Agenda. That way, separate Redis instances aren't running and features such as scheduling, queuing, and Web UI are all present. Agenda UI is optional and can be run separately of course.

Would also recommend setting up a loosely coupled abstraction between your application logic and the queuing / scheduling system so the entire background processing system can be swapped out if needed. In other words, keep as much application / processing logic away from your Agenda job definitions in order to keep them lightweight.

Answer 3

If you want something lightweight, that runs in the same process as the server, I highly recommend Bull. It has a simple API that allows for a fine grained control over your queues.

If you're looking for something that runs as a standalone worker process, perhaps look into Kue. It can run as a RESTful API server, and even has several front-end apps written for it.

If you're familiar with Ruby's Resque, there is a node implementation called Node-resque

Bull, Kue and Node-resque are all backed by Redis, which is ubiquitous among Node.js worker queues. All 3 would be able to do what RoR's DelayedJob does, it's matter of specific features that you want, and your API preferences.

Answer 4

I tried bee-queue & bull and chose bull in the end. I first chose bee-queue b/c it is quite simple, their examples are easy to understand, while bull's examples are bit complicated. bee's wiki Bee Queue's Origin also resonates with me. But the problem with bee is <1> their issue resolution time is quite slow, their latest update was 10 months ago. <2> I can't find an easy way to pause/cancel job.

Bull, on the other hand, frequently updates their codes, response to issues. Node.js job queue evaluation said bull's weakness is "slow issues resolution time", but my experience is the opposite!

But anyway their api is similar so it is quite easy to switch from one to another.

Answer 5

I suggest to use a proper Node.js framework to build you app.

I think that the most powerful and easy to use is Sails.js.

It's a MVC framework so if you are used to develop in ROR, you will find it very very easy!

If you use it, It's already present a powerful (in javascript terms) job manager.

new sails.cronJobs('0 01 01 * * 0', function () {
   sails.log.warn("START ListJob");
}, null, true, "Europe/Dublin");

If you need more info not hesitate to contact me!

Answer 6

Background jobs are not directly related to your web service work, so they should not be in the same process. As you scale up, the memory usage of the background jobs will impact the web service performance. But you can put them in the same code repository if you want, whatever makes more sense.

One good choice for messaging between the two processes would be redis, if dropping a message every now and then is OK. If you want "no message left behind" you'll need a more heavyweight broker like Rabbit. Your web service process can publish and your background job process can subscribe.

It is not necessary for the two processes to be co-hosted, they can be on separate VMs, Docker containers, whatever you use. This allows you to scale out without much trouble.