Symfony2 Job Queue or Parallel Processing?

Question

Does anyone know how to run a number of processes in the background either through a job queue or parallel processing.

I have a number of maintenance updates that take time to run and want to do this in the background.

Answer 1

I would recomment Gearman server, it prooved quite stable, it's totally outside of Symfony2, and you have to have server up and running (don't know what your hosting options are), but it distribues jobs perfectly. In skiniest version, it just keeps all jobs in-memory, but you can configure it to use sqlite database as backup, so for any reason server reboots, or gearman deamon breaks, you can just start it again, and your jobs will be perserved. I konw it has been tested with very large loads (adding up 1k jobs per second), and it stood it's ground. It's probably more stable nowdays, I'm speaking from experience 2 yrs ago, where we offloaded some long-running tasks in ZF application to background processing via Gearman. It should be quite self-explanitory how it works from image below:

Answer 2

Take a look at enqueue libraty. There are a lot of transports (AMQP, STOMP, AmazonSQS, Redis, Filesystem, Doctrine DBAL and more) to choose from. Easy to use and feature rich. That would be enough for simple job queue, though if you need something more sophisticated look at enqueue/job-queue. It can run an exclusive job (only one job running at a given time) or a job with sub-jobs, or a job with something to do after it has been done.

Of course, there is a bundle for it

Answer 3

Checkout RabbitMq. It's the most popular option according to knpbundles.com

Answer 4

Take a look at http://github.com/mmoreram/rsqueue-bundle

Uses Redis as queue core and will be mantained.