Python task queue alternatives and frameworks

Question

There seem to be different implementations of task/job queues for Python 3:

1. Celery, popular but apparently unmaintained and stale;
2. RQ, of which I have little information;
3. TaskTiger, similarly to RQ I know little about it;
4. Huey , similarly to RQ I know little about it;
5. WorQ had its last update in 2016.

Then there are “cloud” based solutions like Google’s Task Queue API or AWS’s Cloud Watch Events, but that’s more of a last resort.

For my project I am looking for a stable and active task queue implementation. I’ve used Celery for the past year, but the lack of support and non-attention to existing bugs is worrisome.

What alternatives exist?

Answer 1

RabbitMQ guys recommend Pika library: https://github.com/pika/pika You can find simple and intuitive examples on their website: https://www.rabbitmq.com/tutorials/tutorial-one-python.html

Answer 2

One new alternative is django-carrot, which came about due to our requirement for a lightweight alternative to Celery, while still providing support for RabbitMQ (I think the other alternatives you've mentioned are all based on Redis)

Django-carrot is still under development but is expected to go into production at my company in the next few weeks

Disclaimer - I'm the author of django-carrot

Answer 3

I wrote Dramatiq specifically because of my increasing frustration with Celery having used it professionally for years. Check out the motivation page the "why" and a feature comparison between Dramatiq, Celery and RQ. Some highlights:

• actively developed and used in production
• great docs
• automatic retries
• code auto-reload
• locks and rate limiting
• redis and rabbitmq support