Can celery celerybeat use a Database Scheduler wit

Question

I have a small infrastructure plan that does not include Django. But, because of my experience with Django, I really like Celery. All I really need is Redis + Celery to make my project. Instead of using the local filesystem, I'd like to keep everything in Redis. My current architecture uses Redis for everything until it is ready to dump the results to AWS S3. Admittedly I don't have a great reason for using Redis instead of the filesystem. I've just invested so much into architecting this with Docker and scalability in mind, it feels wrong not to.

Answer 1

It turns out that you can!

First I created this little project from the tutorial on celeryproject.org.

That went great so I built a Dockerized demo as a proof of concept.

Things I learned from this project

• Docker

  • using --link to create network connections between containers
  • running commands inside containers

• Dockerfile

  • using FROM to build images iteratively
  • using official images
  • using CMD for images that "just work"

• Celery

  • using Celery without Django
  • using Celerybeat without Django
  • using Redis as a queue broker
  • project layout
  • task naming requirements

• Python

  • proper project layout for setuptools/setup.py
  • installation of project via pip
  • using entry_points to make console_scripts accessible
  • using setuid and setgid to de-escalate privileges for the celery deamon

Answer 2

I was searching for a non-Django database scheduler too a while back, but it looked like there's nothing else. So I took the Django scheduler code and modified it to use SQLAlchemy. Should be even easier to make it use Redis instead.