Celery - schedule periodic tasks starting at a spe

问题:

What is the best way to schedule a periodic task starting at specific datetime?

(I'm not using cron for this considering I've the need to schedule about a hundred remote rsyncs, where I compute the remote vs local offset and would need to rsync each path the second the logs are generated in each host.)

By my understanding the celery.task.schedules crontab class only allows specifying hour, minute, day of week. The most useful tip I've found so far was this answer by nosklo.

Is this the best solution? Am I using the wrong tool for the job?

回答1:

Celery seems like a good solution for your scheduling problem: Celery's PeriodicTasks have run time resolution in seconds.

You're using an appropriate tool here, but the crontab entry is not what you want. You want to use python's datetime.timedelta object; the crontab scheduler in celery.schedules has only minute resolution, but using timedelta's to configure the PeriodicTask interval provides strictly more functionality, in this case, per second resolution.

e.g. from the Celery docs

>>> from celery.task import tasks, PeriodicTask
>>> from datetime import timedelta
>>> class EveryThirtySecondsTask(PeriodicTask):
...     run_every = timedelta(seconds=30)
...
...     def run(self, **kwargs):
...         logger = self.get_logger(**kwargs)
...         logger.info("Execute every 30 seconds")

http://ask.github.com/celery/reference/celery.task.base.html#celery.task.base.PeriodicTask

class datetime.timedelta(days=0, seconds=0, microseconds=0, milliseconds=0, minutes=0, hours=0, weeks=0)

The only challenge here is that you have to describe the frequency with which you want this task to run rather than at what clock time you want it to run; however, I would suggest you check out the Advanced Python Scheduler http://packages.python.org/APScheduler/

It looks like Advanced Python Scheduler could easily be used to launch normal (i.e. non Periodic) Celery tasks at any schedule of your choosing using it's own scheduling functionality.

回答2:

I've recently worked on a task that involved Celery, and I had to use it for asynchronous operation as well as scheduled tasks. Suffice to say I resorted back to the old crontab for the scheduled task, although it calls a python script that spawns a separate asynchronous task. This way I have less to maintain for the crontab (to make the Celery scheduler run there needs some further setup), but I am making full use of Celery's asynchronous capabilities.