How to structure celery tasks

问题:

I have 2 types of task: async tasks and schedule tasks. So, here is my dir structure:

proj
  |
  -- tasks
      |
      -- __init__.py
      |
      -- celeryapp.py     => celery instance defined in this file.
      |
      -- celeryconfig.py
      |
      -- async
      |    |
      |    -- __init__.py
      |    |
      |    -- task1.py    => from proj.tasks.celeryapp import celery
      |    |
      |    -- task2.py    => from proj.tasks.celeryapp import celery
      |
      -- schedule
           |
           -- __init__.py
           |
           -- task1.py    => from proj.tasks.celeryapp import celery
           |
           -- task2.py    => from proj.tasks.celeryapp import celery

But when I run celery worker like below, it does not work. It can not accept the task from celery beat scheduler.

 $ celery worker --app=tasks -Q my_queue,default_queue

So, is there any best practice on multiple task files organization?

回答1:

Based on celery documentation you can import a structure of celery tasks like this:

For example if you have an (imagined) directory tree like this:

|
|-- foo
|    |-- __init__.py
|    |-- tasks.py
|
|-- bar
     |-- __init__.py
     |-- tasks.py

Then calling app.autodiscover_tasks(['foo', bar']) will result in the modules foo.tasks and bar.tasks being imported.

回答2:

Celery tasks can be async, sync or scheduled depends on its invocation

task.delay(arg1,arg2)       #will be async
task.delay(arg1,arg2).get() #will be sync
task.delay(arg1,arg2).get() #will be sync
task.apply_async(args = [arg1,arg2], {'countdown' : some_seconds}) #async with delay

There's a lot of invocations depending on your needs
However, you must start celery with -B flag to enable celery scheduler

$ celery worker --app=tasks -B -Q my_queue,default_queue

So the way you take to organize your tasks is something personal and it deppends on your project complexity, but I think that organize them by its type of synchronism wouldn't be the best option.

I've googled this topic and I haven't found any guide or advise, but I've read some cases that organize their task by their functionality.
I've followed this advise, because this isn't a pattern, in my projects. Here one example of how I did

your_app
    |
    -- reports
        |
        -- __init__.py
        -- foo_report.py
        -- bar_report.py
        -- tasks
            |
            -- __init__.py
            -- report_task.py
    -- maintenance
        |
        -- __init__.py
        -- tasks
            |
            -- __init__.py
            -- delete_old_stuff_task.py
    -- twitter
        |
        -- __init__.py
        -- tasks
            |
            -- __init__.py
            -- batch_timeline.py