Adding extra celery configs to Airflow

Question

Anyone know where I can add extra celery configs to airflow celery executor? For instance I want http://docs.celeryproject.org/en/latest/userguide/configuration.html#worker-pool-restarts this property but how do I allow extra celery properties..

Answer 1

Use the just-released Airflow 1.9.0 and this is now configurable.

In airflow.cfg there is this line:

# Import path for celery configuration options
celery_config_options = airflow.config_templates.default_celery.DEFAULT_CELERY_CONFIG

which points to a python file from the import path. The current default version can is https://github.com/apache/incubator-airflow/blob/1.9.0/airflow/config_templates/default_celery.py

If you need a setting that isn't tweakable via that file then create a new module, say 'my_celery_config.py':

CELERY_CONFIG = {
    # ....
}

and put it in your AIRFLOW_HOME dir (i.e. along side the dags/ folder) and then set celery_config_options = my_celery_config.CELERY_CONFIG in the config.