Can airflow be used to run a never ending task?

问题:

Can we use an airflow dag to define a never-ending job (ie. a task which has a unconditional loop to consume stream data) by setting the task/dag timeout to None and manually trigger its running? Would having airflow monitor a never ending task cause a problem? Thanks

回答1:

A bit odd to run this through Airflow, but yeah I don't think that's an issue. Just note that if you restart the worker running the job (assuming CeleryExecutor), you'll interrupt the task and need to kick it off manually again if retries is not set. Also make sure the DAG sets execution_timeout=None (default).

回答2:

You can use a sensor which monitors if "data exists", then triggers a dag which does some processing. Is this what you wanted? please let me know.