Airflow Packaged Dag (Zip) not recognized

Question

I am trying to package my Repository with my Dag in a Zip file like it states here in the documentation. So i have followed the convention in the documentation, which is to keep the dag in the root of the zip, and the sub directories are viewed as packages by airflow.

My zip file has the following contents:

$ unzip -l $AIRFLOW_HOME/dags/test_with_zip.zip 
Archive:  /home/arjunc/Tutorials/airflow/dags/test_with_zip.zip
  Length      Date    Time    Name
---------  ---------- -----   ----
        0  2018-03-29 17:46   helloworld/
      189  2018-03-29 17:22   helloworld/hello.py
        0  2018-03-29 17:18   helloworld/__init__.py
      461  2018-03-29 17:24   test_with_zip_dag.py
---------                     -------
      650                     4 files

Where test_with_zip_dag.py is the file in the root directory with the Dag definitions as follows:

from datetime import datetime

from airflow import DAG
from airflow.operators.python_operator import PythonOperator

from helloworld.hello import HelloWorld


def run():
    return HelloWorld().run()


dag = DAG('test_with_zip', description='Test Dependencies With Zipping',
          schedule_interval='0 12 * * *',
          start_date=datetime(2017, 3, 20), catchup=False)

hello_operator = PythonOperator(task_id='hello_task', python_callable=run, dag=dag)

I have placed this zip in the default dags directory $AIRFLOW_HOME/dags, but my dag isn't recognized!

What am I doing wrong?

Update

When I restarted the webserver, the task test_with_zip has popped up, but it is not runnable because the Scheduler doesn't seem to recognize it. I get the following error for it (from the web interface):

This DAG seems to be existing only locally. The master scheduler doesn't seem to be aware of its existence.

Answer 1

You mention only to restart the webserver.

You also need to start the scheduler with airflow scheduler.

Also, see more steps to check here: Airflow 1.9.0 is queuing but not launching tasks

Answer 2

Which version of airflow are you on? Airflow 1.8.1 had problems with loading dags from zips. This issue was fixed in 1.8.3. https://issues.apache.org/jira/browse/AIRFLOW-1357

I recommend that you should update to the latest version of Airflow ie 1.9.0