Adding logs to Airflow Logs

Question

How can I add my own logs onto the Apache Airflow logs that are automatically generated? any print statements wont get logged in there, so I was wondering how I can add my logs so that it shows up on the UI as well?

Answer 1

I think you can work around this by using the logging module and trusting the configuration to Airflow.

Something like:

import ...

dag = ...

def print_params_fn(**kwargs):
    import logging
    logging.info(kwargs)
    return None

print_params = PythonOperator(task_id="print_params",
                              python_callable=print_params_fn,
                              provide_context=True,
                              dag=dag)

Answer 2

Inside python callable for PythonOperator you can use:

import logging

LOGGER = logging.getLogger("airflow.task")
LOGGER.info("airflow.task >>> 2 - INFO logger test")

This will produce correct output like:

[2019-12-26 09:42:55,813] {operations.py:86} INFO - airflow.task >>> 2 - INFO logger test

For case with your custom logger:

LOGGER = logging.getLogger(__name__)
LOGGER.info("__name__ >>> 2 - INFO logger test")

You'll get duplication of formatting:

[2019-12-26 09:42:55,813] {logging_mixin.py:112} INFO - [2019-12-26 09:42:55,813] {operations.py:79} INFO - __name__ >>> 2 - INFO logger test

Answer 3

If you look at the PythonOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/python_operator.py#L80-L81, looks like there is no way to log STDOUT/STDERR from the python callable into the airflow logs.

However, if you look at the BashOperator: https://github.com/apache/incubator-airflow/blob/master/airflow/operators/bash_operator.py#L79-L94, the STDOUT/STDERR from there is logged along with the airflow logs. So, if logs are important to you, I suggest adding the python code in a separate file and calling it using the BashOperator.