How to redirect TensorFlow logging to a file?

Question

I'm using TensorFlow-Slim, which has some useful logging printed out to console by tf.logging. I would like to redirect those loggings to a text file, but couldn't find a way doing so. I looked at the tf_logging.py source code, which exposes the following, but doesn't seem to have the option to write logs to a file. Please let me know if I missed something.

__all__ = ['log', 'debug', 'error', 'fatal', 'info', 'warn', 'warning',
           'DEBUG', 'ERROR', 'FATAL', 'INFO', 'WARN',
           'flush', 'log_every_n', 'log_first_n', 'vlog',
           'TaskLevelStatusMessage', 'get_verbosity', 'set_verbosity']

Answer 1

import logging

# get TF logger
log = logging.getLogger('tensorflow')
log.setLevel(logging.DEBUG)

# create formatter and add it to the handlers
formatter = logging.Formatter('%(asctime)s - %(name)s - %(levelname)s - %(message)s')

# create file handler which logs even debug messages
fh = logging.FileHandler('tensorflow.log')
fh.setLevel(logging.DEBUG)
fh.setFormatter(formatter)
log.addHandler(fh)

My solution is inspired by this thread.

Answer 2

You are right, there are no knobs for you to do that.

If you truly, positively, absolutely cannot live with that, tf.logging is based on python logging. So, import logging tf.logging._logger.basicConfig(filename='tensorflow.log', level=logging.DEBUG)

Note that you are on your own on an unsupported path, and that behavior may break at anytime.

You may also file a feature request at our github issue page.

Answer 3

If you are using python logging in your project, one of the option will be to define the logger with name "tensorflow" in a logging config file.

Then _logger = _logging.getLogger('tensorflow') will use the logger and specified handlers from your config file.

Answer 4

A easy workaround would be to direct the output from command line to a file. For example,

python training.py 1> output.log 2> error.log
# 1 for stdout stream and 2 for stderr stream

The benefit, in light of the accepted answer, is that you get ALL the logs. The reason is that not all the logging comes from python (Remember the runtime is implemented in C++). For instance, you could get very helpful debugging logging (including info about Tensor memory allocation) by setting the environment variable.

import os
os.environ['TF_CPP_MIN_VLOG_LEVEL'] = '3'

(Do this with CAUTION. The amount of logging is staggering.)

For a glimpse of how logging is implemented in C++,

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/default/logging.h https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/default/logging.cc

In particular it looks like logging messages are written to stderr in this line:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/core/platform/default/logging.cc#L73

which I quote from the discussion here