Python logging and rotating files

Question

I have a python program that is writing to a log file that is being rotated by Linux's logrotate command. When this happens I need to signal my program to stop writing to the old file and start writing to the new one. I can handle the signal but how do I tell python to write to the new file?

I am opening the file like this:

logging.basicConfig(format='%(asctime)s:%(filename)s:%(levelname)s:%(message)s',filename=log_file, level=logging.INFO)

and writing to it like this:

logging.log(level,"%s" % (msg))

The logging modules look very powerful but also overwhelming. Thanks.

Answer 1

from logging import handlers

handler = handlers.TimedRotatingFileHandler(filename, when=LOG_ROTATE)

handler.setFormatter(logging.Formatter(log_format, datefmt="%d-%m-%Y %H:%M:%S"))

#LOG_ROTATE = midnight    
#set your log format

This should help you in handling rotating log

Answer 2

You may want to look at WatchedFileHandler to implement this, or as an alternative, implement log rotation with RotatingFileHandler, both of which are in the logging.handlers module.

Answer 3

Don't use logging.basicConfig, use WatchedFileHandler. Here's how to use it.

import time
import logging
import logging.handlers

def log_setup():
    log_handler = logging.handlers.WatchedFileHandler('my.log')
    formatter = logging.Formatter(
        '%(asctime)s program_name [%(process)d]: %(message)s',
        '%b %d %H:%M:%S')
    formatter.converter = time.gmtime  # if you want UTC time
    log_handler.setFormatter(formatter)
    logger = logging.getLogger()
    logger.addHandler(log_handler)
    logger.setLevel(logging.DEBUG)

log_setup()
logging.info('Hello, World!')
import os
os.rename('my.log', 'my.log-old')
logging.info('Hello, New World!')

Answer 4

Since rotation is already being done by logrotate, in your signal handler you should just call logging.basicConfig(...) again and that should reopen the log file.