How to configure YAML to create fresh log files in

Question

In a python logger implementation given below, each time I run my program, the logs are appended each time to the existing log files. How do I ensure that each time I run my application code, they are written to the fresh log file?

Is it happening because I have set the RotatingFileHandler with backup count as 20 with each file size of 10MB? Should I convert it to simple file handler?

I am using following yaml based log configuration in my python logger.

  1 version: 1
  2
  3 formatters:
  4   simple:
  5     format: '%(asctime)s - %(name)s - %(levelname)s - %(message)s'
  6
  7 handlers:
  8   console:
  9     class: logging.StreamHandler
 10     level: DEBUG
 11     formatter: simple
 12     stream: ext://sys.stdout
 13
 14   info_handler:
 15     class: logging.handlers.RotatingFileHandler
 16     level: INFO
 17     formatter: simple
 18     filename: info.log
 19     maxBytes: 10485760 # 10MB
 20     backupCount: 20
 21     encoding: utf8
 22
 23   error_handler:
 24     class: logging.handlers.RotatingFileHandler
 25     level: ERROR
 26     formatter: simple
 27     filename: errors.log
 28     maxBytes: 10485760 # 10MB
 29     backupCount: 20
 30     encoding: utf8
 31
 32 loggers:
 33   my_module:
 34     level: ERROR
 35     handlers: [console]
 36     propagate: no
 37
 38 root:
 39   level: DEBUG
 40   handlers: [info_handler, info_handler]

I am using the following python logger initializer code to initialize my logger.

  1 import os
  2 import logging.config
  3 import yaml
  4
  5 """Setup logging configuration """
  6 default_path='logging.yaml'
  7 default_level=logging.INFO
  8 env_key='LOG_CFG'
  9
 10 class MyLogger():
 11
 12     def __init__(self):
 13         path = default_path
 14         value = os.getenv(env_key, None)
 15         if value:
 16             path = value
 17         if os.path.exists(path):
 18             with open(path, 'rt') as f:
 19                 config = yaml.safe_load(f.read())
 20             logging.config.dictConfig(config)
 21         else:
 22             logging.basicConfig(filemode='w', level=default_level)

Answer 1

Set the filemode to w, the default is a (append).

Or alternatively just add the following line to overwrite your old log file (after reading the yaml file):

with open(config['handlers']['info_handler']['filename'], 'w') as f:
    pass

Answer 2

I know this already has an accepted answer but I'm not sure it answers the question very cleanly/entirely and I am adding this answer in case others stumble across a similar problem. Instead, one option to include this in your configuration YAML file:

handlers:
    info_file_handler:
       class: logging.FileHandler
       formatter: complex
       filename: log_info.log
       mode: 'w'

The 'w' in mode is the same as the filemode option used in basicConfig() and essentially overwrites the log file instead of appending (which would be option 'a'). I also think the mode key can be used for RotatingFileHandler.

Then I add to my main code so that each filename is changed with each model run (this currently changes the log filename every day [I have a daily program running]). See below:

def setup_logging(default_path="logging_config.yml",
                  default_level=logging.INFO,
                  env_key="LOG_CFG"):

    """
    Setup logging configuration

    This reads a .yml file and extracts the relevant logging information
    required by the logging module.

    """

    path = default_path
    value = os.getenv(env_key, None)

    if value:
         path = value

    # Open the logging configuration file
    if os.path.exists(path):
        with open(path, "rb") as f:
            config = yaml.safe_load(f.read())

            # Append the date stamp to the filename for each handler
            # This is so each log file has a unique filename if run
            # on a separate date.
            # This can be changed to suit needs, hourly/minutely etc.
            for i in (config["handlers"].keys()):
                log_filename = config["handlers"][i]["filename"]
                base, extension = os.path.splitext(log_filename)
                today = datetime.datetime.today()
                log_filename = "{}{}{}".format(base,
                                               today.strftime("_%Y_%m_%d"),
                                               extension)
                config["handlers"][i]["filename"] = log_filename

        logging.config.dictConfig(config)

    else:
        logging.basicConfig(level=default_level)

I hope this helps and apologies if the accepted answer does actually work for you, I just felt this was a cleaner solution to the problem the OP posed.