How to mix logging Handlers (File & Timed) and com

Question

I need to prepare a logging config file in json which rotate it by time, size and compress de file rotated for all modules in my app (I'm stuck now). I want to do it using a sigle json config file, this is my current file, however this configuration only rotate by time:

{
    "version": 1,
    "disable_existing_loggers": false,
    "formatters": {
        "simple": {
            "format": "%(asctime)s - %(name)s - %(levelname)s - %(message)s",
            "datefmt": "%Y-%m-%d %H:%M:%S"
        }
    },

    "handlers": {
        "my_rotate": {
            "level": "DEBUG",
            "class": "logging.handlers.TimedRotatingFileHandler",
            "formatter": "simple",
            "when": "D",
            "interval": 1,
            "backupCount": 5,
            "filename": "/var/log/test.log",
            "encoding": "utf8"
        }
    },

    "loggers": {
        "my_module": {
            "level": "DEBUG",
            "handlers": ["my_rotate"]
        }
    },

    "root": {
        "level": "DEBUG",
        "handlers": ["my_rotate"],
        "propagate": false
    }
}

If I add other handler in handlers section (logging.handlers.RotateFileHandler) to rotate by size the result log file duplicate al messages.

My main program get logger properties using:

config_json_file = open(os.path.join(os.path.dirname(os.path.realpath(__file__)), "logging.json"))
logging_data = json.load(config_json_file)
logging.config.dictConfig(logging_data)

My modules call the logger like:

class StoreData(object):
    def __init__(self):
        self.logger = logging.getLogger(__name__)

How could I mix my three requirements (rotate by size and time, and compress rotated file) in this file?

Thaks a lot!

UPDATE 1

As suggested @Bakuriu I've created a new class to override needed methods to be able to compress rotated files:

class MyRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
    """ My rotating file hander to compress rotated file """

    def __init__(self, **kwargs):
        logging.handlers.TimedRotatingFileHandler.__init__(self, **kwargs)

    def rotate(self, source, dest):
        """ Compress rotated log file """
        os.rename(source, dest)
        f_in = open(dest, 'rb')
        f_out = gzip.open("%s.gz" % dest, 'wb')
        f_out.writelines(f_in)
        f_out.close()
        f_in.close()
        os.remove(dest)

So that "new" class is called fron logging config file where I added a new handler:

"handlers": {
    "my_rotate": {
        "level": "DEBUG",
        "class": "myMainPackage.MyRotatingFileHandler",
        "formatter": "simple",
        "when": "D",
        "interval": 1,
        "backupCount": 5,
        "filename": "/var/log/test.log",
        "encoding": "utf8"
}

In the next update I'll add a method to override the needed method(s), and be able to mix time and file size rotation together.

UPDATE 2

Well, I've overrided "shouldRollover" method, thus I've mix the time and file size for rotation.

class MyRotatingFileHandler(logging.handlers.TimedRotatingFileHandler):
        """ My rotating file hander to compress rotated file """
        def __init__(self, filename, mode='a', maxBytes=0, backupCount=0, encoding=None,
                     delay=0, when='h', interval=1, utc=False):
            if maxBytes > 0:
                mode = 'a'
            logging.handlers.TimedRotatingFileHandler.__init__(
                self, filename, when, interval, backupCount, encoding, delay, utc)
            self.maxBytes = maxBytes
            self.backupCount = backupCount

        def shouldRollover(self, record):
            """ Determine if rollover should occur. """
            # Check rollover by size
            if self.stream is None:                 # delay was set...
                self.stream = self._open()
            if self.maxBytes > 0:                   # are we rolling over?
                msg = "%s\n" % self.format(record)
                self.stream.seek(0, 2)  #due to non-posix-compliant Windows feature
                if self.stream.tell() + len(msg) >= self.maxBytes:
                    return 1
            # Check rollover by time
            t = int(time.time())
            if t >= self.rolloverAt:
                return 1
            return 0

        def rotate(self, source, dest):
            """ Compress rotated log file """
            os.rename(source, dest)
            f_in = open(dest, 'rb')
            f_out = gzip.open("%s.gz" % dest, 'wb')
            f_out.writelines(f_in)
            f_out.close()
            f_in.close()
            os.remove(dest)

And modify my logging config file in json to rotate time by day but check in seconds... it means 86400s (seconds in a day) and a size limit of 5Mb:

"handlers": {
        "my_rotate_timed": {
            "level": "DEBUG",
            "class": "myMainPackage.MyRotatingFileHandler",
            "formatter": "simple",
            "when": "S",
            "interval": 86400,
            "backupCount": 5,
            "maxBytes": 5242880,
            "filename": "/var/log/test.log",
            "encoding": "utf8"
    }

This way reuses the others methods from TimedRotationFileHandler and if rotation is called by maxBytes, then it uses suffix in seconds inherited from init TimedRotationFileHandler method, which format is: "%Y-%m-%d_%H-%M-%S" . This is the reason I've used "{when="S", interval=86400}" and not "{when="D", interval=1}"

Answer 1

This is to be expected. If you add a handler this handler either wont produce messages (due to filtering) and so nothing will change or it will write those messages (thus duplicating them).

The TimedRotatingFileHandler and the RotatingFileHandler only support, respectively, rotating by time and by size. Not both at the same time.

AFAIK there is no built-in way to achieve what you want, so using only the configuration file you wont be able to achieve what you want, you have to code something to put together the functionality.

Consider reading how to create a new rotating handler in the Logging Cookbook. If you save this handler class in a file mypackage.myrotatinghandler you can then specify:

class: 'mypackage.myrotatinghandler.MyHandler'

in the configuration file.

Note that, to add compression when saving a file it is sufficient to set the rotator attribute of the rotating handler with a function that saves a compressed file. Taken from the link above:

def namer(name):
    return name + ".gz"

def rotator(source, dest):
    with open(source, "rb") as sf:
        data = sf.read()
        compressed = zlib.compress(data, 9)
        with open(dest, "wb") as df:
            df.write(compressed)
    os.remove(source)

rh = logging.handlers.RotatingFileHandler(...)
rh.rotator = rotator
rh.namer = namer

The rh handler will behave as a normal RotatingFileHandler but also compress the logs.

However setting the conditions to do a rotation require that you re-write parts of the handler. You may read the sources for the logging.handlers module to see how the built-in handlers are implemented.