Dynamic filepath & filename for FileHandler in log

Question

I have a python log config file with a filehandler of the below form.

[handler_filelog]
class: FileHandler
args = ('/var/tmp/log/client.log','a')

Instead, I need it in the below form (dynamically generated path).

[handler_filelog]
class: FileHandler
args = ('/var/tmp/log_<unique_string>/client.log','a')

Multiple instances of the program may be running and hence non-conflicting log paths and files are to be used. The logger once setup need not change till end of program execution.

Is there a way to handle this using the config file approach? I am not keen on resorting to creating the loggers/handlers/formatters by myself since my log config file has many of these and config file based approach is much nicer.

(Update: I am using python 2.4)

Answer 1

"CallbackFilter" can be used to implement Dynamic filepath & filename for in logger config file in python. You can define write_dynamic_log as below:

def write_dynamic_log(record):
    now = datetime.datetime.now().strftime("%Y%m%d%H%M%S")
    dynamic_log_name = '/var/log/test_%s.log' %now
    log_file = open(dynamic_log_name, "w")
    log_file.write(record.msg)
    log_file.close();
    return True

Then in the config file, you can use this filter like this:

[handler_filelog]
class: FileHandler
formatter: brief
level   : INFO
filters: [write_dynamic_log]
filename: static.log

The INFO or above log will be output to static.log and also to dynamic_log.

I tested it in my django project, in which I wrote config in my settings.py. It works fine. LOGGING will is like:

LOGGING = {
    'version': 1,
    'disable_existing_loggers': False,
    'formatters': {
        'verbose': {
            'format': '[%(levelname)s] %(asctime)s - %(pathname)s:%(lineno)d\n%(message)s'
        },
        'debug': {
            'format': '[%(levelname)s] %(asctime)s - %(pathname)s:%(lineno)d\n\033[34m%(message)s\033[0m'
        },
        'error': {
            'format': 'Component: %(module)s\nErrorCode: %(funcName)s\nReason: %(funcName)s\nDetail: [%(message)s]'
        },
    },
    'filters': {
        'write_error_logs': {
            '()': 'django.utils.log.CallbackFilter',
            'callback': write_error_log,
        }
    },
    'handlers': {
        'console': {
            'class': 'logging.StreamHandler',
            'formatter': 'verbose',
        },
        'debug': {
            'class': 'logging.StreamHandler',
            'formatter': 'debug',
        },
        'error': {
            'level': 'ERROR',
            'class': 'logging.FileHandler',
            'filename': '/var/log/SmartStorageApp.err',
            'formatter': 'error',
            'filters': ['write_error_logs'],
        },
    },
    'loggers': {
        'django': {
            'handlers': ['debug' if DEBUG else 'console', 'error'],
            'level': 'INFO',
            'propagate': True,
        },
    }
}

Answer 2

This does what you need. You should first extend the FileHandler class. Place this in a file, say myHandler.py in your config file's directory:

import logging
import random
import os
class myFileHandler(logging.FileHandler):
    def __init__(self,path,fileName,mode):
        r = random.randint(1,100000)
        path = path+"/log_"+str(r)
        os.mkdir(path)
        super(myFileHandler,self).__init__(path+"/"+fileName,mode)

And then in the config file, you can use this custom FileHandler like this

class: myHandler.myFileHandler
args = ('/var/tmp','client.log','a')

I tested this one on my machine

Answer 3

If you're using Python 2.7 or 3.2, you can use dictionary-based configuration, which allows you to specify arbitrary callables to return handlers - you could e.g. use the process PID to construct the file name.

Update: If you are using 2.4, you can use the logutils package, which should work with Python 2.4 (apart from the LoggerAdapter class, which requires 2.5 or later). The logutils package contains the dictionary-based configuration functionality.