Python: logging module - globally

Question

Hey I was wondering how to implement a global logger that could be used everywhere with your own settings:

I have

class customLogger(logging.Logger):
   ...

in a file with its formatters and other stuff. The logger works perfectly on its own.

I import this module in my main.py file and create an object like this:

self.log = log.customLogger(arguments)

But obviously i cannot access this object from other parts of my code. Am i using a wrong approach? Is there a better way to do this?

Answer 1

Create an instance of customLogger in your log module and use it as a singleton - just use the imported instance, rather than the class.

Answer 2

Use logging.getLogger(name) to create a named global logger.

main.py

import log
logger = log.setup_custom_logger('root')
logger.debug('main message')

import submodule

log.py

import logging

def setup_custom_logger(name):
    formatter = logging.Formatter(fmt='%(asctime)s - %(levelname)s - %(module)s - %(message)s')

    handler = logging.StreamHandler()
    handler.setFormatter(formatter)

    logger = logging.getLogger(name)
    logger.setLevel(logging.DEBUG)
    logger.addHandler(handler)
    return logger

submodule.py

import logging

logger = logging.getLogger('root')
logger.debug('submodule message')

Output

2011-10-01 20:08:40,049 - DEBUG - main - main message
2011-10-01 20:08:40,050 - DEBUG - submodule - submodule message

Answer 3

You can just pass it a string with a common sub-string before the first period. The parts of the string separated by the period (".") can be used for different classes / modules / files / etc. Like so (specifically the logger = logging.getLogger(loggerName) part):

def getLogger(name, logdir=LOGDIR_DEFAULT, level=logging.DEBUG, logformat=FORMAT):
    base = os.path.basename(__file__)
    loggerName = "%s.%s" % (base, name)
    logFileName = os.path.join(logdir, "%s.log" % loggerName)
    logger = logging.getLogger(loggerName)
    logger.setLevel(level)
    i = 0
    while os.path.exists(logFileName) and not os.access(logFileName, os.R_OK | os.W_OK):
        i += 1
        logFileName = "%s.%s.log" % (logFileName.replace(".log", ""), str(i).zfill((len(str(i)) + 1)))
    try:
        #fh = logging.FileHandler(logFileName)
        fh = RotatingFileHandler(filename=logFileName, mode="a", maxBytes=1310720, backupCount=50)
    except IOError, exc:
        errOut = "Unable to create/open log file \"%s\"." % logFileName
        if exc.errno is 13: # Permission denied exception
            errOut = "ERROR ** Permission Denied ** - %s" % errOut
        elif exc.errno is 2: # No such directory
            errOut = "ERROR ** No such directory \"%s\"** - %s" % (os.path.split(logFileName)[0], errOut)
        elif exc.errno is 24: # Too many open files
            errOut = "ERROR ** Too many open files ** - Check open file descriptors in /proc/<PID>/fd/ (PID: %s)" % os.getpid()
        else:
            errOut = "Unhandled Exception ** %s ** - %s" % (str(exc), errOut)
        raise LogException(errOut)
    else:
        formatter = logging.Formatter(logformat)
        fh.setLevel(level)
        fh.setFormatter(formatter)
        logger.addHandler(fh)
    return logger

class MainThread:
    def __init__(self, cfgdefaults, configdir, pidfile, logdir, test=False):
        self.logdir = logdir
        logLevel = logging.DEBUG
        logPrefix = "MainThread_TEST" if self.test else "MainThread"
        try:
            self.logger = getLogger(logPrefix, self.logdir, logLevel, FORMAT)
        except LogException, exc:
            sys.stderr.write("%s\n" % exc)
            sys.stderr.flush()
            os._exit(0)
        else:
            self.logger.debug("-------------------- MainThread created.  Starting __init__() --------------------")

    def run(self):
        self.logger.debug("Initializing ReportThreads..")
        for (group, cfg) in self.config.items():
            self.logger.debug(" ------------------------------ GROUP '%s' CONFIG ------------------------------     " % group)
            for k2, v2 in cfg.items():
                self.logger.debug("%s <==> %s: %s" % (group, k2, v2))
            try:
                rt = ReportThread(self, group, cfg, self.logdir, self.test)
            except LogException, exc:
                sys.stderr.write("%s\n" % exc)
                sys.stderr.flush()
                self.logger.exception("Exception when creating ReportThread (%s)" % group)
                logging.shutdown()
                os._exit(1)
            else:
                self.threads.append(rt)
        self.logger.debug("Threads initialized.. \"%s\"" % ", ".join([t.name for t in self.threads]))
        for t in self.threads:
            t.Start()
        if not self.test:
            self.loop()


class ReportThread:
    def __init__(self, mainThread, name, config, logdir, test):
        self.mainThread = mainThread
        self.name = name
        logLevel = logging.DEBUG
        self.logger = getLogger("MainThread%s.ReportThread_%s" % ("_TEST" if self.test else "", self.name), logdir, logLevel, FORMAT)
        self.logger.info("init database...")
        self.initDB()
        # etc....

if __name__ == "__main__":
    # .....
    MainThread(cfgdefaults=options.cfgdefaults, configdir=options.configdir, pidfile=options.pidfile, logdir=options.logdir, test=options.test)

Answer 4

Since I haven't found a satisfactory answer, I would like to elaborate on the answer to the question a little bit in order to give some insight into the workings and intents of the logging library, that comes with Python's standard library.

In contrast to the approach of the OP (original poster) the library clearly separates the interface to the logger and configuration of the logger itself.

The configuration of handlers is the prerogative of the application developer who uses your library.

That means you should not create a custom logger class and configure the logger inside that class by adding any configuration or whatsoever.

The logging library introduces four components: loggers, handlers, filters, and formatters.

• Loggers expose the interface that application code directly uses.
• Handlers send the log records (created by loggers) to the appropriate destination.
• Filters provide a finer grained facility for determining which log records to output.
• Formatters specify the layout of log records in the final output.

A common project structure looks like this:

Project/
|-- .../
|   |-- ...
|
|-- project/
|   |-- package/
|   |   |-- __init__.py
|   |   |-- module.py
|   |   
|   |-- __init__.py
|   |-- project.py
|
|-- ...
|-- ...

Inside your code (like in module.py) you refer to the logger instance of your module to log the events at their specific levels.

A good convention to use when naming loggers is to use a module-level logger, in each module which uses logging, named as follows:

logger = logging.getLogger(__name__)

The special variable __name__ refers to your module's name and looks something like project.package.module depending on your application's code structure.

module.py (and any other class) could essentially look like this:

import logging
...
log = logging.getLogger(__name__)

class ModuleClass:
    def do_something(self):
        log.debug('do_something() has been called!')

The logger in each module will propagate any event to the parent logger which in return passes the information to its attached handler! Analogously to the python package/module structure, the parent logger is determined by the namespace using "dotted module names". That's why it makes sense to initialize the logger with the special __name__ variable (in the example above name matches the string "project.package.module").

There are two options to configure the logger globally:

• Instantiate a logger in project.py with the name __package__ which equals "project" in this example and is therefore the parent logger of the loggers of all submodules. It is only necessary to add an appropriate handler and formatter to this logger.

• Set up a logger with a handler and formatter in the executing script (like main.py) with the name of the topmost package.

When developing a library which uses logging, you should take care to document how the library uses logging - for example, the names of loggers used.

The executing script, like main.py for example, might finally look something like this:

import logging
from project import App

def setup_logger():
    # create logger
    logger = logging.getLogger('project')
    logger.setLevel(logging.DEBUG)

    # create console handler and set level to debug
    ch = logging.StreamHandler()
    ch.setLevel(level)

    # create formatter
    formatter = logging.Formatter('%(asctime)s [%(levelname)s] %(name)s: %(message)s')

    # add formatter to ch
    ch.setFormatter(formatter)

    # add ch to logger
    logger.addHandler(ch)

if __name__ == '__main__' and __package__ is None:
     setup_logger()
     app = App()
     app.do_some_funny_stuff()

The method call log.setLevel(...) specifies the lowest-severity log message a logger will handle but not necessarily output! It simply means the message is passed to the handler as long as the message's severity level is higher than (or equal to) the one that is set. But the handler is responsible for handling the log message (by printing or storing it for example).

Hence the logging library offers a structured and modular approach which just needs to be exploited according to one's needs.

Logging documentation