Incompatibility between import-time logger naming

Question

I am setting up my Python logging in main.py via reading in a file and using the fileConfig option. I want to be able to switch between testing and live logging configurations, so I want to read in a separate config file first and extract the logging config file path from there.

The problem here is that other files that I import from main.py grab their own logger via log = getLogger(__name__), and this happens at import time. These links then get broken when the new configuration is loaded in, and these modules end up without the logging working the way I expect.

I can't easily delay the importing of these modules without a lot of refactoring, so is there any other way of being able to keep this method of setting up loggers by module name while still loading in the log configuration later?

Answer 1

I'm not sure from your question exactly how things are breaking, but here's how I see it. The various modules which do log = logging.getLogger(__name__) will have valid names for their loggers (logger name = package name), unless you were to somehow actually move the modules to some other package location.

At import time, the logging configuration may or may not have been set, and there shouldn't be any actual logging calls made as a side-effect of the import (if there are, the messages may have nowhere to go).

Loading a new configuration using fileConfig typically just sets handlers, formatters and levels on loggers.

When you subsequently call code in the imported modules, they log via their loggers, which have handlers attached by your previous configuration call - so they will output according to the configuration.

You should be aware that on older versions of Python (<= 2.5), calls to fileConfig would unavoidably disable existing loggers which weren't named in the configuration - in more recent versions of Python (>= 2.6), this is configurable using a disable_existing_loggers=False keyword argument passed to fileConfig. You may want to check this, as it sometimes leads to unexpected behaviour (the default for that parameter is True, for compatibility with behaviour under the older Python versions).

If you post more details about what seems broken, I might be able to provide a better diagnosis of what's going on.