How to route logging messages from the multiproces

Question

In the Python docs I found that logging messages are possible in python scripts that use the multiprocessing module to spawn different processes.

import multiprocessing
import logging

logger = multiprocessing.log_to_stderr()
logger.setLevel(logging.INFO)

So logger.warning('doomed') would give me the message 'doomed', but only to the console.

How can I switch the log-messages to a file?

Answer 1

Multiprocess logging isn't trivial, but isn't difficult either. You'll need the logging module.

The main point is to have a listener process that will receive all the logging records through multiprocessing queues. This listener process will then handle those records with whathever internal handlers you'll decide to use. In your case you may want to use a FileHandler or similar.

If you are in Python 3.2 or later version you'll have both QueueHandler and QueueListener in the standard libary. Quelistener is the listener process that I was talking about before (yes, it does have a start() method).

If you are on a previous Python version take a look at this link that basically shows how to rewrite them (and how to use them).

---

If you want a references from the official doc, take a look at the Logging to a single file from multiple processes session from the Logging Cookbook.

You may also be interested in this answer from the question How should I log while using multiprocessing in Python?