python : print output of each thread to seperate f

Question

I have several threads and each thread writes output to stdout. However I want to redirect the ouput of each thread to a separate file independently of each other, then merge them to keep he flow of each thread together.

What I mean is the following:

Thread1 writes every print, every exception and every other ouput into file1.log Thread2 writes every print, every exception and every other ouput into file2.log and so on. So what I'm looking for is to set the stdout for each thread exclusivly. However setting the stdout only works globally mean that Thread1 and Tread2 will always write to the same defined stdout. I have not found out yet how to do this. I can't use processes though, because f another issue.

How can I do that?

Answer 1

The Python logging module is thread-safe.

Use it to create an individual logger for each thread, and register a FileHandler to (also) log to a file:

logger = logging.getLogger('thread-1')
file_handler = logging.FileHandler('thread-1.log')
logger.addHandler(file_handler)

Here's a more complete example:

import logging
import random
import threading
import time

NUM_THREADS = 5


def worker(delay, logger):
    """A toy worker function, taking the logger as an argument.
    """
    logger.info("Starting work...")
    for i in range(3):
        logger.info('Sleeping %0.02f', delay)
        time.sleep(delay)
    logger.info('Done.')


for n in range(1, NUM_THREADS + 1):
    # create the thread's logger
    logger = logging.getLogger('thread-%s' % n)
    logger.setLevel(logging.DEBUG)

    # create a file handler writing to a file named after the thread
    file_handler = logging.FileHandler('thread-%s.log' % n)

    # create a custom formatter and register it for the file handler
    formatter = logging.Formatter('(%(threadName)-10s) %(message)s')
    file_handler.setFormatter(formatter)

    # register the file handler for the thread-specific logger
    logger.addHandler(file_handler)

    delay = random.random()
    t = threading.Thread(target=worker, args=(delay, logger))
    t.start()


main_thread = threading.currentThread()
for t in threading.enumerate():
    if t is not main_thread:
        t.join()

This will give you five logfiles, thread-1.log through thread-5.log, containing only the output of the respective thread:

thread-1.log

(Thread-1  ) Starting work...
(Thread-1  ) Sleeping 0.53
(Thread-1  ) Sleeping 0.53
(Thread-1  ) Sleeping 0.53
(Thread-1  ) Done.

---

If you still want to log to the console, simply create a StreamHandler and attach it to your logger:

stream_handler = logging.StreamHandler()
logger.addHandler(stream_handler)

This will log to STDERR by default. If you want STDOUT, use

logging.StreamHandler(sys.stdout)

For more information on using the Python logging module, see the Advanced Logging Tutorial.