Realtime output redirection

Question

Currently I am redirecting a script to a log file with the following command:

python /usr/home/scripts/myscript.py 2>&1 | tee /usr/home/logs/mylogfile.log

This seems to work but it does not write to the file as soon as there is a print command. Rather it waits until there is a group of lines that it can print. I want the console and the log file to be written to simultaneously. How can this be done with output redirection. Note that running the script on the console prints everything when it should. Though doing a tail -f on the logfile is not smooth since it writes about 50 lines at a time. Any suggestions?

Answer 1

It sounds like the shell is actually what's doing the buffering, since you say it outputs as expected to the console when not tee'd.

You could look at this post for potential solutions to undo that shell buffering: https://unix.stackexchange.com/questions/25372/turn-off-buffering-in-pipe

But I would recommend doing it entirely within Python, so you have more direct control, and instead of printing to stdout, use the logging module.

This would allow additional flexibility in terms of multiple logging levels, the ability to add multiple sources to the logging object centrally (i.e. stdout and a file -- and one which rotates with size if you'd like with logging.handlers.RotatingFileHandler) and you wouldn't be subject to the external buffering of the shell.

More info: https://docs.python.org/2/howto/logging.html