Disable output buffering

Question

Is output buffering enabled by default in Python's interpreter for sys.stdout?

If the answer is positive, what are all the ways to disable it?

Suggestions so far:

1. Use the -u command line switch
2. Wrap sys.stdout in an object that flushes after every write
3. Set PYTHONUNBUFFERED env var
4. sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

Is there any other way to set some global flag in sys/sys.stdout programmatically during execution?

Answer 1

You can also run Python with stdbuf utility:

stdbuf -oL python <script>

Answer 2

You can also use fcntl to change the file flags in-fly.

fl = fcntl.fcntl(fd.fileno(), fcntl.F_GETFL)
fl |= os.O_SYNC # or os.O_DSYNC (if you don't care the file timestamp updates)
fcntl.fcntl(fd.fileno(), fcntl.F_SETFL, fl)

Answer 3

Variant that works without crashing (at least on win32; python 2.7, ipython 0.12) then called subsequently (multiple times):

def DisOutBuffering():
    if sys.stdout.name == '<stdout>':
        sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

    if sys.stderr.name == '<stderr>':
        sys.stderr = os.fdopen(sys.stderr.fileno(), 'w', 0)

Answer 4

It is possible to override only write method of sys.stdout with one that calls flush. Suggested method implementation is below.

def write_flush(args, w=stdout.write):
    w(args)
    stdout.flush()

Default value of w argument will keep original write method reference. After write_flush is defined, the original write might be overridden.

stdout.write = write_flush

The code assumes that stdout is imported this way from sys import stdout.

Answer 5

def disable_stdout_buffering():
    # Appending to gc.garbage is a way to stop an object from being
    # destroyed.  If the old sys.stdout is ever collected, it will
    # close() stdout, which is not good.
    gc.garbage.append(sys.stdout)
    sys.stdout = os.fdopen(sys.stdout.fileno(), 'w', 0)

# Then this will give output in the correct order:
disable_stdout_buffering()
print "hello"
subprocess.call(["echo", "bye"])

Without saving the old sys.stdout, disable_stdout_buffering() isn't idempotent, and multiple calls will result in an error like this:

Traceback (most recent call last):
  File "test/buffering.py", line 17, in <module>
    print "hello"
IOError: [Errno 9] Bad file descriptor
close failed: [Errno 9] Bad file descriptor

Another possibility is:

def disable_stdout_buffering():
    fileno = sys.stdout.fileno()
    temp_fd = os.dup(fileno)
    sys.stdout.close()
    os.dup2(temp_fd, fileno)
    os.close(temp_fd)
    sys.stdout = os.fdopen(fileno, "w", 0)

(Appending to gc.garbage is not such a good idea because it's where unfreeable cycles get put, and you might want to check for those.)

Answer 6

This relates to Cristóvão D. Sousa's answer, but I couldn't comment yet.

A straight-forward way of using the flush keyword argument of Python 3 in order to always have unbuffered output is:

import functools
print = functools.partial(print, flush=True)

afterwards, print will always flush the output directly (except flush=False is given).

Note, (a) that this answers the question only partially as it doesn't redirect all the output. But I guess print is the most common way for creating output to stdout/stderr in python, so these 2 lines cover probably most of the use cases.

Note (b) that it only works in the module/script where you defined it. This can be good when writing a module as it doesn't mess with the sys.stdout.

Python 2 doesn't provide the flush argument, but you could emulate a Python 3-type print function as described here https://stackoverflow.com/a/27991478/3734258 .