Getting realtime output using subprocess

问题:

I am trying to write a wrapper script for a command line program (svnadmin verify) that will display a nice progress indicator for the operation. This requires me to be able to see each line of output from the wrapped program as soon as it is output.

I figured that I\'d just execute the program using subprocess.Popen, use stdout=PIPE, then read each line as it came in and act on it accordingly. However, when I ran the following code, the output appeared to be buffered somewhere, causing it to appear in two chunks, lines 1 through 332, then 333 through 439 (the last line of output)

from subprocess import Popen, PIPE, STDOUT

p = Popen(\'svnadmin verify /var/svn/repos/config\', stdout = PIPE, 
        stderr = STDOUT, shell = True)
for line in p.stdout:
    print line.replace(\'\\n\', \'\')

After looking at the documentation on subprocess a little, I discovered the bufsize parameter to Popen, so I tried setting bufsize to 1 (buffer each line) and 0 (no buffer), but neither value seemed to change the way the lines were being delivered.

At this point I was starting to grasp for straws, so I wrote the following output loop:

while True:
    try:
        print p.stdout.next().replace(\'\\n\', \'\')
    except StopIteration:
        break

but got the same result.

Is it possible to get \'realtime\' program output of a program executed using subprocess? Is there some other option in Python that is forward-compatible (not exec*)?

回答1:

I tried this, and for some reason while the code

for line in p.stdout:
  ...

buffers aggressively, the variant

while True:
  line = p.stdout.readline()
  if not line: break
  ...

does not. Apparently this is a known bug: http://bugs.python.org/issue3907 (The issue is now \"Closed\" as of Aug 29, 2018)

回答2:

p = subprocess.Popen(cmd, stdout=subprocess.PIPE, bufsize=1)
for line in iter(p.stdout.readline, b\'\'):
    print line,
p.stdout.close()
p.wait()

回答3:

You can try this:

import subprocess
import sys

process = subprocess.Popen(
    cmd, stdout=subprocess.PIPE, stderr=subprocess.PIPE
)

while True:
    out = process.stdout.read(1)
    if out == \'\' and process.poll() != None:
        break
    if out != \'\':
        sys.stdout.write(out)
        sys.stdout.flush()

If you use readline instead of read, there will be some cases where the input message is not printed. Try it with a command the requires an inline input and see for yourself.

回答4:

You can direct the subprocess output to the streams directly. Simplified example:

subprocess.run([\'ls\'], stderr=sys.stderr, stdout=sys.stdout)

回答5:

I ran into the same problem awhile back. My solution was to ditch iterating for the read method, which will return immediately even if your subprocess isn\'t finished executing, etc.

回答6:

Real Time Output Issue resolved: I did encountered similar issue in Python, while capturing the real time output from c program. I added \"fflush(stdout);\" in my C code. It worked for me. Here is the snip the code

<< C Program >>

#include <stdio.h>
void main()
{
    int count = 1;
    while (1)
    {
        printf(\" Count  %d\\n\", count++);
        fflush(stdout);
        sleep(1);
    }
}

<< Python Program >>

#!/usr/bin/python

import os, sys
import subprocess


procExe = subprocess.Popen(\".//count\", shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE, universal_newlines=True)

while procExe.poll() is None:
    line = procExe.stdout.readline()
    print(\"Print:\" + line)

<< OUTPUT>> Print: Count 1 Print: Count 2 Print: Count 3

Hope it helps.

~sairam

回答7:

You may use an iterator over each byte in the output of the subprocess. This allows inline update (lines ending with \'\\r\' overwrite previous output line) from the subprocess:

from subprocess import PIPE, Popen

command = [\"my_command\", \"-my_arg\"]

# Open pipe to subprocess
subprocess = Popen(command, stdout=PIPE, stderr=PIPE)


# read each byte of subprocess
while subprocess.poll() is None:
    for c in iter(lambda: subprocess.stdout.read(1) if subprocess.poll() is None else {}, b\'\'):
        c = c.decode(\'ascii\')
        sys.stdout.write(c)
sys.stdout.flush()

if subprocess.returncode != 0:
    raise Exception(\"The subprocess did not terminate correctly.\")

回答8:

Using pexpect [ http://www.noah.org/wiki/Pexpect ] with non-blocking readlines will resolve this problem. It stems from the fact that pipes are buffered, and so your app\'s output is getting buffered by the pipe, therefore you can\'t get to that output until the buffer fills or the process dies.

回答9:

I used this solution to get realtime output on a subprocess. This loop will stop as soon as the process completes leaving out a need for a break statement or possible infinite loop.

sub_process = subprocess.Popen(my_command, close_fds=True, shell=True, stdout=subprocess.PIPE, stderr=subprocess.PIPE)

while sub_process.poll() is None:
    out = sub_process.stdout.read(1)
    sys.stdout.write(out)
    sys.stdout.flush()

回答10:

This is the basic skeleton that I always use for this. It makes it easy to implement timeouts and is able to deal with inevitable hanging processes.

import subprocess
import threading
import Queue

def t_read_stdout(process, queue):
    \"\"\"Read from stdout\"\"\"

    for output in iter(process.stdout.readline, b\'\'):
        queue.put(output)

    return

process = subprocess.Popen([\'dir\'],
                           stdout=subprocess.PIPE,
                           stderr=subprocess.STDOUT,
                           bufsize=1,
                           cwd=\'C:\\\\\',
                           shell=True)

queue = Queue.Queue()
t_stdout = threading.Thread(target=t_read_stdout, args=(process, queue))
t_stdout.daemon = True
t_stdout.start()

while process.poll() is None or not queue.empty():
    try:
        output = queue.get(timeout=.5)

    except Queue.Empty:
        continue

    if not output:
        continue

    print(output),

t_stdout.join()

回答11:

Complete solution:

import contextlib
import subprocess

# Unix, Windows and old Macintosh end-of-line
newlines = [\'\\n\', \'\\r\\n\', \'\\r\']
def unbuffered(proc, stream=\'stdout\'):
    stream = getattr(proc, stream)
    with contextlib.closing(stream):
        while True:
            out = []
            last = stream.read(1)
            # Don\'t loop forever
            if last == \'\' and proc.poll() is not None:
                break
            while last not in newlines:
                # Don\'t loop forever
                if last == \'\' and proc.poll() is not None:
                    break
                out.append(last)
                last = stream.read(1)
            out = \'\'.join(out)
            yield out

def example():
    cmd = [\'ls\', \'-l\', \'/\']
    proc = subprocess.Popen(
        cmd,
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
        # Make all end-of-lines \'\\n\'
        universal_newlines=True,
    )
    for line in unbuffered(proc):
        print line

example()

回答12:

Found this \"plug-and-play\" function here. Worked like a charm!

import subprocess

def myrun(cmd):
    \"\"\"from http://blog.kagesenshi.org/2008/02/teeing-python-subprocesspopen-output.html
    \"\"\"
    p = subprocess.Popen(cmd, shell=True, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    stdout = []
    while True:
        line = p.stdout.readline()
        stdout.append(line)
        print line,
        if line == \'\' and p.poll() != None:
            break
    return \'\'.join(stdout)

回答13:

Depending on the use case, you might also want to disable the buffering in the subprocess itself.

If the subprocess will be a Python process, you could do this before the call:

os.environ[\"PYTHONUNBUFFERED\"] = \"1\"

Or alternatively pass this in the env argument to Popen.

Otherwise, if you are on Linux/Unix, you can use the stdbuf tool. E.g. like:

cmd = [\"stdbuf\", \"-oL\"] + cmd

See also here about stdbuf or other options.

(See also here for the same answer.)

回答14:

The Streaming subprocess stdin and stdout with asyncio in Python blog post by Kevin McCarthy shows how to do it with asyncio:

import asyncio
from asyncio.subprocess import PIPE
from asyncio import create_subprocess_exec


async def _read_stream(stream, callback):
    while True:
        line = await stream.readline()
        if line:
            callback(line)
        else:
            break


async def run(command):
    process = await create_subprocess_exec(
        *command, stdout=PIPE, stderr=PIPE
    )

    await asyncio.wait(
        [
            _read_stream(
                process.stdout,
                lambda x: print(
                    \"STDOUT: {}\".format(x.decode(\"UTF8\"))
                ),
            ),
            _read_stream(
                process.stderr,
                lambda x: print(
                    \"STDERR: {}\".format(x.decode(\"UTF8\"))
                ),
            ),
        ]
    )

    await process.wait()


async def main():
    await run(\"docker build -t my-docker-image:latest .\")


if __name__ == \"__main__\":
    loop = asyncio.get_event_loop()
    loop.run_until_complete(main())