Running shell commands from Python and printing th

Question

I want to write a function that will execute multiple shell commands one at a time and print what the shell returns in real time.

I currently have the following code which does not print the shell (I am using Windows 10 and python 3.6.2):

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
                     stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
    p.stdin.write((command + "\n").encode("utf-8"))
p.stdin.close()
p.stdout.read()

How can I see what the shell returns in real time ?

Edit : This question is not a duplicate of the two first links in the comments, they do not help printing in real time.

Answer 1

Assuming you want control of the output in your python code you might need to do something like this

import subprocess

def run_process(exe):
    'Define a function for running commands and capturing stdout line by line'
    p = subprocess.Popen(exe.split(), stdout=subprocess.PIPE, stderr=subprocess.STDOUT)
    return iter(p.stdout.readline, b'')


if __name__ == '__main__':
    commands = ["foo", "foofoo"]
    for command in commands:
        for line in run_process(command):
            print(line)

Answer 2

I believe you need something like this

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", shell=True, stdin=subprocess.PIPE, \
                 stdout=subprocess.PIPE, stderr=subprocess.PIPE)
for command in commands:
    p.stdin.write((command + "\n").encode("utf-8"))
out, err = p.communicate()
print("{}".format(out))
print("{}".format(err))

Answer 3

It is possible to handle stdin and stdout in different threads. That way one thread can be handling printing the output from stdout and another one writing new commands on stdin. However, since stdin and stdout are independent streams, I do not think this can guarantee the order between the streams. For the current example it seems to work as intended, though.

import subprocess
import threading

def stdout_printer(p):
    for line in p.stdout:
        print(line.rstrip())

commands = ["foo", "foofoo"]
p = subprocess.Popen("cmd.exe", stdin=subprocess.PIPE, 
                     stdout=subprocess.PIPE, stderr=subprocess.STDOUT,
                     universal_newlines=True)

t = threading.Thread(target=stdout_printer, args=(p,))
t.start()

for command in commands:
    p.stdin.write((command + "\n"))
    p.stdin.flush()

p.stdin.close()
t.join()

Also, note that I am writing stdout line by line, which is normally OK, since it tends to be buffered and being generated a line (or more) at a time. I guess it is possible to handle an unbuffered stdout stream (or e.g. stderr) character-by-character instead, if that is preferable.