I have a small Python program that executes a terminal command from a pip package called commandwrapper (which is a wrapper for subprocess.popen: https://pypi.python.org/pypi/commandwrapper/0.7). I am also trying to capture the real-time output to the console and to a file.
I have the code:
class Tee(object):
def __init__(self, *files):
self.files = files
def write(self, obj):
for f in self.files:
f.write(obj)
f.flush()
def flush(self) :
for f in self.files:
f.flush()
# Set the stdout/stderr to Tee()
out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)
process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()
# Stream the output to the console and stdoutfiles
while cmd.poll() is None:
msg_out = cmd.stdout.readline()
sys.stdout.write(msg_out)
sys.stdout.flush()
out.close()
This works perfect when I run it in PyCharm. The output of command is written to the file AND displayed on the terminal console in real-time.
However, when I run the same code in a terminal, no output is displayed on the console. How can this be? The stdout is correctly captured in a file, but nothing is written to the console.
Can anyone see any reason that this code would work well and as expected in PyCharm, but not display any stdout to the terminal? I'm at a loss here. If anything, I could deal with it if the behavior was reversed.
Using OSX Yosemite and running bash.
You need to change the logic where you have poll, I used Popen but you can use the wrapper if you prefer:
out = open(stdout_log, 'w')
stdout_reset = sys.stdout
sys.stdout = Tee(sys.stdout, out)
from subprocess import Popen,PIPE,STDOUT
process = Popen([list_of_cmds],stdout=PIPE,stderr=STDOUT)
# Stream the output to the console and stdoutfiles
for line in iter(process.stdout.readline,""):
sys.stdout.write(line)
out.close()
Applying the same logic works with the commandwrapper lib:
process = commandwrapper.WrapCommand(command, shell=True) # Also tried False with same behaivor.
cmd = process.prepareToRun()
# Stream the output to the console and stdoutfiles
for line in iter(cmd.stdout.readline,""):
sys.stdout.write(line)
Even if the child process has exited i.e., cmd.poll() is not None there could be a buffered output left in the pipe. You could see it if you call cmd.stdout.read() after the while cmd.poll() is not None loop has ended.
To reproduce the error without Tee, commandwrapper:
#!/usr/bin/env python
import sys
import time
from subprocess import Popen, PIPE
#XXX DO NOT USE THE CODE -- ITS PURPOSE IS TO DEMONSTRATE THE ISSUE
p = Popen([sys.executable, '-c', "import os; os.write(1, b'abc')"],
stdout=PIPE, bufsize=0) # no buffering in the parent
time.sleep(0.3) # allow the child to exit
while p.poll() is None: # poll() should return non-None value
assert 0, "it is never run unless the computer is slow"
line = p.stdout.readline()
print(line)
output = p.stdout.read() #
print("the output left after the child has died: %r" % (output,))
assert len(output) >= 3
See these posts on how to read subprocess' output in real time and how to redirect it to a file and terminal at the same time:
- Python: read streaming input from subprocess.communicate()
- How do I push a subprocess.call() output to terminal and file?
- Subprocess.Popen: cloning stdout and stderr both to terminal and variables
- Python subprocess get children's output to file and terminal?
