append subprocess.Popen output to file?

Question

I can successfully redirect my output to a file, however this appears to overwrite the file's existing data:

import subprocess
outfile = open('test','w') #same with "w" or "a" as opening mode
outfile.write('Hello')
subprocess.Popen('ls',stdout=outfile)

will remove the 'Hello' line from the file.

I guess a workaround is to store the output elsewhere as a string or something (it won't be too long), and append this manually with outfile.write(thestring) - but I was wondering if I am missing something within the module that facilitates this.

Answer 1

Are data in file really overwritten? On my Linux host I have the following behavior: 1) your code execution in the separate directory gets:

$ cat test
test
test.py
test.py~
Hello

2) if I add outfile.flush() after outfile.write('Hello'), results is slightly different:

$ cat test
Hello
test
test.py
test.py~

But output file has Hello in both cases. Without explicit flush() call stdout buffer will be flushed when python process is terminated. Where is the problem?

Answer 2

Well the problem is if you want the header to be header, then you need to flush before the rest of the output is written to file :D

Answer 3

You sure can append the output of subprocess.Popen to a file, and I make a daily use of it. Here's how I do it:

log = open('some file.txt', 'a')  # so that data written to it will be appended
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

(of course, this is a dummy example, I'm not using subprocess to list files...)

By the way, other objects behaving like file (with write() method in particular) could replace this log item, so you can buffer the output, and do whatever you want with it (write to file, display, etc) [but this seems not so easy, see my comment below].

Note: what may be misleading, is the fact that subprocess, for some reason I don't understand, will write before what you want to write. So, here's the way to use this:

log = open('some file.txt', 'a')
log.write('some text, as header of the file\n')
log.flush()  # <-- here's something not to forget!
c = subprocess.Popen(['dir', '/p'], stdout=log, stderr=log, shell=True)

So the hint is: do not forget to flush the output!