Subprocess.Popen behaves differently in interprete

Question

Let's say you have the following:

command = shlex.split("mcf -o -q -e -w %s %s" % (SOLFILE, NETFILE))
task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print "stdout: %s" % stdout #debugging
print "stderr: %s" % stderr #debugging
if stderr:
    sys.exit("MCF crashed on %s" % NETFILE)

It's not necessary to know what mcf is, except that it's a C program which will overflow if it's not given a satisfiable netfile. (Why can't I just ensure that all the netfiles are satisfiable? Well, because the easiest way to check that is to feed it to mcf and see if it overflows...)

Anyway, when I run this in an executable script, task.communicate() doesn't seem to store anything in stdout and stderr. (To be precise, I get stdout == stderr == ''.) Instead, the stderr stream from mcf seems to be "leaking" to the terminal rather than getting captured by the subprocess pipe. Here's some sample output to illustrate:

Netfile: facility3cat_nat5000_wholesaler_capacitation_test_.net
Solfile: facility3cat_nat5000_wholesaler_capacitation_test_.sol
*** buffer overflow detected ***: mcf terminated
======= Backtrace: =========
...
...[fifty lines of Linda Blair-esque output]...
...
stdout: None
stderr: 
...[program continues, since stderr did not evaluate to True]...

This only fails when running the script from the command line. When I step through it line by line in the interpreter, stdout and stderr are correctly assigned:

>>> task = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
>>> stdout, stderr = task.communicate()
>>> stderr
'*** buffer overflow detected ***: mcf terminated\n======= Backtrace: =========\n'
...[more headspinning and vomit]...

Could anyone help me to understand why this works in the interpreter, but not when executed? Thanks in advance!

Answer 1

I wrote a little test script to test the subprocess module with.

#!/bin/bash

echo echo to stderr 1>&2
echo echo to stdout

Then I wrote a small Python script that calls it:

#!/usr/bin/python

import subprocess

command = ('./joe.sh',)
task = subprocess.Popen(command, stdout=subprocess.PIPE,
                                 stderr=subprocess.PIPE)
stdout, stderr = task.communicate()
print 'stdout == %r\nstderr == %r' % (stdout, stderr)

The output of running it looks just like this:

$ python joe.py 
stdout == 'echo to stdout\n'
stderr == 'echo to stderr\n'

The output of running that same sequence in ipython is the same.

So the subprocess module is behaving in the manner you expect, and not how it's behaving for you in your question. I think something other than the subprocess module must be at fault here because what you're doing works for me.

I'm running Python 2.7, so another possibility is that maybe there is some kind of weird bug in older versions of the subprocess module.