Get exit code and stderr from subprocess call

Question

I read up on the functions provided by subprocess - call, check_call, check_output, and understand how each works and differs in functionality from one another. I am currently using check_output, so I can have access to the stdout, and used "try block" to catch the exception, as follows:

# "cmnd" is a string that contains the command along with it's arguments. 
try:
    cmnd_output = check_output(cmnd, stderr=STDOUT, shell=True, timeout=3, universal_newlines=True);                         
except CalledProcessError:                                                                                                   
    print("Status : FAIL")                                                                                                   
print("Output: \n{}\n".format(cmnd_output))                                                                                  

The issue I am running into is when an exception is thrown, "cmnd_output" is not initialized and don't have access to stderr, and I get the following error message:

print("Output: \n{}\n".format(cmnd_output))
UnboundLocalError: local variable 'cmnd_output' referenced before assignment

I think thats because the exception causes the "check_output" to bail immediately without any further processing, aka assignment to "cmnd_output", in the try block. Please correct me if I am wrong.

Is there any way I can get access to stderr (it's ok if it's sent to stout) and have access to the exit code. I can manually check for pass/fail based on exit code with out the exception being throuwn.

Thank you, Ahmed.

Answer 1

Try this version:

import subprocess
try:
    output = subprocess.check_output(
        cmnd, stderr=subprocess.STDOUT, shell=True, timeout=3,
        universal_newlines=True)
except subprocess.CalledProcessError as exc:
    print("Status : FAIL", exc.returncode, exc.output)
else:
    print("Output: \n{}\n".format(output))

This way you will print the output only if the call was successful. In case of a CalledProcessError you print the return code and the output.

Answer 2

The accepted solution covers the case in which you are ok mixing stdout and stderr, but in cases in which the child process (for whatever reason) decides to use stderr IN ADDITION to stdout for a non failed output (i.e. to output a non-critical warning), then the given solution may not desirable.

For example, if you will be doing additional processing on the output, like converting to JSON, and you mix in the stderr, then the overall process will fail since the output will not be pure JSON because of the added stderr output.

I've found the following to work in that case:

cmd_args = ... what you want to execute ...

pipes = subprocess.Popen(cmnd, stdout=subprocess.PIPE, stderr=subprocess.PIPE)
std_out, std_err = pipes.communicate()

if pipes.returncode != 0:
    # an error happened!
    err_msg = "%s. Code: %s" % (std_err.strip(), pipes.returncode)
    raise Exception(err_msg)

elif len(std_err):
    # return code is 0 (no error), but we may want to
    # do something with the info on std_err
    # i.e. logger.warning(std_err)

# do whatever you want with std_out
# i.e. json.loads(std_out)

Answer 3

why not initialize the varible cmnd_output before the try statement? That way it will work the way you expect it to. Following line would work, just add it above try statement :

cmnd_output = ''

Answer 4

Both of the proposed solutions either mix the stdout/stderr, or use Popen which isn't quite as simple to use as check_output. However, you can accomplish the same thing, and keep stdout/stderr separate, while using check_output if you simply capture stderr by using a pipe:

import sys
import subprocess

try:
    subprocess.check_output(cmnd, stderr=subprocess.PIPE)
except subprocess.CalledProcessError as e:
    print('exit code: {}'.format(e.returncode))
    print('stdout: {}'.format(e.output.decode(sys.getfilesystemencoding())))
    print('stderr: {}'.format(e.stderr.decode(sys.getfilesystemencoding())))

In this example, since we captured stderr, it's available in the exception's stderr attribute (without capturing with the pipe, it would just be None).