reading a os.popen(command) into a string

Question

I'm not to sure if my title is right. What I'm doing is writing a python script to automate some of my code writing. So I'm parsing through a .h file. but I want to expand all macros before I start. so I want to do a call to the shell to:

gcc -E myHeader.h

Which should out put the post preprocessed version of myHeader.h to stdout. Now I want to read all that output straight into a string for further processing. I've read that i can do this with popen, but I've never used pipe objects.

how do i do this?

Answer 1

import subprocess

p = subprocess.popen('gcc -E myHeader.h'.split(),
                     stdout=subprocess.PIPE)
preprocessed, _ = p.communicate()

String preprocessed now has the preprocessed source you require -- and you've used the "right" (modern) way to shell to a subprocess, rather than old not-so-liked-anymore os.popen.

Answer 2

Here is another approach which captures both regular output plus error output:

com_str = 'uname -a'
command = subprocess.Popen([com_str], stdout=subprocess.PIPE, shell=True)
(output, error) = command.communicate()
print output

Linux 3.11.0-20-generic  Fri May 2 21:32:55 UTC 2014 GNU/Linux

and

com_str = 'id'
command = subprocess.Popen([com_str], stdout=subprocess.PIPE, shell=True)
(output, error) = command.communicate()
print output

uid=1000(myname) gid=1000(myGID) groups=1000(mygrp),0(root)

Answer 3

The os.popen function just returns a file-like object. You can use it like so:

import os

process = os.popen('gcc -E myHeader.h')
preprocessed = process.read()
process.close()

As others have said, you should be using subprocess.Popen. It's designed to be a safer version of os.popen. The Python docs have a section describing how to switch over.

Answer 4

The os.popen() has been deprecated since Python 2.6. You should now use the subprocess module instead: http://docs.python.org/2/library/subprocess.html#subprocess.Popen

import subprocess

command = "gcc -E myHeader.h"  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None, shell=True)

#Launch the shell command:
output = process.communicate()

print output[0]

In the Popen constructor, if shell is True, you should pass the command as a string rather than as a sequence. Otherwise, just split the command into a list:

command = ["gcc", "-E", "myHeader.h"]  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=None)

If you need to read also the standard error, into the Popen initialization, you can set stderr to subprocess.PIPE or to subprocess.STDOUT:

import subprocess

command = "gcc -E myHeader.h"  # the shell command
process = subprocess.Popen(command, stdout=subprocess.PIPE, stderr=subprocess.PIPE, shell=True)

#Launch the shell command:
output, error = process.communicate()

Answer 5

you should use subprocess.Popen() there are numerous examples on SO

How to get output from subprocess.Popen()