I am trying to use Python's subprocess module. What I require is to send input to the first process whose output becomes the input of the second process.
The situation is basically almost the same as the example given in the documentation here:
http://docs.python.org/library/subprocess.html#replacing-shell-pipeline
except that I need to provide input the first command.
Here is that example copied:
p1 = Popen(["dmesg"], stdout=PIPE)
p2 = Popen(["grep", "hda"], stdin=p1.stdout, stdout=PIPE)
p1.stdout.close() # Allow p1 to receive a SIGPIPE if p2 exits.
output = p2.communicate()[0]
If we change the first line to:
p1 = Popen(["cat"], stdout=PIPE, stdin=PIPE)
How do I provide the input string to the process?
If I attempt it by changing the final line to:
output = p2.communicate(input=inputstring)[0]
This doesn't work.
I do have a working version, which just stores the output of the first command in a string and then passes that to the second command. This isn't terrible as there is essentially no concurrency that can be exploited (in my actual use case the first command will exit rather quickly and produce all of its output at the end).
Here is the working version in full:
import subprocess
simple = """Writing some text
with some lines in which the
word line occurs but others
where it does
not
"""
def run ():
catcommand = [ "cat" ]
catprocess = subprocess.Popen(catcommand,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(catout, caterr) = catprocess.communicate(input=simple)
grepcommand = [ "grep", "line" ]
grepprocess = subprocess.Popen(grepcommand,
stdin=subprocess.PIPE,
stdout=subprocess.PIPE,
stderr=subprocess.PIPE)
(grepout, greperr) = grepprocess.communicate(input=catout)
print "--- output ----"
print grepout
print "--- error ----"
print greperr
if __name__ == "__main__":
run()
I hope I've been clear enough, thanks for any help.
If you do
from subprocess import Popen, PIPE
p1 = Popen(["cat"], stdout=PIPE, stdin=PIPE)
You should do p1.communicate("Your Input to the p1") and that will flow through the PIPE.
The stdin is the process's input and you should communicate to that only.
The program which have given is absolutely fine, there seems no problem with that.
I assume that cat, grep are just example commands otherwise you could use a pure Python solution without subprocesses e.g.:
for line in simple.splitlines():
if "line" in line:
print(line)
Or if you want to use grep:
from subprocess import Popen, PIPE
output = Popen(['grep', 'line'], stdin=PIPE, stdout=PIPE).communicate(simple)[0]
print output,
You can pass the output of the first command to the second one without storing it in a string first:
from subprocess import Popen, PIPE
from threading import Thread
# start commands in parallel
first = Popen(first_command, stdin=PIPE, stdout=PIPE)
second = Popen(second_command, stdin=first.stdout, stdout=PIPE)
first.stdout.close() # notify `first` if `second` exits
first.stdout = None # avoid I/O on it in `.communicate()`
# feed input to the first command
Thread(target=first.communicate, args=[simple]).start() # avoid blocking
# get output from the second command at the same time
output = second.communicate()[0]
print output,
If you don't want to store all input/output in memory; you might need threads (to read/write in chunks without blocking) or a select loop (works on POSIX).
If there are multiple commands, it might be more readable just to use the shell directly as suggested by @Troels Folke or use a library such as plumbum that hides all the gory details of emulating the shell by hand.
Hmm, why not mix in a bit of (ba)sh? :-)
from subprocess import Popen, PIPE
cproc = Popen('cat | grep line', stdin=PIPE, stdout=PIPE, stderr=PIPE, shell=True)
out, err = cproc.communicate("this line has the word line in it")
BEWARE though:
This only works on systems that use a Bourne Shell compatible shell (like most *nix'es)
Usign shell=True and putting user input in the command string is a bad idea, unless you escape the user input first. Read the subprocess docs -> "Frequently Used Arguments" for details.
This is ugly, non portable, non pythonic and so on...
EDIT:
There is no need to use cat though, if all you want to do is grep. Just feed the input directly to grep, or even better, use python regular expressions.
