Subprocess - using several command line tools

Question

I'm a learning newbie to python and to working in the command line, e.g. piping.

I've read that subprocess is encouraged way instead of os.system. I'm creating a script which invokes the shell and I have not been able to do it using subprocess. Using os.system was a snap though:

os.system("cut -f1-4 " + temp1.name + "| uniq --count | sort -rn > " + temp2.name)

I've used subprocess with success for other commands, but not those that combine more than one tools with "|". Reading the subprocess python documentation was confusing and not helpful to me. I've also tried searching other questions but could not find something similar to my problem. This is what I've tried (and failed):

subprocess.call = (["cut", "-f1-4", temp1.name, "|",  "uniq", "--count", "|", "sort". "-rn"], stdout = open(temp2.name, 'w'))

I've also tried substituting sp.call with sp.Popen, but failed. Can anyone please help with some clear examples and explanation? Thanks!

Answer 1

If you want to use pipes you should add shell=True

subprocess.check_output("cut -f1-4 " + temp1.name + "| uniq --count | sort -rn > " +    temp2.name, shell=True)

Please note that if temp1.name or temp2.name come from an untrusted source (e.g. from data supplied by a user in a web application) using shell=True is be a security risk.

Answer 2

It's worth taking a look at the great library python sh, it is a full-fledged subprocess interface for Python that allows you to call any program as if it were a function, and more important, it's pleasingly pythonic.

For your specific need in this case, it provides some 'advanced piping' feature, sth like this:

# the inner command executes first, then sends its data to the outer command
from sh import *
sort(uniq(cut("-f1-4", _in="temp1.name"), "--count"), "-rn", _out="temp2.name")