Python subprocess with heredocs

Question

I was playing around with Python's subprocess module, trying a few examples but I can't seem to get heredoc statements to work.

Here is the trivial example I was playing with:

import subprocess
a = "A String of Text"
p = subprocess.Popen(["cat", "<<DATA\n" + a + "\nDATA"])

I get the following error when I run the code above:

cat: <<DATA\nA String of Text\nDATA: No such file or directory

Am I doing it wrong? Is this even possible? If so how would I go about doing it?

---

Update

Just wanted to say that this should never be performed in a real python program because there are better ways of doing this.

Answer 1

As others have pointed out, you need to run it in a shell. Popen makes this easy with a shell=True argument. I get the following output:

>>> import subprocess
>>> a = "A String of Text"
>>> p = subprocess.Popen("cat <<DATA\n" + a + "\nDATA", shell=True)
>>> A String of Text

>>> p.wait()
0

Answer 2

You're passing shell syntax as an arguments to cat program. You can try to do it like that:

p = subprocess.Popen(["sh", "-c", "cat <<DATA\n" + a + "\nDATA"])

But the concept itself is wrong. You should use Python features instead of calling shell scripts inside your python scripts.

And in this particular case you should that shell's heredoc syntax interpolates variables, so you'll need to escape all the text inside a and make sure there's no DATA line in it.

---

For Python equivalent, I think the closest idea to this (assuming you don't want just print(a) ;-)) is passing the value of the variable to stdin of a spawned process:

p = subprocess.Popen(["program", ...], stdin=subprocess.PIPE)
p.communicate(a)

Answer 3

The shell "heredoc" support is a shell feature. subprocess.Popen does not run your command through the shell by default, so this syntax certainly won't work.

However, since you're using pipes anyway, there isn't any need to use the heredoc support of the shell. Just write your string a to the stdin pipe of the process you just started. This is exactly what the shell would do with the heredoc anyway.

You can do this with Popen.communicate():

p.communicate(a)

The return value of the communicate() function contains the output of the process (in two streams, see the docs).