Why does shell=True eat my subprocess.Popen stdout

Question

It seems that using shell=True in the first process of a chain somehow drops the stdout from downstream tasks:

p1 = Popen(['echo','hello'], stdout=PIPE)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs correctly ('hello\n', None)

Making the first process use shell=True kills the output somehow...

p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)
p2 = Popen('cat', stdin=p1.stdout, stdout=PIPE)
p2.communicate()
# outputs incorrectly ('\n', None)

shell=True on the second process doesn't seem to matter. Is this expected behavior?

Answer 1

When you pass shell=True, Popen expects a single string argument, not a list. So when you do this:

p1 = Popen(['echo','hello'], stdout=PIPE, shell=True)

What happens is this:

execve("/bin/sh", ["/bin/sh", "-c", "echo", "hello"], ...)

That is, it calls sh -c "echo", and hello is effectively ignored (technically it becomes a positional argument to the shell). So the shell runs echo, which prints \n, which is why you see that in your output.

If you use shell=True, you need to do this:

  p1 = Popen('echo hello', stdout=PIPE, shell=True)