What's the difference between subprocess.Popen

Question

I cannot get it it's bash related or python subprocess, but results are different:

>>> subprocess.Popen("echo $HOME", shell=True, stdout=subprocess.PIPE).communicate()
(b'/Users/mac\n', None)
>>> subprocess.Popen(["echo", "$HOME"], shell=True, stdout=subprocess.PIPE).communicate()
(b'\n', None)

Why in second time it's just newline? Where argument are falling off?

Answer 1

When you have shell=True, actual process that runs is the shell process i.e., think of it running /bin/sh -c on unix. The arguments you pass to Popen are passed as arguments to this shell process. So /bin/sh -c 'echo' '$HOME' prints newline and the second argument is ignored. So usually you should only use string arguments with shell=True.

Answer 2

The first argument to subprocess.Popen() tells the system what to run.

When it is a list, you need to use shell=False. It coincidentally happens to work as you hope in Windows; but on Unix-like platforms, you are simply passing in a number of arguments which will typically get ignored. Effectively,

/bin/sh -c 'echo' '$HOME'

which simply causes the second argument to not be used for anything (where I use single quotes to emphasize that these are just static strings).

In my humble opinion, Python should throw an error in this case. On Windows, too. This is an error which should be caught and reported.

(In the opposite case, where shell=False is specified but the string you pass in is not the name of a valid command, you will get an error eventually anyway, and it makes sense if you have even a vague idea of what's going on.)

If you really know what you are doing, you could cause the first argument to access subsequent arguments; for example

/bin/sh -c 'printf "%s\n" "$@"' 'ick' 'foo' 'bar' 'baz'

would print foo, bar, and baz on separate lines. (The "zeroth" argument - here, 'ick' - is used to populate $0.) But this is just an obscure corollary; don't try to use this for anything.

As a further aside, you should not use subprocess.Popen() if you just want a command to run. The subprocess.run() documentation tells you this in some more detail. With text=True you get a string instead of bytes.

result = subprocess.run('echo "$HOME"', shell=True,
    text=True, capture_output=True, check=True)
print(result.stdout, result.stderr)

And of course, os.environ['HOME'] lets you access the value of $HOME from within Python. This also allows you to avoid shell=True which you usually should if you can.

Answer 3

In the documentation found on https://docs.python.org/2/library/subprocess.html#popen-constructor, if you look at the shell argument you will find

The shell argument (which defaults to False) specifies whether to use the shell as the program to execute. If shell is True, it is recommended to pass args as a string rather than as a sequence.

Which means that when you execute the second command it runs as echo and hence you get just a new line.