Actual meaning of 'shell=True' in subproce

Question

I am calling different processes with the subprocess module. However, I have a question.

In the following codes:

callProcess = subprocess.Popen(['ls', '-l'], shell=True)

and

callProcess = subprocess.Popen(['ls', '-l']) # without shell

Both work. After reading the docs, I came to know that shell=True means executing the code through the shell. So that means in absence, the process is directly started.

So what should I prefer for my case - I need to run a process and get its output. What benefit do I have from calling it from within the shell or outside of it.

Answer 1

>>> import subprocess
>>> subprocess.call('echo $HOME')
Traceback (most recent call last):
...
OSError: [Errno 2] No such file or directory
>>>
>>> subprocess.call('echo $HOME', shell=True)
/user/khong
0

Setting the shell argument to a true value causes subprocess to spawn an intermediate shell process, and tell it to run the command. In other words, using an intermediate shell means that variables, glob patterns, and other special shell features in the command string are processed before the command is run. Here, in the example, $HOME was processed before the echo command. Actually, this is the case of command with shell expansion while the command ls -l considered as a simple command.

source: Subprocess Module

Answer 2

The other answers here adequately explain the security caveats which are also mentioned in the subprocess documentation. But in addition to that, the overhead of starting a shell to start the program you want to run is often unnecessary and definitely silly for situations where you don't actually use any of the shell's functionality. Moreover, the additional hidden complexity should scare you, especially if you are not very familiar with the shell or the services it provides.

Wildcard expansion, variable interpolation, and redirection are all simple to replace with native Python constructs. A complex shell pipeline where parts or all cannot be reasonably rewritten in Python (specialized external tools, perhaps closed source?) would be the one situation where perhaps you could consider using the shell. You should still feel bad about it.

In the trivial case, simply replace

subprocess.Popen("command -with -options 'like this' and\\ an\\ argument", shell=True)

with

subprocess.Popen(['command', '-with','-options', 'like this', 'and an argument'])

Notice how the first argument is a list of strings to pass to execvp(), and how quoting strings and backslash-escaping shell metacharacters is generally not necessary (or useful, or correct).

As an aside, you very often want to avoid Popen if one of the simpler wrappers in the subprocess package does what you want. If you have a recent enough Python, you should probably use subprocess.run.

• With check=True it will fail if the command you ran failed.
• With stdout=subprocess.PIPE it will capture the command's output.
• Somewhat obscurely, with universal_newlines=True it will decode output into a proper Unicode string (it's just bytes in the system encoding otherwise, on Python 3).

If not, for many tasks, you want check_output to obtain the output from a command, whilst checking that it succeeded, or check_call if there is no output to collect.

I'll close with a quote from David Korn: "It's easier to write a portable shell than a portable shell script." Even subprocess.run('echo "$HOME"', shell=True) is not portable to Windows.

Answer 3

Executing programs through the shell means that all user input passed to the program is interpreted according to the syntax and semantic rules of the invoked shell. At best, this only causes inconvenience to the user, because the user has to obey these rules. For instance, paths containing special shell characters like quotation marks or blanks must be escaped. At worst, it causes security leaks, because the user can execute arbitrary programs.

shell=True is sometimes convenient to make use of specific shell features like word splitting or parameter expansion. However, if such a feature is required, make use of other modules are given to you (e.g. os.path.expandvars() for parameter expansion or shlex for word splitting). This means more work, but avoids other problems.

In short: Avoid shell=True by all means.

Answer 4

The benefit of not calling via the shell is that you are not invoking a 'mystery program.' On POSIX, the environment variable SHELL controls which binary is invoked as the "shell." On Windows, there is no bourne shell descendent, only cmd.exe.

So invoking the shell invokes a program of the user's choosing and is platform-dependent. Generally speaking, avoid invocations via the shell.

Invoking via the shell does allow you to expand environment variables and file globs according to the shell's usual mechanism. On POSIX systems, the shell expands file globs to a list of files. On Windows, a file glob (e.g., "*.*") is not expanded by the shell, anyway (but environment variables on a command line are expanded by cmd.exe).

If you think you want environment variable expansions and file globs, research the ILS attacks of 1992-ish on network services which performed subprogram invocations via the shell. Examples include the various sendmail backdoors involving ILS.

In summary, use shell=False.

Answer 5

An example where things could go wrong with Shell=True is shown here

>>> from subprocess import call
>>> filename = input("What file would you like to display?\n")
What file would you like to display?
non_existent; rm -rf / # THIS WILL DELETE EVERYTHING IN ROOT PARTITION!!!
>>> call("cat " + filename, shell=True) # Uh-oh. This will end badly...

Check the doc here: subprocess.call()