Python's subprocess does not interpret “~” as

Question

Section 17.1.1.1. of the documentation states:

If shell is True, the specified command will be executed through the shell. This can be useful if you are using Python primarily for the enhanced control flow it offers over most system shells and still want convenient access to other shell features such as shell pipes, filename wildcards, environment variable expansion, and expansion of ~ to a user’s home directory.

However, on cygwin, output from bash isn't the same as from Python's subprocess, viz:

Bash:

$ ls -ls ~rbarakx
total 0
0 drwxr-xr-x 1 Administrator None 0 Aug 21 17:54 bash
0 drwxr-xr-x 1 Administrator None 0 Jul 11 09:11 python

Python:

>>> subprocess.call(["ls","-ls","~rbarakx"],shell=True)
RCS  mecha.py  print_unicode.py  req.py  requests_results.html  selen.py
0

It looks as if subprocess.call is executing just ls.

Can you suggest why?

My Environment:
Python: Python 2.7.3 (default, Dec 18 2012, 13:50:09) [GCC 4.5.3] on cygwin
cygwin: CYGWIN_NT-6.1-WOW64 ... 1.7.22(0.268/5/3) ... i686 Cygwin
windows: Windows 7 Ultimate

Answer 1

On Windows, the shell (cmd.exe) doesn't support the expansion of ~ to a user's home directory. Nor wildcard expansion, for that matter. The Python doc is very UNIX/Linux-oriented on this point.

"But wait!" I hear you cry. "I've installed cygwin, I have bash!" Sure you have, but you can hardly expect Python to know that. You're on Windows, so it's going to use the Windows shell. If it didn't, Python scripts that expected the shell to be cmd.exe would be mighty confused.

Answer 2

Interestingly when passing the command as a string instead of a tuple, it seems to work as if you were executing it in bash.

subprocess.call(["ls -ls ~"], shell=True)

If you insist on using tuple, consider the following workaround:

cmd = subprocess.list2cmdline(["ls","-ls","~rbarakx"])
subprocess.call(cmd, shell=True)

or this:

from os.path import expanduser
subprocess.call(["ls","-ls", expanduser("~rbarakx")])

Answer 3

It looks as if subprocess.call is executing just ls.

Can you suggest why?

Because on Unix (cygwin) your call is equivalent to:

# WRONG: DON'T DO IT
rc = subprocess.call(["/bin/sh", "-c"] + ["ls","-ls","~rbarakx"])

In other words the argument list is passed to the shell itself i.e., ls doesn't see -ls or ~rbarakx they are considerred /bin/sh arguments.

What you want is:

rc = subprocess.call(["/bin/sh", "-c"] + ["ls -ls ~rbarakx"])

Or using shell=True:

rc = subprocess.call("ls -ls ~rbarakx", shell=True)

Here's the quote from subprocess docs for reference:

On Unix with shell=True, the shell defaults to /bin/sh. If args is a string, the string specifies the command to execute through the shell. This means that the string must be formatted exactly as it would be when typed at the shell prompt. This includes, for example, quoting or backslash escaping filenames with spaces in them. If args is a sequence, the first item specifies the command string, and any additional items will be treated as additional arguments to the shell itself. That is to say, Popen does the equivalent of:

Popen(['/bin/sh', '-c', args[0], args[1], ...])

_{emphasis is mine}

You could also execute the command without any shell:

import os
import subprocess

rc = subprocess.call(["ls", "-ls", os.path.expanduser("~rbarakx")])