python subprocess Popen environment PATH?

问题:

I'm confused about how subprocess searches for the executable when using Popen(). It works if given absolute paths to the child process, but I'm trying to use relative paths. I've found that if I set the environment variable PYTHONPATH then I can get imported modules from that path ok, and PYTHONPATH is there in sys.path, but it doesn't seem to help with the behaviour of subprocess.Popen. I've also tried editing the sitecustomize.py file adding PYTHONPATH to os.environ, like so

# copy PYTHONPATH environment variable into PATH to allow our stuff to use
# relative paths for subprocess spawning
import os
if os.getenv('PYTHONPATH') is not None and os.getenv('PATH') is not none:
    os.environ['PATH'] = ':'.join([os.getenv('PATH'), os.getenv('PYTHONPATH')])

and verified that when starting up python , either interactively, with ipython, or by running a script from the command line, that PYTHONPATH is successfully appearing in os.environ. However, subrocess.Popen still doesn't search there for the executable. I thought it was supposed to inherit the parents environment, if no env kwarg is specified? Next I tried giving env explicitly, first by making a copy of os.getenv and secondly just by giving env={'PATH': '/explicit/path/to/search/from'}, and it still does not find the executable. Now I'm stumped.

Hopefully an example will help explain my problem more clearly:

/dir/subdir1/some_executable
/dir/subdir2/some_script.py

# some_script.py
from subprocess import Popen, PIPE
spam, eggs = Popen(['../subdir1/some_executable'], stdout=PIPE, stderr=PIPE).communicate()

If I'm in /dir/subdir2 and I run python some_script.py it works, but if I'm in /dir and I run python subdir2/some_script.py even though /dir/subdir2 is in the os.environ['PATH'], then subprocess will throw OSError: [Errno 2] No such file or directory.

回答1:

(filling in details from a comment to make a separate answer)

First off, relative paths (paths containing slashes) never get checked in any PATH, no matter what you do. They are relative to the current working directory only. If you need to resolve relative paths, you will have to search the PATH manually, or munge the PATH to include the subdirectories and then just use the command name as in my suggestion below.

If you want to run a program relative to the location of the Python script, use __file__ and go from there to find the absolute path of the program, and then use the absolute path in Popen.

Secondly, there is an issue in the Python bug tracker about how Python deals with bare commands (no slashes). Basically, on Unix/Mac Popen uses os.execvp when invoked with shell=False, which means it looks at the value of PATH as it was when Python launched and no amount of changing os.environ will help you fix that. Also, on Windows with shell=False, it pays no attention to PATH at all, and will only look in relative to the current working directory.

If you JUST need path evaluation and don't really want to run your command line through a shell, and are on UNIX, I advise using env instead of shell=True, as in Popen(['/usr/bin/env', 'progtorun', other, args], ...). This lets you pass a different PATH to the env process, which will use it to find the program. It also avoids issues with shell metacharacters and potential security issues with passing arguments through the shell. Obviously, on Windows (pretty much the only platform without a /usr/bin/env) you will need to do something different.

回答2:

You appear to be a little confused about the nature of PATH and PYTHONPATH.

PATH is an environment variable that tells the OS shell where to search for executables.

PYTHONPATH is an environment variable that tells the Python interpreter where to search for modules to import. It has nothing to do with subprocess finding executable files.

Due to the differences in the underlying implementation, subprocess.Popen will only search the path by default on non-Windows systems (Windows has some system directories it always searches, but that's distinct from PATH processing). The only reliable cross-platform way to scan the path is by passing shell=True to the subprocess call, but that has its own issues (as detailed in the Popen documentation)

However, it appears your main problem is that you are passing a path fragment to Popen rather than a simple file name. As soon as you have a directory separator in there, you're going to disable the PATH search, even on a non-Windows platform (e.g. see the Linux documentation for the exec family of functions).

回答3:

A relative path in subprocess.Popen acts relative to the current working directory, not the elements of the systems PATH. If you run python subdir2/some_script.py from /dir than the expected executable location will be /dir/../subdir2/some_executable, a.k.a /subdir2/some_executable.

If you would definitely like to use relative paths from a scripts own directory to a particular executable the best option would be to first construct an absolute path from the directory portion of the __file__ global variable.

#/usr/bin/env python
from subprocess import Popen, PIPE
from os.path import abspath, dirname, join
path = abspath(join(dirname(__file__), '../subdir1/some_executable'))
spam, eggs = Popen(path, stdout=PIPE, stderr=PIPE).communicate()

回答4:

The pythonpath is set to the path from where the python interpreter is executed. So, in second case of your example, the path is set to /dir and not /dir/subdir2 That's why you get an error.