Proper initialization of sys.path when main script

Question

The first entry of sys.path is the directory of the current script, according to the docs. In the following setup, I would like to change this default. Imagine the following directory structure:

src/
    core/
        stuff/
        tools/
            tool1.py
            tool2.py
    gui/
        morestuff/
        gui.py

The scripts tool*.py and gui.py are intended to be run as scripts, like the following:

python src/core/tools/tool2.py
python src/gui/gui.py

Now all tools import from src.core.stuff, and the GUI needs gui.morestuff. This means that sys.path[0] should point to src/, but it points to src/core/tools/ or src/gui/ by default.

I can adjust sys.path[0] in every script (with a construct like the following, e.g., at the beginning of gui.py):

if __name__ == '__main__':
    if sys.path[0]: sys.path[0] = os.path.dirname(os.path.abspath(sys.path[0]))

However, this is sort of redundant, and it becomes tedious for a mature code base with thousands of scripts. I also know the -m switch:

python -m gui.gui

But this requires the current directory to be src/.

Is there a better way to achieve the desired result, e.g. by modifying the __init__.py files?

EDIT: This is for Python 2.7:

~$ python -V
Python 2.7.3

Answer 1

I wanted to do this to avoid having to set PYTHONPATH in the first place

There are other places you can hook into Python's sys.path initialization, using the site module, which is (by default) automatically imported when Python initializes.

Based on the this code in site.py...

# Prefixes for site-packages; add additional prefixes like /usr/local here
PREFIXES = [sys.prefix, sys.exec_prefix]

...it looks as if the intention was that this file was designed to be modified after installation, which is one option, although it also provides other ways you can influence sys.path, e.g. by placing a .pth file somewhere inside your site-packages directory.

Assuming the desired result is to make the code work 'out of the box', this would work, but only for all users on a single system.

If you need it to work on multiple systems, then you'd have to apply the same changes to all systems.

For deployment, this is no big deal. Indeed, many Python packages already do something like this. e.g. on Ubuntu...

~$ dpkg -L python-imaging | grep pth
/usr/share/pyshared/PIL.pth
/usr/lib/python2.7/dist-packages/PIL.pth

...but if your intention is to make it easy for multiple concurrent developers, each using their own system, you may be better off sticking with the current option of adding some 'boilerplate' code to every Python module which is intended to be run as a script.

There may be another option, but it depends on exactly what you're trying to achieve.

Answer 2

The only officially approved way to run a script that is in a package is by using the -m flag. While you could run a script directly and try to do sys.path manipulations yourself in each script, it's likely to be a big pain. If you move a script between folders, the logic for rewriting sys.path may also need to be changed to reflect the new location. Even if you get sys.path right, explicit relative imports will not work correctly.

Now, making python -m mypackage.mymodule work requires that either you be in the project's top level folder (src in your case), or for that top level folder to be on the Python search path. Requiring you to be in a specific folder is awkward, and you've said that you don't want that. Getting src into the search path is our goal then.

I think the best approach is to use the PYTHONPATH environment variable to point the interpreter to your project's src folder so that it can find your packages from anywhere.

This solution is simple to set up (the environment variable can be be set automatically in your .profile, .bashrc or some other equivalent place), and will work for any number of scripts. If you move your project, just update your environment settings and you'll be all set, without needing to do any more work for each script.

Answer 3

You've got three basic options here. I've been through all three in both a production environment and personal projects. In many ways they build on each other. However, my advice is to just skip to the last one.

The fundamental problem is that you need your ./src directory to be in the python search path. This is really what python packaging is all about.

PYTHONPATH

The most straightforward, user defined way to adjust your python path is through the environment variable PYTHONPATH. You can set it at run time, doing something like:

PYTHONPATH=/src python src/gui/gui.py

You can of course also set this up in your global environment so hopefully all processes that need it will find the correct PYTHONPATH. But, just remember, you'll always forget one. Usually at 3 AM when your cron task finally runs.

Site Packages

To avoid needing an environment variable, your options are pretty much to include your software in an existing entry in the source path, or find some additional way to add a new search path. So this can mean dropping the contents of your src directory into /usr/lib/python2.7/site-packages or wherever your system site-packages is located.

Since you may not want to actually include the code in site-packages, you can create a symlink for your two sub-packages.

This is of course less than ideal for a number of reasons. If you're not careful with naming then suddenly every python program on the machine is exposed to potential name conflicts. You're exposing your software to every user on the machine. You might run into issues if python get's updated. If you add a new sub-package, now you have to create a new symlink.

A slightly better approach is to include a .pth file somewhere in your site-packages. When python encounters these files, it adds the contents (which is supposed to be the name of a directory) to the search path. This avoids the problem of having to remember to add a new symlink for each new sub-package.

virtualenv and packaging

The best solution is to just bite the bullet and do real python packaging. This, combined with great tools like virtualenv and pip let you have an isolated (or semi-isolated) python environment.

Under virtualenv, you would have a custom site-packages for just your project where you can easily install your software into it, avoiding all the problems of the earlier solutions. virtualenv also makes it easy to maintain executable scripts so that the python environment it runs under is exactly as you expect.

The one downside is that you have to write and maintain a setup.py which will instruct pip (the python installer) to include your software in the virtualenv. The contents would be something like:

!/usr/bin/env python
# -*- coding: utf-8 -*-

from distutils.core import setup


setup(
    name='myproject',
    package_dir={'myproject': 'src'},
    scripts=['src/gui/gui.py', 'src/core/tools/tool1.py', 'src/core/tools/tool2.py']
)

So, to setup this environment, it's going to look something like this:

virtualenv env
env/bin/pip install -e setup.py

To run your script, then you'd just do something like:

env/bin/tool1.py