What is the cleanest way to add a directory of thi

Question

My context is appengine_config.py, but this is really a general Python question.

Given that we've cloned a repo of an app that has an empty directory lib in it, and that we populate lib with packages by using the command pip install -r requirements.txt --target lib, then:

dirname ='lib'
dirpath = os.path.join(os.path.dirname(__file__), dirname)

For importing purposes, we can add such a filesystem path to the beginning of the Python path in the following way (we use index 1 because the first position should remain '.', the current directory):

sys.path.insert(1, dirpath)

However, that won't work if any of the packages in that directory are namespace packages.

To support namespace packages we can instead use:

site.addsitedir(dirpath)

But that appends the new directory to the end of the path, which we don't want in case we need to override a platform-supplied package (such as WebOb) with a newer version.

The solution I have so far is this bit of code which I'd really like to simplify:

sys.path, remainder = sys.path[:1], sys.path[1:]
site.addsitedir(dirpath)
sys.path.extend(remainder)

Is there a cleaner or more Pythonic way of accomplishing this?

Answer 1

You may want to have a look at the answers in the Stack Overflow thread, "How do I manage third-party Python libraries with Google App Engine? (virtualenv? pip?)," but for your particular predicament with namespace packages, you're running up against a long-standing issue I filed against site.addsitedir's behavior of appending to sys.path instead of inserting after the first element. Please feel free to add to that discussion with a link to this use case.

I do want to address something else that you said that I think is misleading:

My context is appengine_config.py, but this is really a general Python question.

The question actually arises from the limitations of Google App Engine and the inability to install third-party packages, and hence, seeking a workaround. Rather than manually adjusting sys.path and using site.addsitedir. In general Python development, if your code uses these, you're Doing It Wrong.

The Python Packaging Authority (PyPA) describes the best practices to put third party libraries on your path, which I outline below:

1. Create a virtualenv
2. Mark out your dependencies in your setup.py and/or requirements files (see PyPA's "Concepts and Analyses")
3. Install your dependencies into the virtualenv with pip
4. Install your project, itself, into the virtualenv with pip and the -e/--editable flag.

Unfortunately, Google App Engine is incompatible with virtualenv and with pip. GAE chose to block this toolset in an attempt sandbox the environment. Hence, one must use hacks to work around the limitations of GAE to use additional or newer third party libraries.

If you dislike this limitation and want to use standard Python tooling for managing third-party package dependencies, other Platform as a Service providers out there eagerly await your business.

Answer 2

For this answer I assume you know how to use setuptools and setup.py.

Assuming you would like to use the standard setuptools workflow for development, I recommend using this code snipped in your appengine_config.py:

import os
import sys

if os.environ.get('CURRENT_VERSION_ID') == 'testbed-version':
    # If we are unittesting, fake the non-existence of appengine_config.
    # The error message of the import error is handled by gae and must
    # exactly match the proper string.
    raise ImportError('No module named appengine_config')


# Imports are done relative because Google app engine prohibits
# absolute imports.
lib_dir = os.path.join(os.path.dirname(os.path.realpath(__file__)), 'libs')
# Add every library to sys.path.
if os.path.isdir(lib_dir):
    for lib in os.listdir(lib_dir):
        if lib.endswith('.egg'):
            lib = os.path.join(lib_dir, lib)
            # Insert to override default libraries such as webob 1.1.1.
            sys.path.insert(0, lib)

And this piece of code in setup.cfg:

[develop]
install-dir = libs
always-copy = true

If you type python setup.py develop, the libraries are downloaded as eggs in the libs directory. appengine_config inserts them to your path.

We use this at work to include webob==1.3.1 and internal packages which are all namespaced using our company namespace.