google app engine - how to add lib folder?

Question

I keep getting ImportError: No module named twitter when it's in my lib module

Details:

1. the twitter module is located in the lib directory.
2. The lib directory has an empty init.py
3. I'm getting the error on a page just outside the directory
4. When I take the twitter module out of the lib directory and just say import twitter, it works
5. When I put the twitter module back inside the lib and say from lib import twitter, or when I say import twitter, or saying import lib.twitter it says, go to **** and like it
6. I tried doing this: https://cloud.google.com/appengine/docs/python/tools/libraries27?hl=en , which suggests to:

To tell your app how to find libraries in this directory, create (or modify) a file named appengine_config.py in the root of your project, then add these lines:

from google.appengine.ext import vendor

# Add any libraries installed in the "lib" folder.
vendor.add('lib')

and then tried import twitter, and tried from lib import twitter, and it says, "haha, nice try."

Now what?

Answer 1

1. Create a folder called "libs" (in the home directory of your webapp)
2. If one library, then just do: pip install -t libs/ your_library_name
3. if multiple libraries, do: pip freeze > requirements.txt, pip install -r requirements.txt -t libs
4. Now, create a file called appengine_config.py (in the home directory, again)
5. In the above file, add: from google.appengine.ext import vendor vendor.add('libs')

These get uploaded onto your app engine environment and you'll be able to run it!

Answer 2

tl;dr - the comments are right. 'lib' is not a module, it's a folder containing modules. It needs to be on your Python import path before your import twitter, using the vendor.add('lib'), which should set up your import path correctly when you deploy or use dev_appserver.py. If you place the vendor.add(lib) snippet in appengine_config.py file and run dev_appserver.py or deploy, and the twitter module is inside lib, you should be able to import twitter. If you're running outside of App Engine/devappserver, you'll need to make sure lib is on the PYTHONPATH.

Here is the longer explanation of what the lib folder is; what vendoring is; and how they fit into the bigger picture of managing dependencies on App Engine.

With App Engine, you can specify some dependencies provided by the sandbox in app.yaml. You can se the list of available libraries here. For any other dependencies (which can only be pure Python libraries), the libraries need to be directly included inside your project. Directly including the code for these dependencies as part of your project (rather than just listing them in requirements.txt and expecting them to get installed at deploy time) is commonly called 'vendoring'.

Now, even though we are including dependencies directly, we'd rather just use pip and a requirements.txt, for a few reasons I will mention below. But pip is usually used to install into libraries into the system libraries or a virtualenv, not into the project itself. So a feature was added to pip and a feature was added to App Engine to fix this.

The pip feature install dependencies into a folder rather than your system libraries or a virtualenv. You use the -t flag like this:

pip install -r requirements.txt -t lib

Where 'lib' is the folder to install into. This way, you still specify your dependencies in requirements.txt, and use pip to install them, they just get installed right into the directory specified. As the comments noted, the lib folder is itself not a module - it just contains them. So your lib folder should not have an __init__.py, it should just contain a folder like 'twitter' that does have an __init__.py. Since lib is not a module, from lib import twitter doesn't actually make sense. Also note that lib is an arbitrary name, but the one we usually pick by convention.

There are a few big advantages to vendoring using pip rather than just manually downloading dependencies and adding them to the project. One advantage is you don't need to check the dependencies into source control - just add the requirements.txt, and tell other users to use the pip -t command to vendor the dependencies too. Another advantage is it's just more organized to cleanly separate your code from third-party modules. The point of the vendoring features is to keep these advantages while still following App Engine requirements to include the dependencies in the directory at deploy time.

The App Engine vendor extension which you are using was made to recognize that a folder contains modules that have been 'vendored' and add it to the path. That is the vendor extension you are using in your snippet. It makes sure the lib folder is on your import path so you can import the modules you installed into it.

Once you run the vendor.add command, you should be able to import your modules in the lib folder. As the comment notes, you need to make sure it runs before import twitter. A good practice is to make sure those vendor commands run before anything else. You can accomplish this by putting that code inside a file named appengine_config.py in your directory. That's a special file that runs before anything else. Take a look at an example of doing that here.

A few last notes that might be helpful:

• appengine_config.py will run in the GAE environment or when you run dev_appserver, which emulates the GAE environment. However if you're ever running outside of a GAE environment, make sure your PYTHONPATH includes the lib folder you want to import from.
• Since you can vendor libraries using pip, you might ask why would you ever include a dependency using app.yaml? The answer is that since only pure Python libraries can be vendored, you still should use app.yaml dependencies for any libraries which need C libraries, such as MySQL.
• If you have vendored libraries and app.yaml libaries, you don't want to vendor libraries that will be in the GAE sandbox since the versions might conflict. In that case, it's a good idea to have a separate requirements.txt for dependencies you want to vendor, and dependencies you want included only when running locally but that GAE will provide in the sandbox. Here is an example of that for MySQL.