GAE Python dev server crashes intermittently after

Question

I recently upgraded my GAE Python app to Python 2.7. Since then, I periodically get the following error with the dev server and the dev server serves up a blank page:

Traceback (most recent call last):
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 168, in Handle
    handler = _config_handle.add_wsgi_middleware(self._LoadHandler())
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/runtime/wsgi.py", line 206, in _LoadHandler
    handler = __import__(path[0])
  [...]
  File "/Users/joneill/OpenSTV/OpenSTV/trunk/OpaVote-HR/main.py", line 2, in <module>
    import views
  [...]
  File "/Users/joneill/OpenSTV/OpenSTV/trunk/OpaVote-HR/views.py", line 3, in <module>
    from pytz.gae import pytz
  [...]
  File "/Users/joneill/OpenSTV/OpenSTV/trunk/OpaVote-HR/pytz/__init__.py", line 34, in <module>
    from pkg_resources import resource_stream
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 662, in Decorate
    return func(self, *args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 1818, in load_module
    return self.FindAndLoadModule(submodule, fullname, search_path)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 662, in Decorate
    return func(self, *args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 1690, in FindAndLoadModule
    description)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 662, in Decorate
    return func(self, *args, **kwargs)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 1615, in LoadModuleRestricted
    return source_file.load_module(submodule_fullname)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist/py_zipimport.py", line 246, in load_module
    submodname, is_package, fullpath, source = self._get_source(fullmodname)
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/dist/py_zipimport.py", line 207, in _get_source
    source = self.zipfile.read(relpath.replace(os.sep, '/'))
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 867, in read
    return self.open(name, "r", pwd).read()
  File "/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/zipfile.py", line 882, in open
    zef_file = open(self.filename, 'rb')
  File "/Applications/GoogleAppEngineLauncher.app/Contents/Resources/GoogleAppEngine-default.bundle/Contents/Resources/google_appengine/google/appengine/tools/dev_appserver_import_hook.py", line 578, in __init__
    raise IOError(errno.EACCES, 'file not accessible', filename)
IOError: [Errno 13] file not accessible: '/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg'
INFO     2012-01-21 20:50:44,222 dev_appserver.py:2832] "POST /manage HTTP/1.1" 500 -

Some notes:

• This doesn't happen on the production server.
• On the dev server, my app will work for a few minutes and then this error happens.
• If I stop and restart my app on the dev server, it will work again for a few minutes.
• I am using the latest version of gae-pytz and you can see that it fails in an import there.
• The [...] that I removed are similar to the stuff you see near the end.
• I don't know why setuptools is being invoked at the end.
• I'm using a Mac with Lion.

I can use the dev server, but it is really annoying to stop and restart every few minutes. Any ideas how to fix this?

Answer 1

The problem is a bug in the App Engine dev server with Python 2.7. The solution is here: File not accesible error (setuptools) in logs

Answer 2

An alternative answer to the above that I prefer.

The __init__.py file for pytz contains the following lines:

#try:
#    from pkg_resources import resource_stream
#except ImportError:
resource_stream = None

I commented out the first three lines and that fixed the problem.

Answer 3

The actual problem from the stack trace, is your code is trying to import setup tools from site-packages, which the dev server won't do.

'/Library/Frameworks/Python.framework/Versions/2.7/lib/python2.7/site-packages/setuptools-0.6c11-py2.7.egg'

You will need to include setuptools in you application code base. The fact that it works sometimes, suggests that you code paths through various modules vary, and maybe (depending on what your tesing in dev) different import orders mean setup tools has been imported somewhere else, or is only required at certain points in your code.

Have a look at line 4th line of the stack trace where pytz is imported, the next line is from pkg_resources import resource_stream thats whats triggering the rest of the import issue. I use a fake truncated pkg_resources at the root of my project, that doesn't end up trying to import stuff from setup tools. You can run the dev server in debug import mode which will tell you a lot more

Here is a fake pkg_resources.

"""Package resource API
--------------------

A resource is a logical file contained within a package, or a logical
subdirectory thereof.  The package resource API expects resource names
to have their path parts separated with ``/``, *not* whatever the local
path separator is.  Do not use os.path operations to manipulate resource
names being passed into the API.

The package resource API is designed to work with normal filesystem packages,
.egg files, and unpacked .egg files.  It can also work in a limited way with
.zip files and with custom PEP 302 loaders that support the ``get_data()``
method.
"""

import sys, os, zipimport, time, re, imp, new

try:
    frozenset
except NameError:
   from sets import ImmutableSet as frozenset

from os import utime   #, rename, unlink    # capture these to bypass sandboxing
from os import open as os_open

There are probably other/better ways of doing this, but it works for me.

Oh, I would also suggest you use http://code.google.com/p/gae-pytz/ instead of pytz.

Cheers