GDAL in flexible environment

Question

In app engine flexible environment, I'm trying to run django on postgresql db using GIS extensions. Can run locally, and it eventually finished deploying to cloud successfully. However, when run from cloud I get a "ERROR: Server Error" and the log file has the trace shown below.

Essentially, it says it cannot read files in the folder "/env/lib/python3.5/site-packages/django/contrib/gis/gdal". That folder is definitely there locally. Here's the full trace:

Traceback (most recent call last):
 File "/env/lib/python3.5/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
  worker.init_process()
 File "/env/lib/python3.5/site-packages/gunicorn/workers/base.py", line 126, in init_process
  self.load_wsgi()
 File "/env/lib/python3.5/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
  self.wsgi = self.app.wsgi()
 File "/env/lib/python3.5/site-packages/gunicorn/app/base.py", line 67, in wsgi
  self.callable = self.load()
 File "/env/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
  return self.load_wsgiapp()
 File "/env/lib/python3.5/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
  return util.import_app(self.app_uri)
 File "/env/lib/python3.5/site-packages/gunicorn/util.py", line 352, in import_app
  __import__(module)
 File "/home/vmagent/app/mysite/wsgi.py", line 22, in <module>
  application = get_wsgi_application()
 File "/env/lib/python3.5/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
  django.setup(set_prefix=False)
 File "/env/lib/python3.5/site-packages/django/__init__.py", line 27, in setup
  apps.populate(settings.INSTALLED_APPS)
 File "/env/lib/python3.5/site-packages/django/apps/registry.py", line 108, in populate
  app_config.import_models()
 File "/env/lib/python3.5/site-packages/django/apps/config.py", line 202, in import_models
  self.models_module = import_module(models_module_name)
 File "/env/lib/python3.5/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
 File "/env/lib/python3.5/site-packages/django/contrib/auth/models.py", line 4, in <module>
  from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
 File "/env/lib/python3.5/site-packages/django/contrib/auth/base_user.py", line 52, in <module>
  class AbstractBaseUser(models.Model):
 File "/env/lib/python3.5/site-packages/django/db/models/base.py", line 124, in __new__
  new_class.add_to_class('_meta', Options(meta, app_label))
 File "/env/lib/python3.5/site-packages/django/db/models/base.py", line 331, in add_to_class
  value.contribute_to_class(cls, name)
 File "/env/lib/python3.5/site-packages/django/db/models/options.py", line 214, in contribute_to_class
  self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
 File "/env/lib/python3.5/site-packages/django/db/__init__.py", line 33, in __getattr__
  return getattr(connections[DEFAULT_DB_ALIAS], item)
 File "/env/lib/python3.5/site-packages/django/db/utils.py", line 211, in __getitem__
  backend = load_backend(db['ENGINE'])
 File "/env/lib/python3.5/site-packages/django/db/utils.py", line 115, in load_backend
  return import_module('%s.base' % backend_name)
 File "/env/lib/python3.5/importlib/__init__.py", line 126, in import_module
  return _bootstrap._gcd_import(name[level:], package, level)
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
  from .features import DatabaseFeatures
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
  from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/backends/base/features.py", line 4, in <module>
  from django.contrib.gis.db.models import aggregates
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
  from django.contrib.gis.db.models.aggregates import * # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/aggregates.py", line 1, in <module>
  from django.contrib.gis.db.models.fields import ExtentField
 File "/env/lib/python3.5/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
  from django.contrib.gis import forms, gdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
  from .fields import ( # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/forms/fields.py", line 4, in <module>
  from django.contrib.gis.geos import GEOSException, GEOSGeometry
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module>
  from .collections import ( # NOQA
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/collections.py", line 11, in <module>
  from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin
 File "/env/lib/python3.5/site-packages/django/contrib/gis/geos/geometry.py", line 11, in <module>
  from django.contrib.gis import gdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
  from django.contrib.gis.gdal.datasource import DataSource
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
  from django.contrib.gis.gdal.driver import Driver
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
  from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
  from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
 File "/env/lib/python3.5/site-packages/django/contrib/gis/gdal/libgdal.py", line 49, in <module>
  lgdal = CDLL(lib_path)
 File "/opt/python3.5/lib/python3.5/ctypes/__init__.py", line 351, in __init__
  self._handle = _dlopen(self._name, mode)
OSError: /env/lib/python3.5/site-packages/django/contrib/gis/gdal: cannot read file data: Is a directory

Developed on ubuntu 16.04.

I started with simple "pip install django" inside virtualenv, which created the "django/contrib/gis/gdal" folder that says is missing on deploy. I've tried several others ways of installing GDAL to make sure there's no missing header file. Also upgraded to Google Cloud SDK 171 and gsutil 4.27. Have tried including GDAL==2.1.3 in requirements.txt, but then got a build error, so removed it. I also tried "pip freeze" to make sure requirements.txt had all of the needed packages.

Finally, I started again from scratch following this tutorial: https://cloud.google.com/python/django/flexible-environment. I didn't test on a local postgreSQL db, but went directly to the Cloud SQL instance previously used (which had the tables needed). And this time I used python2.7.

It still runs fine locally, but is now missing the folder "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal" in the stack trace. Here's the full trace:

Traceback (most recent call last):
 File "/env/local/lib/python2.7/site-packages/gunicorn/arbiter.py", line 578, in spawn_worker
  worker.init_process()
 File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 126, in init_process
  self.load_wsgi()
 File "/env/local/lib/python2.7/site-packages/gunicorn/workers/base.py", line 135, in load_wsgi
  self.wsgi = self.app.wsgi()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/base.py", line 67, in wsgi
  self.callable = self.load()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 65, in load
  return self.load_wsgiapp()
 File "/env/local/lib/python2.7/site-packages/gunicorn/app/wsgiapp.py", line 52, in load_wsgiapp
  return util.import_app(self.app_uri)
 File "/env/local/lib/python2.7/site-packages/gunicorn/util.py", line 352, in import_app
  __import__(module)
 File "/home/vmagent/app/mysite/wsgi.py", line 22, in <module>
  application = get_wsgi_application()
 File "/env/local/lib/python2.7/site-packages/django/core/wsgi.py", line 13, in get_wsgi_application
  django.setup(set_prefix=False)
 File "/env/local/lib/python2.7/site-packages/django/__init__.py", line 27, in setup
  apps.populate(settings.INSTALLED_APPS)
 File "/env/local/lib/python2.7/site-packages/django/apps/registry.py", line 108, in populate
  app_config.import_models()
 File "/env/local/lib/python2.7/site-packages/django/apps/config.py", line 202, in import_models
  self.models_module = import_module(models_module_name)
 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
  __import__(name)
 File "/env/local/lib/python2.7/site-packages/django/contrib/auth/models.py", line 4, in <module>
  from django.contrib.auth.base_user import AbstractBaseUser, BaseUserManager
 File "/env/local/lib/python2.7/site-packages/django/contrib/auth/base_user.py", line 52, in <module>
  class AbstractBaseUser(models.Model):
 File "/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 124, in __new__
  new_class.add_to_class('_meta', Options(meta, app_label))
 File "/env/local/lib/python2.7/site-packages/django/db/models/base.py", line 331, in add_to_class
  value.contribute_to_class(cls, name)
 File "/env/local/lib/python2.7/site-packages/django/db/models/options.py", line 214, in contribute_to_class
  self.db_table = truncate_name(self.db_table, connection.ops.max_name_length())
 File "/env/local/lib/python2.7/site-packages/django/db/__init__.py", line 33, in __getattr__
  return getattr(connections[DEFAULT_DB_ALIAS], item)
 File "/env/local/lib/python2.7/site-packages/django/db/utils.py", line 211, in __getitem__
  backend = load_backend(db['ENGINE'])
 File "/env/local/lib/python2.7/site-packages/django/db/utils.py", line 115, in load_backend
  return import_module('%s.base' % backend_name)
 File "/usr/lib/python2.7/importlib/__init__.py", line 37, in import_module
  __import__(name)
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/base.py", line 6, in <module>
  from .features import DatabaseFeatures
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/postgis/features.py", line 1, in <module>
  from django.contrib.gis.db.backends.base.features import BaseSpatialFeatures
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/backends/base/features.py", line 4, in <module>
  from django.contrib.gis.db.models import aggregates
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/__init__.py", line 3, in <module>
  from django.contrib.gis.db.models.aggregates import * # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/aggregates.py", line 1, in <module>
  from django.contrib.gis.db.models.fields import ExtentField
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/db/models/fields.py", line 3, in <module>
  from django.contrib.gis import forms, gdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/forms/__init__.py", line 3, in <module>
  from .fields import ( # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/forms/fields.py", line 4, in <module>
  from django.contrib.gis.geos import GEOSException, GEOSGeometry
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/__init__.py", line 5, in <module>
  from .collections import ( # NOQA
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/collections.py", line 11, in <module>
  from django.contrib.gis.geos.geometry import GEOSGeometry, LinearGeometryMixin
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/geos/geometry.py", line 11, in <module>
  from django.contrib.gis import gdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/__init__.py", line 28, in <module>
  from django.contrib.gis.gdal.datasource import DataSource
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/datasource.py", line 39, in <module>
  from django.contrib.gis.gdal.driver import Driver
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/driver.py", line 5, in <module>
  from django.contrib.gis.gdal.prototypes import ds as vcapi, raster as rcapi
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/prototypes/ds.py", line 9, in <module>
  from django.contrib.gis.gdal.libgdal import GDAL_VERSION, lgdal
 File "/env/local/lib/python2.7/site-packages/django/contrib/gis/gdal/libgdal.py", line 45, in <module>
  % '", "'.join(lib_names)
ImproperlyConfigured: Could not find the GDAL library (tried "gdal", "GDAL", "gdal2.1.0", "gdal2.0.0", "gdal1.11.0", "gdal1.10.0", "gdal1.9.0"). Is GDAL installed? If it is, try setting GDAL_LIBRARY_PATH in your settings.

Any help is greatly appreciated.

Dan

Answer 1

The app engine flexible environment comes with the C libraries listed here: https://cloud.google.com/appengine/docs/flexible/python/runtime

Since GDAL is a C library that isn't in the runtimes that come standard, I had to build a custom environment using a Docker container. The trick was getting GDAL installed along with the necessary python libraries in requirements.txt and Python3, then starting the django app thru gunicorn.

Here's my Dockerfile:

FROM gcr.io/google-appengine/python

RUN apt-get update && apt-get install -y \
  binutils \
  gdal-bin \
  python-gdal

# Create a virtualenv for dependencies. This isolates these packages from
# system-level packages.
RUN virtualenv /env -p python3.6

# Setting these environment variables are the same as running
# source /env/bin/activate.
ENV VIRTUAL_ENV /env
ENV PATH /env/bin:$PATH

# Copy the application's requirements.txt and run pip to install all
# dependencies into the virtualenv.
ADD requirements.txt /app/requirements.txt
RUN pip install -r /app/requirements.txt
# Add the application source code.
ADD . /app

# Run a WSGI server to serve the application. gunicorn must be declared as
# a dependency in requirements.txt.
CMD gunicorn -b :$PORT mysite.wsgi

My app.yaml:

# [START runtime]
runtime: custom
#python
env: flex
entrypoint: gunicorn -b :$PORT mysite.wsgi

beta_settings:
    cloud_sql_instances: <your-db-instance-identifying-string>

runtime_config:
  python_version: 3
# [END runtime]

And the minimum requirements.txt (add to as needed):

Django==1.11.4
mysqlclient==1.3.10
wheel==0.29.0
gunicorn==19.7.1
psycopg2==2.7.3

Hope this helps someone,

Dan