Cannot run Google App Engine custom managed VM: --

Question

PROBLEM DESCRIPTION

I am trying to create a custom managed VM for Google App Engine that behaves identically to the standard python27 managed VM provided by Google. (I'm doing this as a first step to adding a C++ library to the runtime).

From google documentation, the following Dockerfile specifies the standard python27 runtime:

FROM gcr.io/google_appengine/python-compat
ADD . /app

I have verified that this is the right Dockerfile by examining the one generated by gcloud preview app run when using the standard python27 runtime. It is identical to this.

But when I run my application with this Dockerfile using dev_appserver.py or with gcloud preview app run I get an error saying:

The --custom_entrypoint flag must be set for custom runtimes

I am using the latest versions of gcloud (1.9.86, with app-engine-python component version 1.9.28) and the standalone python app engine SDK (1.9.28). I had the same problem with earlier versions, so I updated to the latest.

THINGS I HAVE TRIED:

gcloud preview app run --help has the following to say about --custom-entrypoint:

 --custom-entrypoint CUSTOM_ENTRYPOINT
    Specify an entrypoint for custom runtime modules. This is required when
    such modules are present. Include "{port}" in the string (without
    quotes) to pass the port number in as an argument. For instance:
    --custom_entrypoint="gunicorn -b localhost:{port} mymodule:application"

I am not sure what to make of this. Should the docker image not already contain an ENTRYPOINT? Why am I being required to provide one in addition? Also, what should the entrypoint be for the gcr.io/google_appengine/python-compat image be? Google provides no documentation for this.

I have tried a meaningless --custom-entrypoint="echo", which silences the error, but the application does not response to any HTTP requests.

The two other relevant stackoverflow questions I have found have not helped. The accepted answers seem to suggest that this is a bug in the SDK that was resolved. But I have tried it in two versions of the SDK, including the latest, and I still have the problem.

• How to fix “The --custom_entrypoint flag must be set for custom runtimes”?
• Google Managed VM error - custom entry point

STEPS TO REPRORDUCE:

To highlight my problem, I have created a trivial application that generates the error. It consists of just three files:

app.yaml:

module: default
version: 1
runtime: custom
api_version: 1
threadsafe: true
vm: true

handlers:
- url: /.*
  script: wsgi.app

Dockerfile:

FROM gcr.io/google_appengine/python-compat
ADD . /app

This Dockerfile is the same one that is used for the python27 runtime (and in fact literally copy-pasted from the Dockerfile generated by gcloud preview app run when using the python27 runtime), so this should be identical to setting runtime: python27.

wsgi.py:

import webapp2

class Hello(webapp2.RequestHandler):
    def get(self):
        self.response.write(u'Hello')

app = webapp2.WSGIApplication([('/Hello', Hello)], debug=True)

When I run dev_appserver.py app.yaml in the directory containing these three files however, I get the following error:

Traceback (most recent call last):
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 83, in <module>
    _run_file(__file__, globals())
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/dev_appserver.py", line 79, in _run_file
    execfile(_PATHS.script_file(script_name), globals_)
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 1033, in <module>
    main()
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 1026, in main
    dev_server.start(options)
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/devappserver2.py", line 818, in start
    self._dispatcher.start(options.api_host, apis.port, request_data)
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/dispatcher.py", line 194, in start
    _module.start()
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1555, in start
    self._add_instance()
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 1707, in _add_instance
    expect_ready_request=True)
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/custom_runtime.py", line 73, in new_instance
    assert self._runtime_config_getter().custom_config.custom_entrypoint
  File "/home/vagrant/google-cloud-sdk/platform/google_appengine/google/appengine/tools/devappserver2/module.py", line 383, in _get_runtime_config
    raise ValueError('The --custom_entrypoint flag must be set for '
ValueError: The --custom_entrypoint flag must be set for custom runtimes

Answer 1

After trying to get my custom VM working with dev_appserver for the better part of a day, the accepted answer to this post came as a pretty unpleasant surprise. But I figured deploying a dev server couldn't be that much of a hassle because, after all, the VM is a standard Docker image.

Well there did turn out the be a few issues that prevent a straight deployment from working. I've provided a summary of these issues below as well as how I solved them. I may have missed some incompatibilities between the Docker and App Engine environments (especially with the many aspects of App Engine that my project didn't use) but hopefully it's enough to get people up and running.

Sources of trouble

First, I found the python environment run in the Compute Engine VMs to be a bit more lenient than a normal VM environment (e.g. packages like webapp2 are always available). Consequently, deploying to the less forgiving Docker container environment surfaced some latent errors in my project.

That being said, there are some differences in the environments that necessitate some tweaks even if your project is flawless:

• Problem: gunicorn (or the server of your choice) must be installed on the Docker container's path.

  • While this may seem obvious, I ran into trouble because I included gunicorn in my project's requirements.txt file. Unfortunately, I was installing all of these dependencies using pip install -t ... which is only able to install source. As a result, there was no gunicorn binary on the image let alone on the PATH.

• Solution: Explicitly pip install gunicorn

---

• Problem: google.appengine.* packages are not available from the App Engine base Docker image nor are they available via pip (AFAICT).
  • This may be a common source of trouble because google.appengine.ext.vendor is the recommended interface for importing third-party libraries into your App Engine app.
• Solution: I got around this by downloading the entire Google App Engine package hierarchy and placing it on the path of the app.

How to get the script

The script to build and deploy the VM docker image to a docker container running locally is available here.

For an working example, check out my project.

Let me know if you have a comment, a feature request, or if you write prettier bash than me (I feel I've set the bar comfortably low on that count).

Answer 2

EDIT 1: The solution posted by user862857 makes use of Docker itself to build images from the Dockerfile and run them in containers. This is also a good approach to running Managed VMs and Custom Runtimes in development contexts.

---

The accepted answer doesn't seem correct. A github README should not trump the official docs for authoritativeness when dealing with a rapidly-evolving Beta product. It's perfectly possible to a runtime: custom app in the dev environment, using the Dockerfile mentioned in OP's post,

FROM gcr.io/google_appengine/python-compat
ADD . /app

using the --runtime=python-compat flag. They would need to catch requests to /_ah/start and /_ah/health, though. Attempt to run the following command, given the following files, and see for yourself:

devserver command

$ dev_appserver.py app.yaml --runtime=python-compat

app.yaml

runtime: custom
vm: true
api_version: 1
threadsafe: true

handlers:
- url: /.*
  script: main.app

Dockerfile

FROM gcr.io/google_appengine/python-compat

RUN apt-get update

RUN apt-get install -y gwhois

ADD . /app

main.py

import logging
import webapp2
from subprocess import Popen, PIPE

class OkHandler (webapp2.RequestHandler):
    def get (self): 
        self.response.write ('ok')

class MainPage(webapp2.RequestHandler):
    def get(self):
        self.response.headers['Content-Type'] = 'text/plain'
        domain = self.request.get ('domain')
        cmd = ["gwhois", domain]
        process = Popen (cmd, stdout=PIPE, stderr=PIPE)
        output, err = process.communicate()
        exit_code = process.wait()
        self.response.write('stdout: %s' % output)
        logging.info ('stderr: %s' % err)

app = webapp2.WSGIApplication([
    ('/', MainPage),
    ('/_ah/start', OkHandler),
    ('/_ah/health', OkHandler)
], debug=True)

Send a request to /?domain=stackoverflow.com to see this in action.

---

N.B.

If they wished to entirely decouple from the python-compat runtime and simply deploy/test a python WSGI app via, they could also use the --custom_entrypoint flag, so long as they had a command which would start running the appropriate WSGI app on a suitable port (such a command would be uwsgi or gunicorn).

Answer 3

UPDATE

THIS MAY NO LONGER BE ACCURATE. SEE NICK'S ANSWER.

(Though I could not get that working. But I did not try very hard)

---

There is a completely undocumented but absolutely essential piece of information w.r.t. custom managed VMs:

THEY CANNOT BE RUN ON THE DEVELOPMENT SERVER!

If you think this crucial fact would be mentioned anywhere sane, like say, the documentation page for custom managed VMs, or for dev_appserver.py, or even as an error message when running dev_appserver.py, then you are giving Google far too much credit.

The only place where I can find any kind of statement about this is in the Readme file of the appengine-java-vm-guestbook-extras demo on github (seriously):

The Cloud SDK does not support anymore running custom runtimes when a Dockerfile is provided. You'll have to deploy the application to App Engine

Google does not care to:

1. Implement this basic and important feature.
2. Document that the development server is missing such an important feature.
3. Give any reasonable error message when the user tires to perform the action.

I hope this answer saves some sorry developer from the days of torment I suffered because of this.