“Failed to import google/appengine/ext/deferred/ha

I use App Engine Flexible Environment (previously called Managed VMs), and recently upgraded to the latest gcloud SDK. It included some new errors:

ERROR: (gcloud.preview.app.deploy) Error Response: [400] Invalid
character
in filename: lib/setuptools/script (dev).tmpl

ERROR: The [application] field is specified in file [.../app.yaml]. This field is not used
by gcloud and must be removed. Project name should instead be
specified either by `gcloud config set project MY_PROJECT` or by
setting the `--project` flag on individual command executions.

ERROR: (gcloud.preview.app.deploy) There is a Dockerfile in the
current directory, and the runtime field in
.../app.yaml is currently set to
[runtime: python27]. To use your Dockerfile to build a custom runtime,
set the runtime field in .../app.yaml
to [runtime: custom]. To continue using the [python27] runtime, please
omit the Dockerfile from this directory.

I fixed these errors and was able to publish again, but started seeing errors like this:

Failed to import google/appengine/ext/deferred/handler.py
Traceback (most recent call last):
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 549, in GetUserAppAndServe
    app, mod_file = self.GetUserApp(script)
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 410, in GetUserApp
    app = _AppFrom27StyleScript(script)
  File "/home/vmagent/python_vm_runtime/google/appengine/ext/vmruntime/meta_app.py", line 270, in _AppFrom27StyleScript
    app, filename, err = wsgi.LoadObject(script)
  File "/home/vmagent/python_vm_runtime/google/appengine/runtime/wsgi.py", line 85, in LoadObject
    obj = __import__(path[0])
ImportError: Import by filename is not supported.

After a bit of digging, I figured out what is going on. Namely, the code that processes this:

builtins:
- remote_api: on
- appstats: on
- deferred: on

is broken with Managed VMs. The correct fix is to eliminate these and inline the builtin includes instead. You can find the relevant includes inside these subdirectories:

In my case, it was to add this to my handlers: directive:

- url: /_ah/queue/deferred
  script: google.appengine.ext.deferred.application
  login: admin
- url: /_ah/stats.*
  script: google.appengine.ext.appstats.ui.app
- url: /_ah/remote_api(/.*)?
  script: google.appengine.ext.remote_api.handler.application

As to why, you can understand more here. In google/appengine/ext/builtins/__init__.py#L92, it attempts to find the relevant include file by using the runtime: field in your app.yaml. This means where it previously looked up deferred/include-python27.yaml, it now attempts to find deferred/include-custom.yaml (due to fixing the errors above) and fails. So now it defaulting to deferred/include.yaml, which lists the include scrip by path-name instead of module-name. This then breaks in a python27-custom-VM setup (since it is expecting/needing module-names)