Integration with Cloud Pub/Sub APIs from App Engine Standard
I am working on developing a Google app engine app in standard Python environment. For some portions of the code, I need to integrate with Google Cloud pub/sub APIs.
As mentioned here, Pub/Sub can only be integrated in the App Engine flexible environment (BTW it is also only in alpha). Can someone please describe how to integrate with Pub/Sub in the App Engine Standard environment?
My use case description
I am trying to integrate MQTT with google app engine by using Agosto IOT broker. I will be using MQTT for clients (Currently mobile platforms) and on server side, I plan to use pub/sub for receiving/sending the messages and saving relevant data to the database.
You might want to try instead to use the new Google Cloud IoT Core product (full disclosure, I worked on it) instead of hosting MQTT on App Engine. Cloud IoT Core lets you connect to a Google-provided MQTT bridge that will put your data into Google Cloud PubSub. You can use Google Cloud DataFlow to move your data from PubSub to your data warehouse for analytics or can use your own database as the output from DataFlow.
The connection details for communicating with the Google Cloud IoT Core MQTT bridge are discussed in detail in the documentation but the important connection properties you will need to be aware of are the hostname (mqtt.googleapis.com) port (8883 or 443) and the MQTT password / client ID which are going to be based on the devices you've provisioned for the service.
Your actual MQTT client will need to be chosen depending on which programming language you're trying to access the MQTT bridge. If you're trying from Android, you could start from the Java MQTT client sample and would probably end up with something like the Android Things Cloud IoT sensor hub connector from the AndroidThings team.
TL;DR - App Engine standard does not support the newer Google Cloud Client libraries. You will instead need to use the older Google Cloud API Client libraries to communicate with Cloud Pub/Sub.
Cloud API Client libraries (older) vs Google Cloud Client libraries (newer)
The Cloud Pub/Sub client library documentation you're pointing to advises you to use the older Google API Client libraries (which is supported on App Engine Standard environment) instead of Google Cloud Client libraries (which is supported on App Engine Flexible environment but not on standard)
The client libraries are explained in detail here.
Google API Client libraries for Cloud Pub/Sub
Here are all the list of APIs which are supported using Google API Client libraries. Cloud Pub/Sub APIs are also part of this list.
Using Google API Client libraries with App Engine Standard
If you scroll down that page, there is a section describing how this API library can be used in App Engine Standard environment. In short, you will need to bundle the library along with your application just like other third-party libraries you use.
App Engine
Because the Python client libraries are not installed in the App
Engine Python runtime environment, they must be vendored into your
application just like third-party libraries.
This warning that you will see on the page, advises you to use the regular Cloud Client library if possible. But since App Engine Standard does not support it, you can ignore it for that use case.
While this library is still supported, we suggest trying the newer
Cloud Client Library for Google Cloud Pub/Sub, especially for new
projects. See Google Cloud Pub/Sub Libraries for installation and
usage details.
Examples using google-api-python-client library to invoke PubSub APIs
Using credentials from a service account json file
The following example shows you how you can use a service account to authenticate with Google Cloud PubSub APIs and invoke them. The information about how to use credentials from a Service account is available here.
You will need to have the following python packages pre-installed for this example to work: google-api-python-client and oauth2client.
If you're using pip, you can do:
pip install google-api-python-client oauth2client
Example which I have tested personally:
from googleapiclient import discovery
from httplib2 import Http
from oauth2client.service_account import ServiceAccountCredentials
# BEGIN CONFIG
PRIVATE_KEY_JSON = 'path/to/service_account_private_key.json'
API_SCOPES = ['https://www.googleapis.com/auth/pubsub']
PROJECT_NAME = 'FILL_IN_PROJECT_NAME_HERE'
# END CONFIG
# The format of project name expected by PubSub
PROJECT = 'projects/{0}'.format(PROJECT_NAME)
# Create a ServiceAccountCredentials object by reading the credentials from
# your JSON file.
credentials = ServiceAccountCredentials.from_json_keyfile_name(
PRIVATE_KEY_JSON, scopes=API_SCOPES)
# Build the Cloud PubSub API object which you will be using for
# invoking the corresponding APIs using the credentials object
# you created previously
pubsub = discovery.build('pubsub', 'v1', credentials=credentials)
# List all topics the specified project
topics = pubsub.projects().topics().list(
project=PROJECT).execute()
print topics
# Add a new topic
topic_name = 'TOPIC_NAME_TO_ADD'
added_topic_response = pubsub.projects().topics().create(
name='{0}/topics/{1}'.format(PROJECT, topic_name), body={}).execute()
print added_topic_response
Using credentials from a service account within an App Engine app
There is some info here regarding how to use Service account credentials from your App Engine apps.
The above example will work for the most part for invoking PubSub APIs, except for the part where you will initialize the credentials object. That part can be replaced roughly as described below:
Service Accounts
If your App Engine application needs to call an API to access data
owned by the application's project, you can simplify OAuth 2.0 by
using Service Accounts. These server-to-server interactions do
not involve a user, and only your application needs to authenticate
itself. Use the AppAssertionCredentials class to create a
Credentials object without using a Flow object.
In the following code snippet, a Credentials object is created and an
Http object is authorized:
import httplib2
from google.appengine.api import memcache
from oauth2client.contrib.appengine import AppAssertionCredentials
...
credentials = AppAssertionCredentials(scope='https://www.googleapis.com/auth/devstorage.read_write')
http = credentials.authorize(httplib2.Http(memcache))
pubsub = discovery.build('pubsub', 'v1', http=http)
...
Once you have an authorized Http object, you can pass it to the
build() or execute() functions as you normally would.
Using Application Default Credentials
You can also utilize Application Default credentials for your local testing and also works within App Engine environment.
from oauth2client.client import GoogleCredentials
...
credentials = GoogleCredentials.get_application_default()
pubsub = discovery.build('pubsub', 'v1', credentials=credentials)
...
