How do I perform a large query in google app engin

Question

I have collection of apps that total almost 1 million users. I am now adding a push notification system using Google cloud messaging to create alerts. My database contains an entity with the GcmId and app name (ex. "myApp1").

Now, I want to send a GCM message to all users of "myApp1". The objectify documents do not describe the .limit function well though. For example, from the GCM demo app:

        List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).limit(10).list();

will send to the first 10 entries. But I need all entries that match appType="myApp1". This is harder because the query can be large and could potentially match half a million users and I need to send the GCM push to all of them.

How is such a large query performed?

EDIT: I am currently using

        List<RegistrationRecord> records = ofy().load().type(RegistrationRecord.class).filter("app","myApp1").list();

for testing and it is working fine. However when pushed live, the dataset is huge, and I don't know what the repercussions are.

Answer 1

i believe you are looking it from the wrong angle.

objectify or low level appengine deal very well with paginated results using cursors, so you need to process results by chunks. i wont go into details on how to do that because that would end up costing you a lot of $ for all those datastore reads and you will need task queues.

instead, look at topics in google cloud messaging: https://developers.google.com/cloud-messaging/topic-messaging

the user (client side app) subscribes to the topic (the appid in your case). you then send a single topic push which is much easier from an appengine frontend instance (limited to 30 seconds response and such).

I found this blog post to be a great example of a full implementation and how to properly handle possible errors:

https://blog.pushbullet.com/2014/02/12/keeping-google-cloud-messaging-for-android-working-reliably-techincal-post/

the only issue i can see is that a push from server is documented to take up to 30 seconds. An appengine frontend instance also has a 30 seconds limit total, so while it waits for GCM push to complete the servlet itself can timeout. one way to solve this is to send the push from taskqueue which would give you 60 seconds for urlfetch calls (i assume that limit applies to any api call as well): https://cloud.google.com/appengine/docs/java/urlfetch/