app engine datastore: model for progressively upda

Question

Users submit rectangular axis-aligned regions associated with "terrain maps". At any time users can delete regions they have created.

class Region(db.Model):
  terrain_map = reference to TerrainMap
  top_left_x = integer
  top_left_y = integer
  bottom_right_x = integer
  bottom_right_y = integer

I want to maintain a "terrain height map", which progressively updates as new rectangular regions are added or deleted.

class TerrainMap(db.Model):
  terrain_height = blob

Here's a "picture" of what the map could look like with two overlapping regions: http://pastebin.com/4yzXSFC5

So i thought i could do this by adding a RegionUpdate model, created when Region entity is either created or deleted, and also enqueuing a Task which would churn through a query for "RegionUpdate.applied = False"

class RegionUpdate(db.Model):
  terrain_map = reference to TerrainMap
  top_left_x = integer
  top_left_y = integer
  bottom_right_x = integer
  bottom_right_y = integer
  operation = string, either "increase" or "decrease"
  applied = False/True

The problem is it seems all RegionUpdates and Region entities have to be under the same entity group as their TerrainMap: RegionUpdates must only get created when Region entities are created or deleted, so this must be done transactionally; TerrainMap.terrain_height is updated in a Task, so this must be an idempotent operation - i can only think of doing this by transactionally grabbing a batch of RegionUpdate entities, then applying them to the TerrainMap.

That makes my entity group much larger than the "rule of thumb" size of about "a single user's worth of data or smaller".

Am i overlooking some better way to model this?

Answer 1

As I suggested in the Reddit question, I think that Brett's data pipelines talk has all the information you need to build this. The basic approach is this: Every time you insert or update a Region, add a 'marker' entity in the same entity group. Then, in that task, update the corresponding TerrainMap with the new data, and leave a Marker entity as a child of that, too, indicating that you've applied that update. See the talk for full details.

On the other hand, you haven't specified how many Regions you expect per TerrainMap, or how frequently they'll be updated. If the update rate to a single terrain map isn't huge, you could simply store all the regions as child entities of the TerrainMap to which they apply, and update the map synchronously or on the task queue in a single transaction, which is much simpler.