App Engine JDO Transaction on multiple many-to-one

Question

I have a simple domain model as follows

Driver - key(string), run-count, unique-track-count

Track - key(string), run-count, unique-driver-count, best-time

Run - key(?), driver-key, track-key, time, boolean-driver-update, boolean-track-updated

I need to be able to update a Run and a Driver in the same transaction; as well as a Run and a Track in the same transaction (obviously to make sure i don't update the statistics twice, or miss out on an increment counter)

Now I have tried assigning as run key, a key made up of driver-key/track-key/run-key(string)

This will let me update in one transaction the Run entity and the Driver entity.

But if I try updating the Run and Track entities together, it will complain that it cannot transact over multiple groups. It says that it has both the Driver and the Truck in the transaction and it can't operate on both...

tx.begin();

run = pmf.getObjectById(Run.class, runKey);
track = pmf.getObjectById(Track.class, trackKey);
//This is where it fails;

incrementCounters();
updateUpdatedFlags();
tx.commit();

Strangely enough when I do a similar thing to update Run and Driver it works fine.

Any suggestions on how else I can map my domain model to achieve the same functionality?

Answer 1

I think I found a lateral but still clean solution which still makes sense in my domain model.

The domain model changes slightly as follows:

Driver - key(string-id), driver-stats - ex. id="Michael", runs=17

Track - key(string-id), track-stats - ex. id="Monza", bestTime=157

RunData - key(string-id), stat-data - ex. id="Michael-Monza-20101010", time=148

TrackRun - key(Track/string-id), track-stats-updated - ex. id="Monza/Michael-Monza-20101010", track-stats-updated=false

DriverRun - key(Driver/string-id), driver-stats-updated - ex. id="Michael/Michael-Monza-20101010", driver-stats-updated=true

I can now update atomically (i.e. precisely) the statistics of a Track with the statistics from a Run, immediately or in my own time. (And same with the Driver / Run statistics).

So basically I have to expand a little bit the way I model my problem, in a non-conventional relational way. What do you think?

Answer 2

realize this is late, but..

Have you seen this method for Bank Account transfers? http://blog.notdot.net/2009/9/Distributed-Transactions-on-App-Engine

It seems to me that you could do something similar by breaking out your increment counters into two steps as a IncrementEntity and process that, picking up the pieces later if a transaction fails etc.

From the blog:

1. In a transaction, deduct the required amount from the paying account, and create a Transfer child entity to record this, specifying the receiving account in the 'target' field, and leaving the 'other' field blank for now.
2. In a second transaction, add the required amount to the receiving account, and create a Transfer child entity to record this, specifying the paying account in the 'target' field, and the Transfer entity created in step 1 in the 'other' field.
3. Finally, update the Transfer entity created in step 1, setting the 'other' field to the Transfer we created in step 2.

The blog has code examples in Python, but is should be easy to adapt

Answer 3

With Google App Engine, all of the datastore operations must be on entities in the same entity group. This is because your data is usually stored across multiple tables, and Google App Engine cannot do transactions across multiple tables.

Entities with owned one-to-one and one-to-many relationships are automatically in the same entity group. So if an entity contains a reference to another entity, or a collection of entities, you can read or write to both in the same transactions. For entities that don't have an owner relationship, you can create an entity with an explicit entity group parent.

You could put all of the objects in the same entity group, but you might get some contention if too many users are trying to modify objects in an entity group at the same time. If every object is in its own entity group, you can't do any meaningful transactions. You want to do something in between.

One solution is to have Track and Run in the same entity group. You could do this by having Track contain a List of Runs (if you do this, then Track might not need run-count, unique-driver-count and best-time; they could be computed when needed). If you do not want Track to have a List of Runs, you can use an unowned one-to-many relationship and specify the entity group parent of the Run be its Track (see "Creating Entities With Entity Groups" on this page). Either way, if a Run is in the same entity group as its track, you could do transactions that involve a Run and some/all of its Tracks.

For many large systems, instead of using transactions for consistency, changes are done by making operations that are idempotent. For instance, if Driver and Run were not in the same entity group, you could update the run-count for a Driver by first doing a query to get the count of all runs before some date in the past, then, in a transaction, update the Driver with the new count and the date when it was last computed.

Keep in mind when using dates that machines can have some kind of a clock drift, which is why I suggested using a date in the past.

Answer 4

There's an interesting google io session on this topic http://www.google.com/events/io/2010/sessions/high-throughput-data-pipelines-appengine.html

I guess you could update the Run stats and then fire two tasks to update the Driver and the Track individually.