App Engine, transactions, and idempotency

Question

Please help me find my misunderstanding.

I am writing an RPG on App Engine. Certain actions the player takes consume a certain stat. If the stat reaches zero the player can take no more actions. I started worrying about cheating players, though -- what if a player sent two actions very quickly, right next to each other? If the code that decrements the stat is not in a transaction, then the player has a chance of performing the action twice. So, I should wrap the code that decrements the stat in a transaction, right? So far, so good.

In GAE Python, though, we have this in the documentation:

Note: If your app receives an exception when submitting a transaction, it does not always mean that the transaction failed. You can receive Timeout, TransactionFailedError, or InternalError exceptions in cases where transactions have been committed and eventually will be applied successfully. Whenever possible, make your Datastore transactions idempotent so that if you repeat a transaction, the end result will be the same.

Whoops. That means that the function I was running that looks like this:

def decrement(player_key, value=5):
  player = Player.get(player_key)
  player.stat -= value
  player.put()

Well, that's not gonna work because the thing isn't idempotent, right? If I put a retry loop around it (do I need to in Python? I've read that I don't need to on SO... but I can't find it in the docs) it might increment the value twice, right? Since my code can catch an exception but the datastore still committed the data... huh? How do I fix this? Is this a case where I need distributed transactions? Do I really?

Answer 1

First, Nick's answer is not correct. DHayes's transaction is not idempotent, so if it's run multiple times (ie. a retry when the first attempt was thought to have failed, when it didn't), then the value will have been decremented multiple times. Nick says that "the datastore checks if the entities have been modified since they were fetched", but that doesn't prevent the problem since the two transactions had separate fetches, and the second fetch was AFTER the first transaction completed.

To solve the problem, you can make the transaction idempotent by creating a "transaction Key" and recording that key in a new entity as part of the transaction. The second transaction can check for that transaction key, and if found, will do nothing. The transaction key can be deleted once you're satisfied that the transaction completed, or you give up retrying.

I'd like to know what "extremely rare" means for AppEngine (1-in-a-million, or 1-in-a-billion?), but my advice is that idempotent transactions is required for financial matters, but not for game scores, or even "lives" ;-)

Answer 2

If you think carefully about what you're describing, it might not actually be an issue. Think about it this way:

You player has one stat point left. He then maliciously sends 2 actions (A1 and A2) instantaneously which each need to consume that point. Both A1 and A2 are transactional.

Here's what could happen:

A1 succeeds. A2 will then abort. All good.

A1 fails legitimately (without changing data). Retry scheduled. A2 then tries, succeeds. When A1 tries again, it will abort.

A1 succeeds but reports an error. Retry scheduled. The next time either A1 or A2 tries, they will abort.

For this to work, you do need to be keeping track of whether A1 and A2 have completed - maybe give them a task UUID and store a list of finished tasks? Or even just use the task queue.

Answer 3

Shouldn't you try to store this kind of information in Memcache, which is much faster than the Datastore (something you'll need if this stat is oft used in your application). Memcache provides you a nice function: decr which:

Atomically decrements a key's value. Internally, the value is a unsigned 64-bit integer. Memcache doesn't check 64-bit overflows. The value, if too large, will wrap around.

Search for decr here. You should then use a task to save the value in this key to the datastore either every x seconds or when a certain condition is met.

Answer 4

Edit: This is incorrect - please see the comments.

Your code is fine. The idempotence the docs refer to is with regards to side-effects. As the docs explain, your transactional function may be run more than once; in such situations if the function has any side-effects, they will be applied multiple times. Since your transaction function doesn't do that, it'll be fine.

An example of a problematic function with regards to idempotence would be something like this:

def do_something(self):
  def _tx():
    # Do something transactional
    self.counter += 1
  db.run_in_transaction(_tx)

In this case, self.counter may be incremented by 1, or potentially more than 1. This could be avoided by doing the side-effects outside the transaction:

def do_something(self):
  def _tx():
    # Do something transactional
    return 1
  self.counter += db.run_in_transaction(_tx)