Autoincrement ID in App Engine datastore

Question

I'm using App Engine datastore, and would like to make sure that row IDs behave similarly to "auto-increment" fields in mySQL DB.

Tried several generation strategies, but can't seem to take control over what happens:

• the IDs are not consecutive, there seem to be several "streams" growing in parallel.
• the ids get "recycled" after old rows are deleted

Is such a thing at all possible ? I really would like to refrain from keeping (indexed) timestamps for each row.

Answer 1

It sounds like you can't rely on IDs being sequential without a fair amount of extra work. However, there is an easy way to achieve what you are trying to do:

We'd like to delete old items (older than two month worth, for example)

Here is a model that automatically keeps track of both its creation and its modification times. Simply using the auto_now_add and auto_now parameters makes this trivial.

from google.appengine.ext import db

class Document(db.Model):
  user = db.UserProperty(required=True)
  title = db.StringProperty(default="Untitled")
  content = db.TextProperty(default=DEFAULT_DOC)
  created = db.DateTimeProperty(auto_now_add=True)
  modified = db.DateTimeProperty(auto_now=True)

Then you can use cron jobs or the task queue to schedule your maintenance task of deleting old stuff. Find the oldest stuff is as easy as sorting by created date or modified date:

db.Query(Document).order("modified")
# or
db.Query(Document).order("created")

Answer 2

I think this is probably a fairly good solution, however be aware that I have not tested it in any way, shape or form. The syntax may even be incorrect!

The principle is to use memcache to generate a monotonic sequence, using the datastore to provide a fall-back if memcache fails.

class IndexEndPoint(db.Model):
    index = db.IntegerProperty (indexed = False, default = 0)

def find_next_index (cls):
    """ finds the next free index for an entity type """
    name = 'seqindex-%s' % ( cls.kind() )

    def _from_ds ():
        """A very naive way to find the next free key.

        We just take the last known end point and loop untill its free.
        """

        tmp_index = IndexEndPoint.get_or_insert (name).index

        index = None
        while index is None:
            key = db.key.from_path (cls.kind(), tmp_index))
            free = db.get(key) is None
            if free:
                index = tmp_index
            tmp_index += 1

        return index

    index = None

    while index is None:   
        index = memcache.incr (index_name)
        if index is None:  # Our index might have been evicted
            index = _from_ds ()
            if memcache.add (index_name, index):  # if false someone beat us to it
                index = None

    # ToDo:
    # Use a named task to update IndexEndPoint so if the memcache index gets evicted
    # we don't have too many items to cycle over to find the end point again.

    return index


def make_new (cls):
    """ Makes a new entity with an incrementing ID """

    result = None

    while result is None:
        index = find_next_index (cls)


        def txn ():
            """Makes a new entity if index is free.

            This should only fail if we had a memcache miss 
            (does not have to be on this instance).
            """
            key = db.key.from_path (cls.kind(), index)
            if db.get (key) is not None:
                return

            result = cls (key)
            result.put()
            return result

        result = db.run_in_transaction (txn)

Answer 3

What I know, is that auto-generated ID's as Long Integers are available in Google App Engine, but there's no guarantee that the value's are increasing and there's also no guarantee that the numbers are real one-increments.

So, if you nee timestamping and increments, add a DateTime field with milliseconds, but then you don't know that the numbers are unique.

So, the best this to do (what we are using) is: (sorry for that, but this is indeed IMHO the best option)

• use a autogenerated ID as Long (we use Objectify in Java)
• use a timestamp on each entity and use a index to query the entities (use a descending index) to get the top X