Expired queries and appengine

Question

Within a task, I'm iterating over a collection of items with a query. After each entity is fetched from the query, I am also performing a URL request. After iterating over a large number of these items, I'm seeing the following error:

BadRequestError: The requested query has expired. Please restart it with the last cursor to read more results.

What is the lease on a query once you create it?

Answer 1

This issue might shed some light on your problem: https://code.google.com/p/googleappengine/issues/detail?id=4432

Even though offline requests can currently live up to 10 minutes (and background instances can live forever) datastore queries can still only live for 30 seconds. We plan to improve this, but since a 'consistent' view of the data is only preserved for a limit period of time, there is an upper bound to how long a query can last (which is < 10 minutes).

...

Instead of running a single long query, consider fetching batches from the query using query cursors.

Answer 2

Simple create defered tasks for each element in your sequence. There is good article with examples how to do it right way "Background work with the deferred library".

Answer 3

I wrote a simple helper to do this - you call it with the batch_size, the object class for the query, and the callback that handles the elements in the query.

(Note, I am using djangoappengine and therefore django query format - but you could modify it to suit.)

def loop_over_objects_in_batches(batch_size, object_class, callback):
    logging.info("Calling batched loop with batch_size: %d, object_class: %s, callback: %s" % (batch_size, object_class, callback))

    num_els = object_class.objects.all().count()
    num_loops = num_els / batch_size
    remainder = num_els - num_loops * batch_size
    offset = 0
    while offset < num_loops * batch_size:
        logging.info("Processing batch (%d:%d)" % (offset, offset+batch_size))
        query = object_class.objects.all()[offset:offset + batch_size]
        for q in query:
            callback(q)

        offset = offset + batch_size

    if remainder:
        logging.info("Processing remainder batch (%d:-)" % offset)
        query = object_class.objects.all()[offset:]
        for q in query:
            callback(q)