Simple DB query on Google App Engine taking a lot

Question

I'm fairly new to Google App Engine and Python, but I did just release my first real-world site with it. But now I'm getting problems with one path that is using significantly more CPU (and API CPU) time than the other paths. I've narrowed it down to a single datastore fetch that's causing the problem: Carvings.all().fetch(1000)

Under the App Engine dashboard it's reporting "1040cpu_ms 846api_cpu_ms" pretty reliably for each request to that path. It has seemed like this may be the source to some unresponsiveness that my client has experienced with the site in general.

So I can't figure out what is so expensive about this query. Here is the related data model:

class Carving(db.Model):
    title = db.StringProperty(required=True)
    reference_number = db.StringProperty()
    main_category = db.StringProperty()
    sub_category = db.StringProperty()
    image = db.ReferenceProperty(CarvingImage)
    description = db.TextProperty()
    price = db.FloatProperty()
    size = db.StringProperty()
    material = db.StringProperty()
    added_at = db.DateTimeProperty(auto_now_add=True)
    modified_at = db.DateTimeProperty(auto_now=True)

In other places in the app when I pull this model from the datastore I do more filtering and I guess that's why they aren't causing any troubles. But the total number of entities for this model is just above 90 and I just can't imagine why this is so expensive.

Answer 1

It could be the image (and/or Text property) that is taking time to load & marshall into objects, depending on how big those properties are.

First prize: just use the memcache as others say. Then the overhead is incurred only on the first hit.

Second prize: I'm not sure how often your images are being changed and how many you might have, but you could consider uploading them as static files and simply linking to them in your HTML. Then it'd be just an HTTP GET from the browser - much lower overhead.

Answer 2

Sometimes you'll get better performance if you do an indexed query, rather than a query of "all" elements in the model.

Also, consider using memcache.

Answer 3

• Memcache, if you haven't already, and especially if the same carvings are going to be fetched again and again. If you only have 90 total, I would imagine they would all be in the cache pretty quickly, and then you should be golden.

• Do you need all the properties of the Carvings? For example, if you're just displaying a list of carvings, you could have a separate Entity that was something like CarvingSummary that only had a few properties. This would mean your schema was denormalized, but sometimes that's the price you pay for speed.

Also, I'm assuming this is not the first page the user will always hit? If that were the case it could be the cloud spinning up a a new instance.

Answer 4

Do you actually need 1000 entities? CPU time goes up more or less linearly with the number of results retrieved, so if you don't actually need all the results, you may be wasting a lot of time fetching and decoding them.