Updating Model Schema in Google App Engine?

Question

Google is proposing changing one entry at a time to the default values ....

http://code.google.com/appengine/articles/update_schema.html

I have a model with a million rows and doing this with a web browser will take me ages. Another option is to run this using task queues but this will cost me a lot of cpu time

any easy way to do this?

Answer 1

Instead of what the docs suggest, I would suggest to use low level GAE API to migrate.

The following code will migrate all the items of type DbMyModel:

• new_attribute will be added if does not exits.
• old_attribute will be deleted if exists.
• changed_attribute will be converted from boolean to string (True to Priority 1, False to Priority 3)

Please note that query.Run returns iterator returning Entity objects. Entity objects behave simply like dicts:

from google.appengine.api.datastore import Query, Put
query = Query("DbMyModel")
for item in query.Run():
    if not 'new_attribute' in item:
        item['attribute'] = some_value
    if 'old_attribute' in item:
        del item['old_attribute']
    if ['changed_attribute'] is True:
        item['changed_attribute'] = 'Priority 1'
    elif ['changed_attribute'] is False:
        item['changed_attribute'] = 'Priority 3'
    #and so on...
    #Put the item to the db:
    Put(item)

In case you need to select only some records, see the google.appengine.api.datastore module's source code for extensive documentation and examples how to create filtered query.

Using this approach it is simpler to remove/add properties and avoid issues when you have already updated your application model than in GAE's suggested approach.

For example, now-required fields might not exist (yet) causing errors while migrating. And deleting fields does not work for static properties.

Answer 2

Because the datastore is schema-less, you do literally have to add or remove properties on each instance of the Model. Using Task Queues should use the exact same amount of CPU as doing it any other way, so go with that.

Before you go through all of that work, make sure that you really need to do it. As noted in the article that you link to, it is not the case that all entities of a particular model need to have the same set of properties. Why not change your Model class to check for the existence of new or removed properties and update the entity whenever you happen to be writing to it anyhow.

Answer 3

This doesn't help OP but may help googlers with a tiny app: I did what Alex suggested, but simpler. Obviously this isn't appropriate for production apps.

1. deploy App Engine Console
2. write code right inside the web interpreter against your live datastore

like so:

from models import BlogPost  
for item in BlogPost.all():  
    item.attr="defaultvalue"  
    item.put()