manual serialization / deserialization of AppEngin

Question

Is it possible to manually define the logic of the serialization used for AppEngine Datastore?

I am assuming Google is using reflection to do this in a generic way. This works but proves to be quite slow. I'd be willing to write (and maintain) quite some code to speed up the serialization / deserialization of datastore objects (I have large objects and this consumes quite some percentage of the time).

Answer 1

The datastore uses Protocol-Buffers internally, and there is no way round, as its the only way your application can communicate with the datastore. (The implementation can be found in the SDK/google/appengine/datastore/entity_pb.py) If you think, (de)serialization is too slow in your case, you probably have two choices

• Move to a lower DB API. There's another API next to the two well-documented ext.db and ext.ndb APIs at google.appengine.datastore. This hasn't all the fancy model-stuff and provides a simple (and hopefully fast) dictionary-like api. This will keep your datastore-layout compatible with the other two DB APIs.
• Serialize the object yourself, and store it in a dummy entry consisting just of a text-field. But you'll probably need to duplicate data into your base entry, as you cannot filter/sort by data inside your self-serialized text.