Store a list of dictionaries in GAE

Question

I have a list of about 20 objects and for each object I return a list of 10 dictionaries.
I am trying to store the list of 10 dictionaries for each object in the list on GAE; I do not think I am writing the code correctly to store this information to GAE.
Here is what I have: Before my main request handler I have this class:

class Tw(db.Model):
  tags = db.ListProperty()
  ip = db.StringProperty()

In my main request handler I have the following:

for city in lst_of_cities: # this is the list of 20 objects
  dict_info = hw12.twitter(city) # this is the function to get the list of 10 dictionaries for each object in the list
  datastore = Tw() # this is the class defined for db.model
  datastore.tags.append(dict_info) # 
  datastore.ip = self.request.remote_addr
datastore.put()

data = Data.gql("") #data entities we need to fetch

I am not sure if this code is write at all. If anyone could please help it would be much appreciated.

Answer 1

Welcome to Stack Overflow!

I see a few issues:

1. Dictionaries are not supported value types for App Engine properties.
2. You're only storing the last entity; the rest are discarded.
3. You're using a ListProperty, but instead of appending each element of dict_info, you're doing a single append of the entire list.

Since you can't store a raw dictionary inside a property, you need to serialize it to some other format, like JSON or pickle. Here's a revised example using pickle:

from google.appengine.ext import db
import pickle

class Tw(db.Model):
  tags = db.BlobProperty()
  ip = db.StringProperty()

entities = []
for city in lst_of_cities:
  dict_info = hw12.twitter(city)
  entity = Tw()
  entity.tags = db.Blob(pickle.dumps(dict_info))
  entity.ip = self.request.remote_addr
  entities.append(entity)

db.put(entities)

When you fetch the entity later, you can retrieve your list of dictionaries with pickle.loads(entity.tags).

Answer 2

Since this was written, the App Engine has pushed out their experimental "ndb" Python database model, which contains in particular the JsonProperty, something that pretty well directly implements what you want.

Now, you need to be running the Python 2.7 version of the App Engine, which is still not quite ready for production, but it all seems pretty stable these days, GvR himself seems to be writing a lot of the code which bodes well for the code quality, and I'm intending to use this in production sometime this year...

Answer 3

When I deal with data types that are not directly supported by Google App Engine like dictionaries or custom data type, I usually adopt the handy PickleProperty.

from google.appengine.ext import db
import pickle

class PickleProperty(db.Property):
    def get_value_for_datastore(self, model_instance):
        value = getattr(model_instance, self.name, None)
        return pickle.dumps(value)

    def make_value_from_datastore(self, value):
        return pickle.loads(value)

Once declared the PickleProperty class in your commons.py module, you can use it to store your custom data with something like this:

from google.appengine.ext import db
from commons import PickleProperty

class Tw(db.Model):
  tags = PickleProperty()
  ip = db.StringProperty()

entities = []
for city in lst_of_cities:
  dict_info = hw12.twitter(city)
  entity = Tw()
  entity.tags = dict_info
  entity.ip = self.request.remote_addr
  entities.append(entity)

db.put(entities)

To retrieve the data back go with:

entity.tags