ndb Model - retrieve ordered set of property names

Question

I'm often asked to export data stored in NDB models to csv. For that purpose, I usually ended up writing a model like this:

from google.appengine.ext import ndb

class Foo(ndb.Model):
    monty = ndb.StringProperty()
    python = ndb.StringProperty()

    @property
    @classmethod
    def fieldnames(cls):
        return ['monty', 'python']

and in the export module something along the lines of

# pseudocode ...

query = Foo.gql('where monty = :1', 'bunny')
data = [littlefoo._to_dict() for littlefoo in query]
fieldnames = Foo.fieldnames
with open('outfile.csv', 'w') as f:
writer = csv.DictWriter(f, fieldnames, dialect=dialect)
writer.writerows(data)

Note that the fieldnames method is needed for determining the order of the fields in the output csv. The problem with this approach is that for any model with a significant number of attributes, adding the fieldnames method is ugly duplicate work. Ideally, I'd like to simply order the properties as I declare them, and retrieve them in the same order for fieldnames. Is there a way to retrieve these properties in order? Foo._properties is ordered alphabetically.

Answer 1

To my knowledge, this isn't possible without parsing the source for yourself. Luckily, with python's "batteries included" mentality, this isn't too hard. You can use inspect to get the source code and then you can use ast to parse the source and order things:

import ast
import inspect

class NodeTagger(ast.NodeVisitor):
    def __init__(self):
        self.class_attribute_names = {}

    def visit_Assign(self, node):
        for target in node.targets:
            self.class_attribute_names[target.id] = target.lineno

    # Don't visit Assign nodes inside Function Definitions.
    def visit_FunctionDef(self, unused_node):
        return None

def order_properties(model):
    properties = model._properties
    source = inspect.getsource(model)
    tree = ast.parse(source)
    visitor = NodeTagger()
    visitor.visit(tree)
    attributes = visitor.class_attribute_names
    model._ordered_property_list = sorted(properties, key=lambda x:attributes[x])
    return model


@order_properties
class Foo(object):
    c = 1
    b = 2
    a = 3

    # Add a _properties member to simulate an `ndb.Model`.
    _properties = {'a': object, 'b': object, 'c': object}

print Foo._ordered_property_list

Note that the approach here is almost general. I used the knowledge that ndb.Models have a _properties attribute, but that information could probably be gleaned from dir or inspect.getmembers so order_properties could be modified so that it works completely generally.

Answer 2

I'm just making this up, it's probably full of errors, but hopefully it starts you down the right path:

from google.appengine.ext.ndb import Property

def get_field_names(ndbmodel):
    result = []
    all_fields = dir(ndbmodel)
    for field in all_fields:
        if isinstance(getattr(ndbmodel, field), Property):
            result.append(field)
    return result.sort()

Answer 3

If using google.appengine.ext.db:

db.Property instances have an attribute called creation_counter, which is incremented each time a new instance is created. So to get a list of properties sorted in declaration order, you can do something like:

sorted(Foo.properties().items(), key=lambda np: np[1].creation_counter)

(where Foo is an instance of db.Model)

If using google.appengine.ext.ndb:

Same thing, except the ndb.Property attribute is called _creation_counter, so the equivalent code would be:

sorted(Foo._properties.items(), key=lambda np: np[1]._creation_counter)

(where Foo is an instance of ndb.Model)