Storing song, artist and album data on App Engine

Question

I need to store information on artists, albums and songs in Google App Engine for a project I'm working on. The information is meta data taken from a directory of MP3s (using Python) that needs to be sent to App Engine for display to users. Along with the meta data, the song's path will need to be stored.

Currently while scanning I'm storing the data in an list of dictionaries named Artists, each artist dictionary has a name and a list of Album dictionaries and each Album dictionary has a name and list of song dictionaries, each song then contains some meta data and the path to the MP3.

I've been thinking of ways to store this data and have tried sending the data in JSON format to App Engine, then processing this into three models: Artist, containing the name and a repeated KeyProperty for each Album, Album then has a name and a repeated KeyProperty for each song and the song contains the rest of the meta data. Each of these will also contain a KeyProperty related the the Group that they belong to.

The problems with this are: Lots of repeated data (Group Keys) and processing the data not only often exceeds the request deadline, but also uses an obscene amount of datastore writes.

The only way I could think of to get around these problems would be to store the JSON provided after the scan as a JsonProperty and then pass this directly to the user for processing on the client side using JavaScript. The only issue I could see with that is that I don't particularly want to provide the path to the user (as this will need to be passed back and actioned on).

Does anyone have experience using or storing this kind of data, or can provide any outside the box solutions?

Answer 1

What you probably want is a big Songs table with everything in it. Your table would look something like:

Song | Artist | Album | Meta data | Track Number | Release Year | etc

Quit thinking of this as a relational database. Think of it instead as how your user is going to retrieve the data. All your searching is going to be from that main table, most likely.

You'll want an Artists table as well but that could just contain the artist and their bio. If you want to see all their songs, then just filter the Songs table by the artist name or ID. If you want to see their albums, you can do the same thing and use a distinct clause (or just stuff them all into a set).

Album is very similar to Artists except it might have its own metadata.

If you want code, try this other similar question.

Answer 2

I would process the data from json, and place it in Model. As far as schema goes, you really need not worry about having redundancies as you cannot really think of the ndb as a relational database. So do not bother yourself too much about normalising the schema.

But don't process on the client side, it is really not a good way to design it like that.