Building a sharded list in Google App Engine

Question

I am looking for a good design pattern for sharding a list in Google App Engine. I have read about and implemented sharded counters as described in the Google Docs here but I am now trying to apply the same principle to a list. Below is my problem and possible solution - please can I get your input?

Problem: A user on my system could receive many messages kind of like a online chat system. I'd like the server to record all incoming messages (they will contain several fields - from, to, etc). However, I know from the docs that updating the same entity group often can result in an exception caused by datastore contention. This could happen when one user receives many messages in a short time thus causing his entity to be written to many times. So what about abstracting out the sharded counter example above:

• Define say five entities/entity groups
• for each message to be added, pick one entity at random and append the message to it writing it back to the store,
• To get list of messages, read all entities in and merge...

Ok some questions on the above:

1. Most importantly, is this the best way to go about things or is there a more elegant/more efficient design pattern?
2. What would be a efficient way to filter the list of messages by one of the fields say everything after a certain date?
3. What if I require a sharded set instead? Should I read in all entities and check if the new item already exists on every write? Or just add it as above and then remove duplicates whenever the next request comes in to read?

Answer 1

why would you want to put all messages in 1 entity group ?

If you don't specify a ancestor, you won't need sharding, but the end user might see some lagging when querying the messages due to eventual consistency.

Depends if that is an acceptable tradeoff.