Enforcing Unique Constraint in GAE

Question

I am trying out Google App Engine Java, however the absence of a unique constraint is making things difficult. I have been through this post and this blog suggests a method to implement something similar. My background is in MySQL.Moving to datastore without a unique constraint makes me jittery because I never had to worry about duplicate values before and checking each value before inserting a new value still has room for error.

"No, you still cannot specify unique during schema creation."

-- David Underhill talks about GAE and the unique constraint (post link)

What are you guys using to implement something similar to a unique or primary key?

I heard about a abstract datastore layer created using the low level api which worked like a regular RDB, which however was not free(however I do not remember the name of the software)

Schematic view of my problem

sNo = biggest serial_number in the db
sNo++
Insert new entry with sNo as serial_number value //checkpoint
User adds data pertaining to current serial_number 
Update entry with data where serial_number is sNo 

However at line number 3(checkpoint), I feel two users might add the same sNo. And that is what is preventing me from working with appengine.

Answer 1

This and other similar questions come up often when talking about transitioning from a traditional RDB to a BigTable-like datastore like App Engine's.

It's often useful to discuss why the datastore doesn't support unique keys, since it informs the mindset you should be in when thinking about your data storage schemes. The reason unique constraints are not available is because it greatly limits scalability. Like you've said, enforcing the constraint means checking all other entities for that property. Whether you do it manually in your code or the datastore does it automatically behind the scenes, it still needs to happen, and that means lower performance. Some optimizations can be made, but it still needs to happen in one way or another.

The answer to your question is, really think about why you need that unique constraint.

Secondly, remember that keys do exist in the datastore, and are a great way of enforcing a simple unique constraint.

my_user = MyUser(key_name=users.get_current_user().email())
my_user.put()

This will guarantee that no MyUser will ever be created with that email ever again, and you can also quickly retrieve the MyUser with that email:

my_user = MyUser.get(users.get_current_user().email())

In the python runtime you can also do:

my_user = MyUser.get_or_create(key_name=users.get_current_user().email())

Which will insert or retrieve the user with that email.

Anything more complex than that will not be scalable though. So really think about whether you need that property to be globally unique, or if there are ways you can remove the need for that unique constraint. Often times you'll find with some small workarounds you didn't need that property to be unique after all.

Answer 2

You can generate unique serial numbers for your products without needing to enforce unique IDs or querying the entire set of entities to find out what the largest serial number currently is. You can use transactions and a singleton entity to generate the 'next' serial number. Because the operation occurs inside a transaction, you can be sure that no two products will ever get the same serial number.

This approach will, however, be a potential performance chokepoint and limit your application's scalability. If it is the case that the creation of new serial numbers does not happen so often that you get contention, it may work for you.

EDIT: To clarify, the singleton that holds the current -- or next -- serial number that is to be assigned is completely independent of any entities that actually have serial numbers assigned to them. They do not need to be all be a part of an entity group. You could have entities from multiple models using the same mechanism to get a new, unique serial number.

I don't remember Java well enough to provide sample code, and my Python example might be meaningless to you, but here's pseudo-code to illustrate the idea:

1. Receive request to create a new inventory item.
2. Enter transaction.
3. Retrieve current value of the single entity of the SerialNumber model.
4. Increment value and write it to the database
5. Return value as you exit transaction.

Now, the code that does all the work of actually creating the inventory item and storing it along with its new serial number DOES NOT need to run in a transaction.

Caveat: as I stated above, this could be a major performance bottleneck, as only one serial number can be created at any one time. However, it does provide you with the certainty that the serial number that you just generated is unique and not in-use.

Answer 3

I encountered this same issue in an application where users needed to reserve a timeslot. I needed to "insert" exactly one unique timeslot entity while expecting users to simultaneously request the same timeslot.

I have isolated an example of how to do this on app engine, and I blogged about it. The blog posting has canonical code examples using Datastore, and also Objectify. (BTW, I would advise to avoid JDO.)

I have also deployed a live demonstration where you can advance two users toward reserving the same resource. In this demo you can experience the exact behavior of app engine datastore click by click.

If you are looking for the behavior of a unique constraint, these should prove useful.

-broc

Answer 4

I first thought an alternative to the transaction technique in broc's blog, could be to make a singleton class which contains a synchronized method (say addUserName(String name)) responsible adding a new entry only if it is unique or throwing an exception. Then make a contextlistener which instantiates a single instance of this singleton, adding it as an attribute to the servletContext. Servlets then can call the addUserName() method on the singleton instance which they obtain through getServletContext.

However this is NOT a good idea because GAE is likely to split the app across multiple JVMs so multiple singleton class instances could still occur, one in each JVM. see this thread

A more GAE like alternative would be to write a GAE module responsible for checking uniqueness and adding new enteries; then use manual or basic scaling with...

<max-instances>1</max-instances>

Then you have a single instance running on GAE which acts as a single point of authority, adding users one at a time to the datastore. If you are concerned about this instance being a bottleneck you could improve the module, adding queuing or an internal master/slave architecture.

This module based solution would allow many unique usernames to be added to the datastore in a short space of time, without risking entitygroup contention issues.