App Engine Errors when trying to insert many entit

Question

I want to insert many user entities in bulk. User entity has property Name & UserNo. The requirement is that the property UserNo will automatically increase by 1 every time a User entity was inserted.

The following code works fine if I try to insert 1 record at a time.

public void insertUser(String name){
    Entity userEntity=new Entity("User");
    long maxID=Utility.getPropertyMaxID("User", "UserNo")+1;
    userEntity.setProperty("Name",name);
    datastore.put(userEntity); 
}

public static long getPropertyMaxID(String entityName, String propertyName){
        // Order alphabetically by property name:
        Query q = new Query(entityName)
                        .addSort(propertyName, SortDirection.DESCENDING);

        List<Entity> results = datastore.prepare(q)
                .asList(FetchOptions.Builder.withDefaults());
        if(results.size()>0){
            Entity e=results.get(0);
            long maxID=(Long)e.getProperty(propertyName);
            return maxID;
        }
        else{
            return 0;
        }

}

However, if I try to insert many entities at once

for(int i=0; i<10; i++){
    insertUser(names[i]);
}

Then, when checking data I can see the name got inserted correctly. However, the UserNo got messed up as it did not increase 1 by 1 correctly.

So, I think this could be the "Eventual consistency" of Google App Engine datastore because the data takes time to be inserted so the program may take the maxUserNo incorrectly.

So, what should I do? Should not not to use "UserNo"?

I did try Transaction, but it didn't work anyway

public void insertUser(String name){
     Transaction tx=datastore.beginTransaction();
     try{
        Entity userEntity=new Entity("User");
        long maxID=Utility.getPropertyMaxID("User", "UserNo")+1;
        userEntity.setProperty("Name",name);
        datastore.put(userEntity); 
        tx.commit();}
     catch (Exception ex){ ex.printStackTrace()};
}

So, what is the best solution for this?

Note: I heard that "Non-ancestor queries are always eventually consistent". SO the query inside getPropertyMaxID method is not Strongly consistent.

Extra: what about if I make a fake Person and force all Users to be children of that person & then make Ancestor query on all Users. Loop though all users to check the email for example? See this code:

Entity personEntity=new Entity("Person", "Tom");
Key personKey=personEntity.getKey();
Entity userEntity=new Entity("User", userName);
userEntity.setProperty("Email", email);
datastore.put(userEntity);

Then make a function to check the unique email

public static boolean checkUniqueEmail(String email){
Entity personEntity = new Entity("Person", "Tom");
        Key personKey = personEntity.getKey();

        Query query = new Query("User")
        .setAncestor(personKey);  


        List<Entity> results = datastore.prepare(query)
                       .asList(FetchOptions.Builder.withDefaults());

        for(Entity e : results){

            String em=(String)e.getProperty("Email");
             if(e.equals(email)) return false;
        }
}

Answer 1

I found the answer

Check this link

To understand how to structure your data for strong consistency, compare two different approaches for the guestbook example application from the App Engine Getting Started exercise. The first approach creates a new root entity for each greeting:

import com.google.appengine.api.datastore.Entity;

Entity greeting = new Entity("Greeting");
// No parent key specified, so Greeting is a root entity.

greeting.setProperty("user", user);
greeting.setProperty("date", date);
greeting.setProperty("content", content);

It then queries on the entity kind Greeting for the ten most recent greetings.

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

Query query = new Query("Greeting")
                    .addSort("date", Query.SortDirection.DESCENDING);

List<Entity> greetings = datastore.prepare(query)
                                  .asList(FetchOptions.Builder.withLimit(10));

However, because we are using a non-ancestor query, the replica used to perform the query in this scheme may not have seen the new greeting by the time the query is executed. Nonetheless, nearly all writes will be available for non-ancestor queries within a few seconds of commit. For many applications, a solution that provides the results of a non-ancestor query in the context of the current user's own changes will usually be sufficient to make such replication latencies completely acceptable.

If strong consistency is important to your application, an alternate approach is to write entities with an ancestor path that identifies the same root entity across all entities that must be read in a single, strongly-consistent ancestor query:

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

String guestbookName = req.getParameter("guestbookName");
Key guestbookKey = KeyFactory.createKey("Guestbook", guestbookName);
String content = req.getParameter("content");
Date date = new Date();

// Place greeting in same entity group as guestbook
Entity greeting = new Entity("Greeting", guestbookKey);
greeting.setProperty("user", user);
greeting.setProperty("date", date);
greeting.setProperty("content", content);
You will then be able to perform a strongly-consistent ancestor query within the entity group identified by the common root entity:

import com.google.appengine.api.datastore.DatastoreService;
import com.google.appengine.api.datastore.DatastoreServiceFactory;
import com.google.appengine.api.datastore.Entity;

DatastoreService datastore = DatastoreServiceFactory.getDatastoreService();

Key guestbookKey = KeyFactory.createKey("Guestbook", guestbookName);
Query query = new Query("Greeting", guestbookKey)
                    .setAncestor(guestbookKey)
                    .addSort("date", Query.SortDirection.DESCENDING);

List<Entity> greetings = datastore.prepare(query)
                                  .asList(FetchOptions.Builder.withLimit(10));

This approach achieves strong consistency by writing to a single entity group per guestbook, but it also limits changes to the guestbook to no more than 1 write per second (the supported limit for entity groups). If your application is likely to encounter heavier write usage, you may need to consider using other means: for example, you might put recent posts in a memcache with an expiration and display a mix of recent posts from the memcache and the Datastore, or you might cache them in a cookie, put some state in the URL, or something else entirely. The goal is to find a caching solution that provides the data for the current user for the period of time in which the user is posting to your application. Remember, if you do a get, an ancestor query, or any operation within a transaction, you will always see the most recently written data.