Update all objects in JPA entity

Question

I'm trying to update all my 4000 Objects in ProfileEntity but I am getting the following exception:

javax.persistence.QueryTimeoutException: The datastore operation timed out, or the data was temporarily unavailable.

this is my code:

public synchronized static void  setX4all() 
{

    em = EMF.get().createEntityManager();

    Query query = em.createQuery("SELECT p FROM ProfileEntity p");
    List<ProfileEntity> usersList = query.getResultList();

    int a,b,x;
    for (ProfileEntity profileEntity : usersList) 
    {
        a = profileEntity.getA();
        b = profileEntity.getB();
        x = func(a,b);
        profileEntity.setX(x);

        em.getTransaction().begin();        
        em.persist(profileEntity);        
        em.getTransaction().commit();

    }

    em.close();

}

I'm guessing that I take too long to query all of the records from ProfileEntity. How should I do it?

• I'm using Google App Engine so no UPDATE queries are possible.

Edited 18/10
In this 2 days I tried:
using Backends as Thanos Makris suggested but got to a dead end. You can see my question here.
reading DataNucleus suggestion on Map-Reduce but really got lost.

I'm looking for a different direction. Since I only going to do this update once, Maybe I can update manually every 200 objects or so.
Is it possible to to query for the first 200 objects and after it the second 200 objects and so on?

Answer 1

If you want to set(x) for all object's, better to user update statement (i.e. native SQL) using JPA entity manager instead of fetching all object's and update it one by one.

Answer 2

Put the transaction outside of the loop:

em.getTransaction().begin();        
for (ProfileEntity profileEntity : usersList) {
...
}
em.getTransaction().commit();

Answer 3

Change the update process to use something like Map-Reduce. This means all is done in datastore. The only problem is that appengine-mapreduce is not fully released yet (though you can easily build the jar yourself and use it in your GAE app - many others have done so).

Answer 4

Maybe you should consider the use of the Task Queue API that enable you to execute tasks up to 10min. If you want to update such a number of entities that Task Queues do not fit you, you could also consider the user of Backends.

Answer 5

Your class behaves not very well - JPA is not suitable for bulk updates this way - you just starting a lot of transaction in rapid sequence and produce a lot of load on the database. Better solution for your use case would be scalar query setting all the objects without loading them into JVM first ( depending on your objects structure and laziness you would load much more data as you think )

See hibernate reference: http://docs.jboss.org/hibernate/orm/3.3/reference/en/html/batch.html#batch-direct

Answer 6

Given your scenario, I would advice to run a native update query:

 Query query = em.createNativeQuery("update ProfileEntity pe set pe.X = 'x'");
 query.executeUpdate();

Please note: Here the query string is SQL i.e. update **table_name** set ....

This will work better.