Java neo4j, REST and memory

2020-04-14 02:17发布

问题:

I've deployed an app using neo4j java embedded version under Jersey tomcat for REST API. By measuring memory usage with jconsole I noticed each REST call adds 200Mb of memory (which I think it's because the entire graph gets loaded into memory). Therefore with just 5 calls, server allocates 1Gb of memory, which is a lot! To clean up memory I have to wait garbage collector (threshold set to 1Gb).

Is it a normal behavior because I'm using the neo4j java embedded version or am I doing something terribly wrong? What I'm supposed to do to free memory when the API call ends?

Here example code:

@GET
@Produces(MediaType.APPLICATION_JSON + ";charset=utf-8")
public Response getApi( @QueryParam("q")    String query){
try{
// new neo instance here with EmbeddedGraphDatabase
... some code
// stop neo
}catch(Exception ex){
// stop neo
}
return response.ok("json data here").build();   
}

Thanks, Daniele

-------- COMPLETE CLASS CODE ----------

 import javax.ws.rs.GET;
import javax.ws.rs.Path;
import javax.ws.rs.Produces;
import javax.ws.rs.QueryParam;
import javax.ws.rs.core.MediaType;
import javax.ws.rs.core.Response;

import org.neo4j.graphdb.GraphDatabaseService;
import org.neo4j.kernel.EmbeddedGraphDatabase;

@Path("/API")
public class API {

     @GET
     @Produces(MediaType.APPLICATION_JSON)
     public Response apiCall(@QueryParam("q") String query){
        GraphDatabaseService graphDb; 
        try{
            // start neo
            graphDb = new EmbeddedGraphDatabase( "/var/neo4jdb/" );
            this.registerShutdownHook( graphDb );

            // API + NEO code here..

            // stop neo
            graphDb.shutdown();

        }catch(Exception ex){
            // stop neo
            graphDb.shutdown();
        }

        Response response = null;
        return response.ok("This is your query: "+query).build(); 

     }

    /**
     * Server shutdown
     */
    public void registerShutdownHook( final GraphDatabaseService graphDb ){
        // Registers a shutdown hook for the Neo4j instance so that it
        // shuts down nicely when the VM exits (even if you "Ctrl-C" the
        // running example before it's completed)
        Runtime.getRuntime()
                .addShutdownHook( new Thread()
                {
                    @Override
                    public void run()
                    {
                        graphDb.shutdown();
                    }
                } );
    }

}

And then I call the REST service via browser like this http://localhost:8080/API?q=test

UPDATED WITH SINGLETON

     import javax.ws.rs.GET;
    import javax.ws.rs.Path;
    import javax.ws.rs.Produces;
    import javax.ws.rs.QueryParam;
    import javax.ws.rs.core.MediaType;
    import javax.ws.rs.core.Response;

    import org.neo4j.graphdb.GraphDatabaseService;
    import org.neo4j.kernel.EmbeddedGraphDatabase;

    @Path("/API")
    public class API {

         @GET
         @Produces(MediaType.APPLICATION_JSON)
         public Response apiCall(@QueryParam("q") String query){
            GraphDatabaseService graphDb; 
            try{
                // start neo
                Neo4jSingleton neo4jInstance = new Neo4jSingleton();
                GraphDatabaseService graphDb = null;
                graphDb = neo4jInstance.getInstance(DB_PATH);
                this.registerShutdownHook( graphDb );

                // API + NEO code here..
                // cypher query
                        ExecutionEngine engine = new ExecutionEngine(graphDb); 
                        String queryString = "Cypher query code";
                        ExecutionResult result = engine.execute( queryString );
                        // fetch results here..

                // never stop neo now with singleton


            }catch(Exception ex){
                // stop neo
                graphDb.shutdown();
            }

            Response response = null;
            return response.ok("This is your query: "+query).build(); 

         }

        /**
         * Server shutdown
         */
        public void registerShutdownHook( final GraphDatabaseService graphDb ){
            // Registers a shutdown hook for the Neo4j instance so that it
            // shuts down nicely when the VM exits (even if you "Ctrl-C" the
            // running example before it's completed)
            Runtime.getRuntime()
                    .addShutdownHook( new Thread()
                    {
                        @Override
                        public void run()
                        {
                            graphDb.shutdown();
                        }
                    } );
        }

    }
public class Neo4jSingleton {

    private static GraphDatabaseService db;

    public Neo4jSingleton() {

    }

    /*
     * il metodo di restituire un'unica istanza
     * contenente il database neo4j
     */
    public static GraphDatabaseService getInstance(String DB_PATH)
      {

        //Boolean isDbChange=verifyDbChange();

        if (db == null /*|| isDbChange*/)
        {
          db = new EmbeddedGraphDatabase(DB_PATH);
        }

        return db;
      }
}

回答1:

You must not create Neo4j instance for every request. Please create it just once and then pass it in, either hacky in a static field (as you're resource instance is recreate for each request) or as in Neo4j server with an Provider which is injected with @Context.

Regarding memory usage. Neo4j builds up internal caches according to your usage to serve the same queries faster the next time. So that might amount for some of the used memory.

Btw. how large is your graph and what are the typical operations you do?



回答2:

Java won't necessarily reclaim the memory as soon as it is available to be GCed. You might find that you're not seeing significant GC action until you get close to the heap limit. So even if you put your heap limit to 10gb, then you might find memory jumps up again. That isn't necessarily a problem.

However, to solve the issue of neo using N x Memory, you should consider sharing the embedded instance between all your endpoints. Perhaps by moving it to a service class and having a shared instance between the endpoints. If you are using Spring with Jersey then this would be easy to do as you could wire it in by Spring.



回答3:

If you're not using Spring or Guice you could get around this by using a static member and static initialization for your neo4J object. If having that static is a deal breaker another option would be to use a static factory to get at a singleton instance of that neo4J object...