Unable to shut down Neo4j Jetty server within Tomc

Currently I am working on an webapp that utilizes Neo4j. Our app needs to be deployed in a Tomcat environment (customer requirement). We decided to embed Neo4j, because that way we can use the Neo4j provided Java API, it's easier to deploy and we get improved performance. However, we also need access to the REST API, because we have a single page webapp written in Angular that is currently making use this interface. The embedded Neo4j database however does not expose the REST api. The Neo4j-server artifact contains code that can bootstrap a jetty server with an embedded graph database. So our Tomcat webapp is starting a Jetty server. We can access the graphDb via the webapp deployed in Tomcat on one port (8080) and the Neo4j REST interface and Neo4j browser on another port (7474). Though this is a bit odd it works fine, except for when we try to stop our webapp (for redeployment for example). When shutting down our webapp we receive these errors from tomcat:

SEVERE: The web application [/chainmonitor] appears to have started a thread named [GC-Monitor] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [RRD4J Sync Pool [Thread-1]] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [Statistics Gatherer[primitives]] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-1] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [pool-1-thread-2] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [DateCache] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-31-selector-3] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-32-selector-0] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-33-selector-1] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-34-selector-2] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-35-selector-4] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-36-selector-5] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-37-acceptor-0-ServerConnector@1425c689{HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-38-acceptor-1-ServerConnector@1425c689{HTTP/1.1}{localhost:7474}] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [HashSessionScavenger-0] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/chainmonitor] appears to have started a thread named [qtp835579386-54] but has failed to stop it. This is very likely to create a memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [org.eclipse.jetty.http.HttpFields$1] (value [org.eclipse.jetty.http.HttpFields$1@4a3d3a5e]) and a value of type [org.eclipse.jetty.http.HttpFields.DateGenerator] (value [org.eclipse.jetty.http.HttpFields$DateGenerator@24f0137b]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:17 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/chainmonitor] created a ThreadLocal with key of type [scala.util.DynamicVariable$$anon$1] (value [scala.util.DynamicVariable$$anon$1@281a099d]) and a value of type [java.lang.Integer] (value [0]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.HostConfig undeploy
INFO: Undeploying context [/chainmonitor]
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF\lib] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor\WEB-INF] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar deleteDir
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems
jun 10, 2014 6:43:27 PM org.apache.catalina.startup.ExpandWar delete
SEVERE: [C:\dev\apache-tomcat-7.0.54\webapps\chainmonitor] could not be completely deleted. The presence of the remaining files may cause problems

Our pom.xml looks like this:

    <dependency>
        <groupId>org.neo4j</groupId>
        <artifactId>neo4j</artifactId>
        <version>2.0.3</version>
    </dependency>
    <dependency>
        <groupId>org.neo4j.app</groupId>
        <artifactId>neo4j-server</artifactId>
        <version>2.0.3</version>
    </dependency>

This is how we start the embedded database and start the jetty server:

graphDb = (EmbeddedGraphDatabase) new GraphDatabaseFactory().
      newEmbeddedDatabaseBuilder(dataDir).loadPropertiesFromURL(Neo4jPropertiesUrl).newGraphDatabase();
bootstrapper = new WrappingNeoServerBootstrapper(graphDb);

Currently neo4j-server.properties is empty. I've not tried many different configurations, but doing so doesn't seem to be a problem.

This is how we stop the embedded database and jetty server:

    @Override
    public void contextDestroyed(ServletContextEvent event) {
            graphDb.shutdown();
            bootstrapper.stop();
    }

This is apparently caused by Tomcat's memory leak protection (http://wiki.apache.org/tomcat/MemoryLeakProtection).

The result is that our webapp can only be redeployed by completely killing the Tomcat process (kill -9), which is very undesirable.

We've tried a bunch of things:

not shutting down the embedded grapbDb. No effect.
not explicitly shutting down the bootstrapper. No effect. (the bootstrapper has its own shutdown hook)
shutting down the bootstrapper first and then the embedded database.
thread immolator: https://github.com/Neo4j/Neo4j/issues/1070 . The code runs and immolates 59 thread local values, but with no apparent effect. Interestingly when I run this in a loop for a minute during the shutdown it keeps saying it immolated 6 values. To me this is an indication that this process does not work at all, because Neo4j takes some time while shutting down.
putting in a delay after the shutting down. The reason we tried this is because after tomcat has received a shutdown signal we need to wait a few seconds before actually killing the process otherwise we get an error on our next start that our graph data has been corrupted (with a message that is has not been properly shut down). Delaying by 5+ seconds seems to have an effect too, because afterwards we get only 6 severe errors instead of 16. This is also the reason why I think the immolation process is not working at all.
We tried as an alternative to add our embedded Neo4j to a cluster, thus being able to use both the REST api as wel as the embedded Neo4j. However after about four hours trying to get this to work we gave up. During the startup of the embedded Neo4j database our webapplication just froze. After some debugging it very much seemed like a deadlock within Neo4j's code. We tried to set all HA timeout configurations to some smaller values, but with no effect.

The problem is caused by the Neo4j jetty server. Without it this problem does not arise. We need the REST API though. Do you know how we can fix this? Or have you got an alternative way to use the embedded database and the REST API within a Tomcat environment?

We are using:

Java: 1.7.0_60
Tomcat 7.0.54
Neo4j: 2.0.3 (we have tried 2.1.1 but got exactly the same result)
The same behviour has been observed on Windows 7 and Linux Redhat

回答1:

You might have leaking transactions which are not closed yet, then Neo4j waits up to 20 seconds on shutdown to close those transactions.

If you wait 5+ seconds which of those reported issues remain?

回答2:

We have a very similar setup and need. Here is the solution we have for this issue. The JVM Runtime shutdownHook does not work in Tomcat, so we hold a reference to a wrapper object holding our GraphDatabaseService and place that wrapper in a repository manager object. Use the @PreDestroy annotation on a method in the repository manager object which will shutdown the repositories before the repository manager class is destroyed.

import javax.annotation.PreDestroy;

...

@PreDestroy
public void destroy() {
    log.info("Destroying Neo4jRepositoryManager...");
    shutdown(Neo4jPropertyKeys.SANDBOX_DATABASE.getUrlKey());
    shutdown(Neo4jPropertyKeys.PRODUCTION_DATABASE.getUrlKey());
    shutdown(Neo4jPropertyKeys.TEST_DATABASE.getUrlKey());
}

private void shutdown(String urlKey) {
    if(repositories.containsKey(urlKey)) {
        LabeledNodeGraphRepository graphdb = repositories.get(urlKey);
        graphdb.getGraphDatabaseService().shutdown();
        log.info(urlKey + " shutdown...");
    }
}

<bean id="repositoryManager" class="mypath.neo4j.repository.Neo4jRepositoryManager"/>

Our environment is: