I am currently experiencing some onStop issues with my play application under Tomcat. I am using play 2.2.2, sbt 0.13.0, scala 2.10.4 and Tomcat 7 and jdk1.6.
To create a war file I am using the play2war plugin(1.2) with:
Play2WarKeys.servletVersion := "2.5"
So deploying and running the application as well as Tomcat itself is running without any issues. But as soon as I try to stop the server with the default shutdown.sh I get
SEVERE: The web application [/WEBSERVICE] appears to have started a thread named [play-scheduler-1] but has failed to stop it. This is very likely to create a memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/WEBSERVICE] appears to have started a thread named [play-akka.actor.default-dispatcher-3] but has failed to stop it. This is very likely to create a memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/WEBSERVICE] appears to have started a thread named [play-akka.actor.default-dispatcher-4] but has failed to stop it. This is very likely to create a memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader clearReferencesThreads
SEVERE: The web application [/WEBSERVICE] appears to have started a thread named [play-akka.actor.default-dispatcher-5] but has failed to stop it. This is very likely to create a memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@384e9bea]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@4e57dc21]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [scala.concurrent.forkjoin.ThreadLocalRandom$1] (value [scala.concurrent.forkjoin.ThreadLocalRandom$1@4679cf8c]) and a value of type [scala.concurrent.forkjoin.ThreadLocalRandom] (value [scala.concurrent.forkjoin.ThreadLocalRandom@67291479]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@384e9bea]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@39ff48d8]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@384e9bea]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@27077aa7]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [akka.actor.ActorCell$$anon$1] (value [akka.actor.ActorCell$$anon$1@5c057df5]) and a value of type [scala.collection.immutable.Nil$] (value [List()]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [java.lang.ThreadLocal] (value [java.lang.ThreadLocal@384e9bea]) and a value of type [scala.concurrent.forkjoin.ForkJoinPool.Submitter] (value [scala.concurrent.forkjoin.ForkJoinPool$Submitter@6c908f05]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
Aug 21, 2014 6:15:18 PM org.apache.catalina.loader.WebappClassLoader checkThreadLocalMapForLeaks
SEVERE: The web application [/WEBSERVICE] created a ThreadLocal with key of type [scala.concurrent.forkjoin.ThreadLocalRandom$1] (value [scala.concurrent.forkjoin.ThreadLocalRandom$1@4679cf8c]) and a value of type [scala.concurrent.forkjoin.ThreadLocalRandom] (value [scala.concurrent.forkjoin.ThreadLocalRandom@69dc8f2]) but failed to remove it when the web application was stopped. Threads are going to be renewed over time to try and avoid a probable memory leak.
After that the application and tomcat are down but I still see the tomcat prozess zombieing through
ps -ef
The only possibility to kill it completly is by kill -9 <pid>
.
So I started searching through the web an stumbled upon similar problems suggesting to perform the .shutdown()
and awaitTermination()
on the actor system that is used.
So I created a Global object with overriding the onStop method:
object Global extends GlobalSettings {
val actorSystem = Application.system
override def onStop(app: Application) {
implicit val timeout = Timeout(4 seconds)
Logger.info("Shutting down Actorsystem")
Akka.system.shutdown()
Akka.system.awaitTermination(timeout.duration)
actorSystem.shutdown()
actorSystem.awaitTermination(timeout.duration)
}
}
But it won't solve the issue. I tried to shutdown my own ActorSystem as well as the default play actorsystem through Akka.system
but it has no effect.
The onStop method gets execute as I see the Log statements in catalina.out.
So to nail down the problem I set up a completly new play 2.3.3 with just a simple string response. Without a actor system besides the play default one and integrated the play2war plugin to see if the problems are caused by my code or by play itself. And the issue was the same.
So I am locking for some advice how it could be possible to shutdown these scheduler and dispatchers and so on that are spawned by play but not killed on shutdown?
I really would appreciate any help!
Edit:
I also tried the solution provided in Oracle driver memory leak - Tomcat by removing the oracle driver in my application lib - but without any change in the tomcat behaviour
There are also multiple Topics within Play2war Github based on this topic but unfortunatly there are no solutions:
Leak error and Tomcat is never shutdown #108
tomcat 6: hangs on shutdown #161