Tomcat doesn't stop. How can I debug this?

Question

I have a Tomcat 7 running in Linux that I start via $CATALINA_HOME/bin/startup.sh and shutdown via $CATALINA_HOME/bin/shutdown.sh
from /etc/init.d

All is ok except 1 problem. Sometimes tomcat does not stop.
Although I stop it and I see in catalina.out logs that is going down, if I do ps -ef I can still see the process running.

What could be the problem? How can I debug this? My feeling is, that this is related to threads.

So the parts that are suspicious are the following:
1) I use Log4j's LogManager to detect if the log4j configuration has been changed, but I do Log4jManager.shutdown on a contextDestroyed ServletContextListener
2) I use H2 database and I see on shutdown:

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 Log Writer MYAPPLICATION] but has failed to stop it.
This is very likely to create a memory leak

SEVERE: The web application [/MyApplication] appears to have started a
thread named [H2 File Lock Watchdog
/opt/myOrg/tomcat/webapps/MyApplication/db/myDatabase.lock.db] but has
failed to stop it. This is very likely to create a memory leak. Apr 2,
2012 9:08:08 AM org.apache.catalina.loader.WebappClassLoader
clearReferencesThreads SEVERE: The web application [/MyApplication]
appears to have started a thread named [FileWatchdog] but has failed
to stop it. This is very likely to create a memory leak.

Any help please? How can I detect the problem here?

UPDATE:
I did a kill -3 as suggested by @daveb, and in the catalina.out I see:

JVMDUMP006I Processing dump event "user", detail "" - please wait. JVMDUMP032I JVM requested Java dump using '/etc/init.d/javacore.20120402.093922.2568.0001.txt' in response to an event JVMDUMP010I Java dump written to /etc/init.d/javacore.20120402.093922.2568.0001.txt JVMDUMP013I Processed dump event "user", detail "".

There is a javacore in /etc/init.d but I don't know how to process it. I.e. what parts should I investigate

Answer 1

In my case I had one rogue JPA EntityManager that wasn't being properly closed after use. Fixed that, and now I can Clean-and-Build again without killing the damn Java process everytime :)

Answer 2

I also had the same problem. There was a ThrottledThreadPoolExecutor in my application that wasn't getting shutdown. When I shut it down properly, tomcat would stop cleanly. In order to figure out the problem, I had to remove all the apps from my tomcat webapps directory and then add them one by one and see which one was causing the problem

Answer 3

Check if your Web Application has some Scheduler active, like Quartz.

If you don't stop it, Web Application Thread never ending until you kill it

Answer 4

Find out what threads are still running (or blocked, waiting to run) by using jstack or sending a signal to the process:

kill -3 pid

When you know this, you can make whatever it was that started them hook into the shutdown notification to stop the threads. Or make those threads deamon threads.

See This tomcat shutdown question for more details on this.

If you don't know where your threads were created, then consider adding names to them - executors can take thread factories, and you can use those factories to set the deamon status of a thread and also to name it - so your stack trace will be clearer.

Answer 5

I had the exact same problem. Sometimes, the command ./shutdown.sh does not stop the tomcat process, and its java process stays in the running processes.

I had solved this problem using the Tomcat version in the Ubuntu's software repositories, by:

sudo apt-get install tomcat7

After installing it from package manager and configuring some settings, I did not have any problems on stopping/starting Tomcat. I used this command to stop, and it never failed:

service tomcat7 stop

which is nearly the same as

/etc/init.d/tomcat7 stop

Using this command runs the code block from the init script, specifically, the codes from the file /etc/init.d/tomcat7. So I looked into it to see what it does to always kill the tomcat process succesfully. Here is the code block that runs when you use service tomcat7 stop command:

log_daemon_msg "Stopping $DESC" "$NAME"

        set +e
        if [ -f "$CATALINA_PID" ]; then
                start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                        --user "$TOMCAT7_USER" \
                        --retry=TERM/20/KILL/5 >/dev/null
                if [ $? -eq 1 ]; then
                        log_progress_msg "$DESC is not running but pid file exists, cleaning up"
                elif [ $? -eq 3 ]; then
                        PID="`cat $CATALINA_PID`"
                        log_failure_msg "Failed to stop $NAME (pid $PID)"
                        exit 1
                fi
                rm -f "$CATALINA_PID"
                rm -rf "$JVM_TMP"
        else
                log_progress_msg "(not running)"
        fi
        log_end_msg 0
        set -e
        ;;

The important part is this:

start-stop-daemon --stop --pidfile "$CATALINA_PID" \
                            --user "$TOMCAT7_USER" \
                            --retry=TERM/20/KILL/5 >/dev/null

This means "retry stopping until the process is stopped. Here is the --retry command documentation from start-stop-daemon manual:

   -R|--retry timeout|schedule
          With  --stop,  specifies  that  start-stop-daemon  is  to  check
          whether  the  process(es)  do  finish.  It will check repeatedly
          whether any matching processes are running, until none are.   If
          the  processes  do  not exit it will then take further action as
          determined by the schedule.

          If timeout is specified instead of schedule  then  the  schedule
          signal/timeout/KILL/timeout  is used, where signal is the signal
          specified with --signal.
          ...

So, --retry=TERM/20/KILL/5 means "Send TERM signal to the process, wait 20 seconds, if it's still running, send KILL signal, wait 5 seconds, if it's still running, there is a problem.

This means you can configure the tomcat to run as a deamon and use a command like this, or write a script to do that kind of action to stop tomcat, or just use Ubuntu and get the tomcat from the package manager.

Answer 6

If you're using a Scheduler or some other entity in your web app, you need to shut it down. Typically you have to use a ServletContextListener to provide the hook to make your shutdown call. A shutdown hook won't work in this case because the JVM is not shutting down (yet). Believe me, I've tried. If your code is in agent code or something outside the container/webapp, then a shutdown hook SHOULD work, though often it's a hair pulling experience to figure out why it's STILL not working. Note, I'm bald.