Tomcat 6 going down after reaching its maximum num

Question

Our Tomcat 6.0.29 goes down after reaching its maximum number of Threads. I would really appreciate any help with it because it is a production server.

Here is part of the catalina.log file:

INFO: Maximum number of threads (600) created for connector with address null and port 80
Mar 8, 2011 11:19:37 AM org.apache.coyote.http11.Http11Protocol pause

INFO: Pausing Coyote HTTP/1.1 on http-80
Mar 8, 2011 11:19:38 AM org.apache.catalina.core.StandardService stop

INFO: Stopping service Catalina
Mar 8, 2011 11:19:38 AM org.apache.catalina.core.StandardWrapper unload

INFO: Waiting for 8 instance(s) to be deallocated

Answer 1

Have a look at swap memory and other resources. I had an experience where the memory usage of each thread was too high (bad app design), causing the server to become unresponsive. If this is the case (or simply to test) reduce the number of threads and see if it still happens.

Answer 2

It looks like your Executor is set to 500 or 600 threads. If each thread usually accesses a DB connection, do you have the dbcp pool set to the same amount or more?

You might also check your dbcp pool configuration

Answer 3

You could give BTrace a go.

Attach a probe on org.apache.catalina.startup.Catalina class, stopServer methods. If that doesn't help, try finding any other shutdown hooks and attaching BTrace profiler to them.

You should be able to get the full stack trace pointing to whoever requested that shutdown.

Answer 4

Take a Thread Dump to see what is holding onto all your threads. The stacktrace for each thread will be an indication of your problem more then the catalina.out error was.

Windows

on the console window press ctrl + break

Unix

on the command line enter: kill -3 <pid>

These can be safely taken on a running production instance.