Taking thread dumps in production

Question

I am analyzing the differences between approaches for taking thread dumps. Below are the couple of them I am researching on

1. Defining a jmx bean which triggers jstack through Runtime.exec() on clicking a declared bean operation.

2. Daemon thread executing "ManagementFactory.getThreadMXBean().dumpAllThreads(true, true)" repeatedly after a predefined interval.

Comparing the thread dump outputs between the two, I see the below disadvantages with approach 2

1. Thread dumps logged with approach 2 cannot be parsed by open source thread dump analyzers like TDA
2. The ouput does not include the native thread id which could be useful in analyzing high cpu issues (right?)
3. Any more?

I would appreciate to get suggestions/inputs on

1. Are there any disadvantages of executing jstack through Runtime.exec() in production code? any compatibility issues on various operating systems - windows, linux?

2. Any other approach to take thread dumps?

Thank you.

Edit -

A combined approach of 1 and 2 seems to be the way to go. We can have a dedicated thread running in background and printing the thread dumps in the log file in a format understood by the thread dump analyzers. If any extra information is need (like say probably the native thread id) which is logged only by the jstack output, we do it manually as required.

Answer 1

With Java 8 in picture, jcmd is the preferred approach.

jcmd <PID> Thread.print

Following is the snippet from Oracle documentation :

The release of JDK 8 introduced Java Mission Control, Java Flight Recorder, and jcmd utility for diagnosing problems with JVM and Java applications. It is suggested to use the latest utility, jcmd instead of the previous jstack utility for enhanced diagnostics and reduced performance overhead.

However, shipping this with the application may be licensing implications which I am not sure.

Answer 2

You can use

jstack {pid} > stack-trace.log

running as the user on the box where the process is running.

If you run this multiple times you can use a diff to see which threads are active more easily.

---

For analysing the stack traces I use the following sampled periodically in a dedicated thread.

 Map<Thread, StackTraceElement[]> allStackTraces = Thread.getAllStackTraces();

Using this information you can obtain the thread's id, run state and compare the stack traces.

Answer 3

I'd suggest you do all the heap analysis on a staging environment if there is such an env, then reflect your required Application Server tuning on production if any. If you need the dumps for analysis of your application's memory utilization, then perhaps you should consider profiling it for a better analysis.

Heap dumps are usually generated as a result of OutOfMemoryExceptions resulting from memory leaks and bad memory management.

Check your Application Server's documentation, most modern servers have means for producing dumps at runtime aside from the normal cause I mentioned earlier, the resulting dump might be vendor specific though.

Answer 4

If its a *nix I'd try kill -3 <PID>, but then you need to know the process id and maybe you don't have access to console?