How to detect the cause of OutofMemoryError?

Question

I have a complaint that a server application of mine is crashing on high load.
It is a web app running in Tomcat 5.
I see the thread dumps and I see that there is an OutOfMemory error

1TISIGINFO Dump Event "systhrow" (00040000) Detail
"java/lang/OutOfMemoryError" "Failed to create thread: retVal -1073741830, errno 12" >received 1TIDATETIME Date: 2012/07/17 at 20:03:17 1TIFILENAME >Javacore filename:C:\ServerApplication\Tomcat5\bin\javacore.34545719.4646464.4172.0003.txt

The heap information is the following:

Maximum Java heap size : 1500m    
Initial Java heap size : 256m

This is the configuration for initial and Max heap size (32 bit java)

I also see that there is available free heap space

1STHEAPFREE    Bytes of Heap Space Free: 2D19F3C0   
1STHEAPALLOC   Bytes of Heap Space Allocated: 5DC00000

This is arround 750MB free space, right?

And from the thread method analysis I see that the number of Threads is 695 of which 49% is java/lang/Object.wait(Native Method) and 39% is in sun/misc/Unsafe.park(Native Method)
Also I see this NO JAVA STACK 1% not sure what that means.
Also 0 treads deadlocked and 2% is Runnable.

I am not sure how to interpret this information or how to go on from here to detect the underlying cause.
Any help on this?

Answer 1

Fix

Followed the IBM technote java.lang.OutOfMemoryError while creating new threads, specifically 'ulimit' command to increase the value from the default 1024.

Symptom

[2/25/15 12:47:34:629 EST] 00000049 SystemErr     R java.lang.OutOfMemoryError: Failed to create a thread: retVal -1073741830, errno 11
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at java.lang.Thread.startImpl(Native Method)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at java.lang.Thread.start(Thread.java:936)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at org.eclipse.osgi.framework.internal.core.InternalSystemBundle.stop(InternalSystemBundle.java:251)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.RuntimeBundleActivator.shutdownEclipse(RuntimeBundleActivator.java:54)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook$1.run(ServerCollaborator.java:878)
[2/25/15 12:47:34:630 EST] 00000049 SystemErr     R     at com.ibm.ws.security.auth.ContextManagerImpl.runAs(ContextManagerImpl.java:5459)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.security.auth.ContextManagerImpl.runAsSystem(ContextManagerImpl.java:5585)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$ShutdownHook.run(ServerCollaborator.java:850)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.runtime.component.ServerCollaborator$StopAction.alarm(ServerCollaborator.java:809)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ejs.util.am._Alarm.run(_Alarm.java:133)
[2/25/15 12:47:34:631 EST] 00000049 SystemErr     R     at com.ibm.ws.util.ThreadPool$Worker.run(ThreadPool.java:1815)

Environment

CentOS 6.6 64 bit IBM WAS 8.5.0.2 64 bit

References

• Guidelines for setting ulimits (WebSphere Application Server)

Answer 2

You should start the JVM with the -XX:+HeapDumpOnOutOfMemoryError flag. This will produce a heap dump when the OutOfMemoryError is generated.

Then, as @Steve said, you can use a tool like MAT to analyze the dump and see which objects are allocated, and who is keeping references to them. This usually will give you some insight on why your JVM is exhausting its memory.

Answer 3

A similar message on IBM WebSphere shows this line

"Failed to create a thread: retVal"

as indicative of a native OOM which means some thread (of the process) is trying to request a large portion of memory on the heap.

The IBM link above has a series of steps - some of which are IBM specific. Have a look.

From a native memory usage perspective:

• Maximum Java heap settings
• JDBC driver
• JNI code or Native libraries
• garbage collection of unused classes. Ensurethat -Xnoclassgc is not set.
• Thread Pool settings (fixed size thread pools)
• Too many classloaders etc, but these are not very common.
• Number of classes/classloaders from javacores.

Another thing you could look at is the PermGenSpace - how large is it?

This link http://www.codingthearchitecture.com/2008/01/14/jvm_lies_the_outofmemory_myth.html suggests

Increasing the heap allocation actually exacerbates this problem! It decreases the headroom the compiler, and other native components, have to play with. So the solution to my problem was: 1.reduce the heap allocated to the JVM. 2. remove the memory leaks caused by native objects not being freed in a timely fashion.

Also have you configured a value in server.xml for maxThreads ? The default is 200 but your app seems to have 695 ?

Answer 4

I know what you mean, it can be confusing to find somewhere to begin.

Have a look at Eclipse Memory Analyzer (MAT). It will use JHat to dump a memory snapshot of your program into a file, which you can re-open and analyze.

The browser for this file outlines all the objects created by the program very neatly, and you can look into various levels to find if something is suspicious.

---

Appending my comments to answer...

Right when your executable webapp crashes, dump it to MAT. MAT will tell you what object is being created a bunch of times. If it's a custom object, and it often is, it's easy to find. If not, you can see its parent, amputate it, and dribble down from there (sorry for the graphic example, I'm not entirely focused on SO at the moment :).

Oh, and I forgot to mention, you can run the program several times under several conditions, and make a dump each time. Then, you can analyze each dump for the trend.

---

But in my case what should I use?I have a web app running in Tomcat

Sorry, missed this too. If I'm not mistaken, MAT dumps the JVM process, so as long as the VM is running on your box you can dump its process and see what's going on.

---

Another comment mutated into partial solution...

This is becoming more difficult than it actually is. Seriously, it's pretty easy, after you run MAT once or twice to get the hang of things. Run your app until the thing crashes. Dump it. Change something. Run, crash, dump. Repeat. Then, open the dumps in MAT, and compare what looks suspicious.

The trickiest part when I was learning this was finding the process ID to dump - which is still not too mind-numbing.

Answer 5

According to this post:

There are two possible causes of the java.lang.OutOfMemoryError: Failed to create a thread message:

• There are too many threads running and the system has run out of internal resources to create new threads.
• The system has run out of native memory to use for the new thread. Threads require a native memory for internal JVM structures, a Java™ stack, and a native stack.

So this error may well be completely unrelated to memory, just that too many threads are created...

EDIT:

As you've got 695 threads, you would need 695 times the stack size as memory. Considering this post on thread limits, I suspect that you are trying to create too many threads for the available virtual memory space.