Finding Java memory leak when not all used heap is

Question

I am looking into a potential memory leak (or at least memory waste) in a largish Java based system. The JVM is running with a maximum heap size of 5 GB and 2-3GB heap usage is an expected base line for the application. (There can be peaks that are higher)

In an overload scenario which I am investigating the heap gets filled up. Analyzing the a heap-dump with the "Eclipse MemoryAnalyzer Tool" shows (no surprise) that the heap is entirely used up.

MAT shows 2 potential leak candidates, both roughly retaining 2.5GB: java.lang.Thread and a domain object from the system which is used extensively during transaction processing in the system. All these domain objects are however (no surprise) reachable from the Thread instances. Those threads are processing the transactions, after all. Thus, the 2.5 GB attributed to java.lang.Thread is almost entirely caused by those domain objects. No surprise here.

Listing the object tree of all java.lang.Thread instances and summing up the retained heap of all threads results in 2.5 GB of retained heap.

Where should I look for the other 2.5 GB that are needed to fill up the heap, if they are not reachable from an instance of java.lang.Thread? - There is nothing in the finalizer queue - There is not a significant amount of unreachable objects pending GC

I think another way to put this question is: "How do I find all objects that are not reachable from an instance of java.lang.Thread? Maybe an OQL query?, and the other question: "What kind of Objects are there that are not reachable from an instance of java.lang.Thread other then Objects in the Finalizer Queue and unreferenced objects pending GC?"

Answer 1

Since the extra memory is not showing in MAT it's hard to know what to suggest. My apologies if some (or even most) of this is things you already know, I've just tried to pull together everything I could think of.

FindBugs

FindBugs is a static analysis tool that will scan your code looking for common anti-patterns and problems and giving you a nice report on them. It does pick up on a lot of causes of potential memory and resource leaks.

Manual dump

You could try using something like jmap or visualvm to take a heap dump for analysis manually and see if you get different results from letting eclipse do it:

http://docs.oracle.com/javase/1.5.0/docs/tooldocs/share/jmap.html

http://java.dzone.com/articles/java-heap-dump-are-you-task

Analyzer Quirks

The memory analyzer FAQ:

http://wiki.eclipse.org/MemoryAnalyzer/FAQ

says:

Symptom: When monitoring the memory usage interactively, the used heap size is much bigger than what MAT reports.

During the index creation, the Memory Analyzer removes unreachable objects because the various garbage collector algorithms tend to leave some garbage behind (if the object is too small, moving and re-assigning addresses is to expensive). This should, however, be no more than 3 to 4 percent. If you want to know what objects are removed, enable debug output as explained here: MemoryAnalyzer/FAQ#Enable_Debug_Output

Another reason could be that the heap dump was not written properly. Especially older VM (1.4, 1.5) can have problems if the heap dump is written via jmap.

Enabling debug output will allow you to see what is going on there and confirm there is nothing odd in that area.

Some of these tips may be relevant

http://eclipsesource.com/blogs/2013/01/21/10-tips-for-using-the-eclipse-memory-analyzer/

Answer 2

I too faced the problem with memory leaks at our site,
Use yourkit java profiler which provide lots of information and with its ability you can have a wider image where all the memory is being utilized.
You can find a great tutorial Find Java Memory Leaks with the above tool.

Your question,

"What kind of Objects are there that are not reachable from an instance of java.lang.Thread other then Objects in the Finalizer Queue and unreferenced objects pending GC?"

There are four kinds of object,

1. Strong reachable, objects that can be reached directly via references from live objects
2. Weak/Soft reachable, objects that are having weak/Soft reference associated with them
3. Pending Finalization, objects that are pending for finalization and whose reference can be reached through finalizer queue
4. Unreachable these are objects that are unreachable from GC roots, but not yet collected

Besides these JVM also uses Native memory whose information you can find on IBM Heap and native memory use by the JVM and Thanks for the memory and according to YourKit the JVM Memory Structure has Non-Heap Memory whose definition according to them is

Also, the JVM has memory other than the heap, referred to as non-heap memory. It is created at the JVM startup and stores per-class structures such as runtime constant pool, field and method data, and the code for methods and constructors, as well as interned Strings.

Answer 3

Maybe you should look for memory leaks in database connector code or maybe ORM. Because if you are using raw connection library when you don't close cursor you can get potentially memory leak. Also my second thought is also related to database connector. Because some of them (may be not yours) uses native code beneath and this is source of this leak. Due to heavy concurrent usage that makes sens for me. You can check that if you want.

Answer 4

Since the extra memory is not showing in MAT it's hard to know what to suggest. It isn't true. MAT show unreachable objects. Just go to de Preferences and select check box enabling this options. After MAT restart you will see these objects with details. Of course roots to GC will be not available.

Answer 5

Use JProfiler and break the heap object count down by class - find which class has lots of instances and start your hunt there.

You can also take a couple of snapshots a short time apart and compare the two heap dumps to see what objects were created during that time. This is particularly handy if you know that a certain action is causing the problem and you want to ignore all the background JVM object noise and just examine the delta.

I have used it with great success to find a memory leak. It isn't free, but it's worth the licence fee.

FYI: I have no affiliation with JProfiler.