How can a Java process with -Xmx1024m occupy 3GB r

Question

It's a Java web application on Websphere6.1, Solaris 10, JDK 1.5.0_13. We set the maximum heap size to 1024m. jmap shows the heap status is healthy. The heap memory usage is only 57%. No OutOfMemory at all.

But we saw very high RSS (3GB) for this java process from ps. pmap shows a block of 1.9G private memory.

3785:   /dmwdkpmmkg/was/610/java/bin/java -server -Dwas.status.socket=65370 -X
 Address  Kbytes     RSS    Anon  Locked Pgsz Mode   Mapped File
...
0020A000    2008    2008    2008       -   8K rwx--    [ heap ]
00400000 1957888 1957888 1957888       -   4M rwx--    [ heap ]
8D076000      40      40      40       -   8K rw--R    [ stack tid=10786 ]
...

Is it a C heap memory leak in native code? What approach is recommended to find out the root cause?

Answer 1

Are you using JNI libraries? I'm not sure how native code allocates RAM but that's where I'd start looking.

Answer 2

This might happen even when there isn't a native memory leak (like unclosed zip resources).

I ran in to the same problem. This is a known problem with glibc >= 2.10

The cure is to set this env variable export MALLOC_ARENA_MAX=4

IBM article about setting MALLOC_ARENA_MAX https://www.ibm.com/developerworks/community/blogs/kevgrig/entry/linux_glibc_2_10_rhel_6_malloc_may_show_excessive_virtual_memory_usage?lang=en

Google for MALLOC_ARENA_MAX or search for it on SO to find a lot of references.

You might want to tune also other malloc options to optimize for low fragmentation of allocated memory:

# tune glibc memory allocation, optimize for low fragmentation
# limit the number of arenas
# requires glibc >= 2.16 since there was a bug in 
# MALLOC_ARENA_TEST parameter handling that cause MALLOC_ARENA_MAX not to work
export MALLOC_ARENA_MAX=2
# disable dynamic mmap threshold, see M_MMAP_THRESHOLD in "man mallopt"
export MALLOC_MMAP_THRESHOLD_=131072
export MALLOC_TRIM_THRESHOLD_=131072
export MALLOC_TOP_PAD_=131072
export MALLOC_MMAP_MAX_=65536

You can call the native malloc_info function to get information about memory allocations. Here is an example of using JNA to call the native malloc_info method.

Answer 3

The heap size os the Java heap size, there is still the VM and other libraries which are part of the process.

Try running Hello World with a 1024m heap size and a "for(;;)" in it and see how much it takes up. That should give you a baseline for overall memory usage.

Answer 4

This Troubleshooting Memory Leaks document from Sun may help you with finding the problem why your high RSS, specially in section 3.4.

As you are running Websphere, maybe you can use the -memorycheck on your VM. For details see here.

It's not necessarily a leak in native code. If you look here, on Solaris there might be an issue with files being kept open.

It's just a bunch of links and hints, but maybe helpful to track down your problem.