How does a JVM process allocate its memory?

I have a little gap in understanding how a JVM process allocates its own memory. As far as I know

RSS = Heap size + MetaSpace + OffHeap size

where OffHeap consists of thread stacks, direct buffers, mapped files (libraries and jars) and JVM code itself;

At the moment I’m trying to analyze my Java application (Spring Boot + Infinispan) which RSS is 779M (it runs in a docker container, so pid 1 is ok):

[ root@daf5a5ae9bb7:/data ]$ ps -o rss,vsz,sz 1
RSS    VSZ    SZ
798324 6242160 1560540

According to jvisualvm, committed Heap size is 374M

Metasapce size is 89M

In other words, I want to explain 799M - (374M + 89M) = 316M of OffHeap memory.

My app has (in average) 36 live threads.

Each of these threads consumes 1M:

[ root@fac6d0dfbbb4:/data ]$ java -XX:+PrintFlagsFinal -version |grep ThreadStackSize    
intx CompilerThreadStackSize                   = 0
intx ThreadStackSize                           = 1024
intx VMThreadStackSize                         = 1024

So, here we can add 36M.

The only place where the app uses DirectBuffer is NIO. As far as I can see from JMX, it doesn’t consume a lot of resources - only 98K

The last step is mapped libs and jars. But according to pmap (full output)

[ root@daf5a5ae9bb7:/data ]$ pmap -x 1 | grep ".so.*" | awk '{ sum+=$3} END {print sum}'

12896K

plus

root@daf5a5ae9bb7:/data ]$ pmap -x 1 | grep “.jar" | awk '{ sum+=$3} END {print sum}'

9720K

we only have 20M here.

Hence, we still have to explain 316M - (36M + 20M) = 260M :(

Does anyone have any idea what I missed?

Approach:

You may want to use Java HotSpot Native Memory Tracking (NMT).

This may give you an exact list of memory allocated by the JVM, splitted up into the different areas heap, classes, threads, code, GC, compiler, internal, symbols, memory tracking, pooled free chunks, and unknown.

Usage:

You can start your application with -XX:NativeMemoryTracking=summary.
Observations of the current heap can be done with jcmd <pid> VM.native_memory summary.

Where to find jcmd / pid:

On a default OpedJDK installation on Ubuntu this can be found at /usr/bin/jcmd.

By just running jcmd without any parameter, you get a list of running Java applications.

user@pc:~$ /usr/bin/jcmd
5169 Main                       <-- 5169 is the pid

Output:

You will then receive a complete overview over your heap, looking something like the following:

Total: reserved=664192KB, committed=253120KB <--- total memory tracked by Native Memory Tracking

Java Heap (reserved=516096KB, committed=204800KB) <--- Java Heap

(mmap: reserved=516096KB, committed=204800KB)

Class (reserved=6568KB, committed=4140KB) <--- class metadata

(classes #665) <--- number of loaded classes

(malloc=424KB, #1000) <--- malloc'd memory, #number of malloc

(mmap: reserved=6144KB, committed=3716KB)

Thread (reserved=6868KB, committed=6868KB) (thread #15) <--- number of threads

(stack: reserved=6780KB, committed=6780KB) <--- memory used by thread stacks

(malloc=27KB, #66)

(arena=61KB, #30) <--- resource and handle areas

Code (reserved=102414KB, committed=6314KB)

(malloc=2574KB, #74316)

(mmap: reserved=99840KB, committed=3740KB)

GC (reserved=26154KB, committed=24938KB)

(malloc=486KB, #110)

(mmap: reserved=25668KB, committed=24452KB)

Compiler (reserved=106KB, committed=106KB)

(malloc=7KB, #90)

(arena=99KB, #3)

Internal (reserved=586KB, committed=554KB)

(malloc=554KB, #1677)

(mmap: reserved=32KB, committed=0KB)

Symbol (reserved=906KB, committed=906KB)

(malloc=514KB, #2736)

(arena=392KB, #1)

Memory Tracking (reserved=3184KB, committed=3184KB)

(malloc=3184KB, #300)

Pooled Free Chunks (reserved=1276KB, committed=1276KB)

(malloc=1276KB)

Unknown (reserved=33KB, committed=33KB)

(arena=33KB, #1)

This gives a detailed overview of the different memory areas used by the JVM, and also shows the reserved and commited memory.

I don't know of a technique that gives you a more detailed memory consumption list.

Further reading:

You can also use -XX:NativeMemoryTracking=detail in combination with further jcmd commands. A more detailed explaination can be found at Java Platform, Standard Edition Troubleshooting Guide - 2.6 The jcmd Utility. You can check possible commands via "jcmd <pid> help"