Docker Container - JVM Memory Spike - Arena Chunk

Question

I'm observing a large, discrete spike in JVM memory during performance tests against my Java web application running in an ECS/EC2/Docker/Centos7/Tomcat/OpenJDK8 environment.

The performance test is quite simple and it consists of continuous concurrent requests to an AWS Application Load Balancer sitting in front of a pair of Docker containers running on EC2 hosts managed by Elastic Container Service. Typically the concurrency level is 30 simultaneous load test client connections/threads. Within a few minutes, one of the Docker containers is usually afflicted.

The memory spike appears to be in non-heap memory. Specifically, the memory spike seems to be related to the Arena Chunk memory space. When comparing the memory footprint of a JVM that hasn't experienced the spike with one that has, the Thread and Arena Chunk memory spaces stand out.

Below is a comparison of the VM internal memory using the jcmd utility.

Notice the absurdly high number for Arena Chunk memory and the comparatively high number for Thread memory.

The concurrency level of the test can create an immediate demand for threads in the Tomcat request thread pool. However, the spike doesn't always occur in the first wave of requests.

Have you seen something similar? Do you know what is causing the spike?

---

Docker Stats

Memory Spike Container:

Mon Oct  9 00:31:45 UTC 2017
89440337e936        27.36%              530 MiB / 2.93 GiB      17.67%              15.6 MB / 24.1 MB   122 MB / 2.13 MB    0
Mon Oct  9 00:31:48 UTC 2017
89440337e936        114.13%             2.059 GiB / 2.93 GiB    70.29%              16.3 MB / 25.1 MB   122 MB / 2.13 MB    0

Normal Container:

Mon Oct  9 00:53:41 UTC 2017
725c23df2562        0.08%               533.4 MiB / 2.93 GiB   17.78%              5 MB / 8.15 MB      122 MB / 29.3 MB    0
Mon Oct  9 00:53:44 UTC 2017
725c23df2562        0.07%               533.4 MiB / 2.93 GiB   17.78%              5 MB / 8.15 MB      122 MB / 29.3 MB    0

---

VM Internal Memory

Memory Spike JVM:

# jcmd 393 VM.native_memory summary
393:

Native Memory Tracking:

Total: reserved=1974870KB, committed=713022KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB) 

-                     Class (reserved=1096982KB, committed=53466KB)
                            (classes #8938)
                            (malloc=1302KB #14768) 
                            (mmap: reserved=1095680KB, committed=52164KB) 

-                    Thread (reserved=8423906KB, committed=8423906KB)
                            (thread #35)
                            (stack: reserved=34952KB, committed=34952KB)
                            (malloc=114KB #175) 
                            (arena=8388840KB #68)

-                      Code (reserved=255923KB, committed=37591KB)
                            (malloc=6323KB #8486) 
                            (mmap: reserved=249600KB, committed=31268KB) 

-                        GC (reserved=6321KB, committed=6321KB)
                            (malloc=4601KB #311) 
                            (mmap: reserved=1720KB, committed=1720KB) 

-                  Compiler (reserved=223KB, committed=223KB)
                            (malloc=93KB #276) 
                            (arena=131KB #3)

-                  Internal (reserved=2178KB, committed=2178KB)
                            (malloc=2146KB #11517) 
                            (mmap: reserved=32KB, committed=32KB) 

-                    Symbol (reserved=13183KB, committed=13183KB)
                            (malloc=9244KB #85774) 
                            (arena=3940KB #1)

-    Native Memory Tracking (reserved=1908KB, committed=1908KB)
                            (malloc=8KB #95) 
                            (tracking overhead=1900KB)

-               Arena Chunk (reserved=18014398501093554KB, committed=18014398501093554KB)
                            (malloc=18014398501093554KB) 

-                   Unknown (reserved=38388KB, committed=38388KB)
                            (mmap: reserved=38388KB, committed=38388KB) 

Normal JVM:

# jcmd 391 VM.native_memory summary
391:

Native Memory Tracking:

Total: reserved=1974001KB, committed=710797KB
-                 Java Heap (reserved=524288KB, committed=524288KB)
                            (mmap: reserved=524288KB, committed=524288KB) 

-                     Class (reserved=1096918KB, committed=53738KB)
                            (classes #9005)
                            (malloc=1238KB #13654) 
                            (mmap: reserved=1095680KB, committed=52500KB) 

-                    Thread (reserved=35234KB, committed=35234KB)
                            (thread #35)
                            (stack: reserved=34952KB, committed=34952KB)
                            (malloc=114KB #175) 
                            (arena=168KB #68)

-                      Code (reserved=255261KB, committed=35237KB)
                            (malloc=5661KB #8190) 
                            (mmap: reserved=249600KB, committed=29576KB) 

-                        GC (reserved=6321KB, committed=6321KB)
                            (malloc=4601KB #319) 
                            (mmap: reserved=1720KB, committed=1720KB) 

-                  Compiler (reserved=226KB, committed=226KB)
                            (malloc=96KB #317) 
                            (arena=131KB #3)

-                  Internal (reserved=2136KB, committed=2136KB)
                            (malloc=2104KB #11715) 
                            (mmap: reserved=32KB, committed=32KB) 

-                    Symbol (reserved=13160KB, committed=13160KB)
                            (malloc=9221KB #85798) 
                            (arena=3940KB #1)

-    Native Memory Tracking (reserved=1890KB, committed=1890KB)
                            (malloc=8KB #95) 
                            (tracking overhead=1882KB)

-               Arena Chunk (reserved=178KB, committed=178KB)
                            (malloc=178KB) 

-                   Unknown (reserved=38388KB, committed=38388KB)
                            (mmap: reserved=38388KB, committed=38388KB) 

Answer 1

A glibc/malloc option seems to fix this MALLOC_PER_THREAD=0. However, I decided to use the debian/openjdk docker base image rather than centos and that also fixed the issue.