tomcat performance monitoring in JavaMelody and ab

Question

I am new to JavaMelody and benchmark. I am using JavaMelody to monitor my Jersey Application in tomcat following https://dzone.com/articles/a-step-by-step-guide-to-tomcat-performance-monitor.

My application setup: 4 docker containers setup by docker-compose: myapplication (tomcat, jersey application), kafka, postgres, zookeeper.

I got the following details from javaMelody monitoring page:

Host:   16a180e86bba@172.18.0.5
Java memory used:   784 Mb / 7,124 Mb       ++++++++++++
Nb of http sessions:    0   
Nb of active threads
(current http requests):    9   
Nb of active jdbc connections:  0   
Nb of used jdbc connections
(opened if no datasource):  0
System load 27.89   
% System CPU    93.36       ++++++++++++

##############
# Details
##############

OS:     OS Linux, 4.11.0-32-generic , amd64/64 (8 cores)
Java:   OpenJDK Runtime Environment, 1.8.0
JVM:    OpenJDK 64-Bit Server VM
PID of process:     1
Nb of opened files  418 / 1,048,576    ++++++++++++
Server:     Server Apache Tomcat/8.0.50
Webapp context:     /event-bus
Start:  2/28/18 2:47 PM
JVM arguments:  -Djava.util.logging.config.file=/usr/local/tomcat/conf/logging.properties
-Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager
-Djdk.tls.ephemeralDHKeySize=2048
-Djava.protocol.handler.pkgs=org.apache.catalina.webresources
-Xdebug
-Xrunjdwp:transport=dt_socket,server=y,suspend=n,address=9999
-Dignore.endorsed.dirs=
-Dcatalina.base=/usr/local/tomcat
-Dcatalina.home=/usr/local/tomcat
-Djava.io.tmpdir=/usr/local/tomcat/temp
Mean age of http sessions (min):    -1
Tomcat "http-apr-8080":     Busy threads = 17 / 200     ++++++++++++
Bytes received = 65,071,259
Bytes sent = 4,411,314
Request count = 538,335
Error count = 19,253
Sum of processing times (ms) = 17,998,558
Max processing time (ms) = 3,161
Memory:     Non heap memory = 99 Mb (Perm Gen, Code Cache),
Buffered memory = 33 Mb,
Loaded classes = 9,535,
Garbage collection time = 1,420 ms,
Process cpu time = 854,410 ms,
Committed virtual memory = 14,718 Mb,
Free physical memory = 21,865 Mb,
Total physical memory = 32,052 Mb,
Free swap space = 32,651 Mb,
Total swap space = 32,651 Mb
Free disk space:    408,262 Mb
Dependencies:   Dependencies Dependencies     View Maven's pom View Maven's pom
Threads Threads
Threads on 16a180e86bba@172.18.0.5: Number = 624, Maximum = 681, Total started = 1,182

I have some questions:

1, I setup 4 docker containers and tomcat is running in one of them. Does JavaMelody give out accurate result on CPU and memory usage of my application or just docker containers?

2, There are 8 cores in my machine, does % System CPU 93.36 mean my application used 93.36% of all 8 cores CPU resource? I used docker stats <docker id> and got CPU % of my application is about 336%. I am confused.

3, what does system load mean?

4, what does 7124 Mb mean in Java memory used: 784 Mb / 7,124 Mb ++++++++++++?

5, In Tomcat "http-apr-8080": Busy threads = 17 / 200 ++++++++++++ and Threads on 16a180e86bba@172.18.0.5: Number = 624, Maximum = 681, Total started = 1,182.

Does 17/200 mean the maximum number of threads to accept connections is 200, but only 17 are used. but what does Number = 624, Maximum = 681, Total started = 1,182 mean?

6, I used ab to send 20000 requests in 200 concurrency. But why only 17/200 used? How to make 200/200?

7, from report of ab, I found 1.3% failed requests. Does this mean I setup too many concurrency in ab test? Or does this mean tomcat cannot handle all requestion (but only 17/200 used in tomcat)?

Thanks.

Answer 1

1. Both: for the application and for the operating system
2. System CPU is the %CPU for the OS. And 93% is 93% of all cores for the last minute (100% is the maximum). 336% is probably for the last seconds only.
3. System load means the load average of the OS.
4. It means the maximum size of the heap (like in Xmx).
5. Yes, it means that the max of http threads is 200, but only 17 are used. But 17 used http threads is quite high already for an application with 8 cores, generally speaking. Theads on ...: Number = 624, Maximum = 681, Total started = 1,182 means 624 threads are live, 681 were alive at the maximum some time ago, 1182 threads were started since starting Tomcat
6. I doubt that you can have 200 used threads concurrently with 8 cores. There is probably not enough cores for that. Generally speaking, to have 200 threads active and not waiting for the database or something else, you would need theoretically 200 cores. In your case, either your server is fast enough to send responses faster than ab can throw at it or there is something else between ab and the threads which slows threads allocations (I think that cpu at 93% may be the cause of that).
7. CPU seems saturated and 200 ab threads is probably too much so it is normal to have some failures.