We are building a web application with aggressive performance SLAs which are periodically being violated due to the JVM going out to lunch due to System.gc() calls. We've done some debugging, and determined that in all cases it's the internal app server code which is invoking System.gc(). This occurs a few times while the app server is booting or the application is deployed, which we aren't concerned about. However, System.gc() is also periodically be triggered when the app is up and running via internal app server calls the NIO classes. Here's a stack trace we were able to capture of this event:
3XMTHREADINFO "WebContainer : 25" J9VMThread:0x0000000006FC5D00, j9thread_t:0x00007F60E41753E0, java/lang/Thread:0x000000060B735590, state:R, prio=5
3XMJAVALTHREAD (java/lang/Thread getId:0xFE, isDaemon:true)
3XMTHREADINFO1 (native thread ID:0x1039, native priority:0x5, native policy:UNKNOWN)
3XMTHREADINFO2 (native stack address range from:0x00007F6067621000, to:0x00007F6067662000, size:0x41000)
3XMCPUTIME CPU usage total: 80.222215853 secs
3XMHEAPALLOC Heap bytes allocated since last GC cycle=1594568 (0x1854C8)
3XMTHREADINFO3 Java callstack:
4XESTACKTRACE at java/lang/System.gc(System.java:329)
4XESTACKTRACE at java/nio/Bits.syncReserveMemory(Bits.java:721)
5XESTACKTRACE (entered lock: java/nio/Bits@0x000000060000B690, entry count: 1)
4XESTACKTRACE at java/nio/Bits.reserveMemory(Bits.java:766(Compiled Code))
4XESTACKTRACE at java/nio/DirectByteBuffer.<init>(DirectByteBuffer.java:123(Compiled Code))
4XESTACKTRACE at java/nio/ByteBuffer.allocateDirect(ByteBuffer.java:306(Compiled Code))
4XESTACKTRACE at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateBufferDirect(WsByteBufferPoolManagerImpl.java:706(Compiled Code))
4XESTACKTRACE at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateCommon(WsByteBufferPoolManagerImpl.java:612(Compiled Code))
4XESTACKTRACE at com/ibm/ws/buffermgmt/impl/WsByteBufferPoolManagerImpl.allocateDirect(WsByteBufferPoolManagerImpl.java:527(Compiled Code))
4XESTACKTRACE at com/ibm/io/async/ResultHandler.runEventProcessingLoop(ResultHandler.java:507(Compiled Code))
4XESTACKTRACE at com/ibm/io/async/ResultHandler$2.run(ResultHandler.java:905(Compiled Code))
4XESTACKTRACE at com/ibm/ws/util/ThreadPool$Worker.run(ThreadPool.java:1864(Compiled Code))
3XMTHREADINFO3 Native callstack:
4XENATIVESTACK (0x00007F61083DD122 [libj9prt26.so+0x13122])
4XENATIVESTACK (0x00007F61083EA79F [libj9prt26.so+0x2079f])
....
Is anyone aware of what the impact would be if we shut down the calls to System.gc() by enabling -XX:+DisableExplicitGC (or actually in our case by setting -Xdisableexplicitgc since we're running Websphere on the IBM JRE, which does the same thing)? We certainly don't want to create a memory leak. I haven't been able to find a direct reference as to why the System.gc() calls in NIO are actually necessary, and there isn't a code comment specifically addressing it where it occurs in JDK code, either: http://hg.openjdk.java.net/jdk8u/jdk8u-dev/jdk/file/4a1e42601d61/src/share/classes/java/nio/Bits.java
If it's a bad idea to completely disable System.gc() due to the use of NIO, is there at least something we can do to reduce the frequency at which it's called? It appears that we can set -XX:MaxDirectMemorySize, but this appears as if it would only set up upper bound on the amount of allocated memory, and would just as likely have an adverse affect.