Java 8 with Jetty on linux memory issue

Can you please help me resolving the following issue:

Context:

We are trying to migrate our existing application which is currently running on Java6( on Glassfish) in production to Java8 (on Jetty9) setup. Earlier, We were able to successfully migrate the same setup on Java7(on jetty9). But the customer has decided to go with Java 8 now. In this process we have encountered some memory issues and below are the details:

Problem Description:

After starting the Jetty server, initial (RES)memory usage of java process is around 5.5g. After running the application for sometime the memory usage slowly shoots-up and consumes maximum available physical memory(8g) on the machine, eventually crashing the server/system.

This issue is encountered only on the linux environment. No such issues are found on Windows environment.

Profiler findings:

Monitored the server with VisualVM and jconsole. In both the profilers JVM's memory(heap & non-heap) usage is under the allocated limits.

Environment details:

Java Version :    8
Server       :     Jetty 9.2.10
OS           :    linux on a virtual machine(Linux version 2.6.32-279.14.1.el6.x86_64 (mockbuild@c6b8.bsys.dev.centos.org) (gcc version 4.4.6 20120305 (Red Hat 4.4.6-4) (GCC) ) #1 SMP Tue Nov 6 23:43:09 UTC 2012
Java Options :        
        -Xms3072M

        -server

        -XX:+UnlockDiagnosticVMOptions

        -XX:+LogVMOutput

        -XX:+UseG1GC

        -XX:MaxGCPauseMillis=75

        -Xmx3072M

        -Xss1024K

        -XX:InitialCodeCacheSize=192M

        -XX:CodeCacheExpansionSize=3M

        -XX:CodeCacheMinimumFreeSpace=3M

        -XX:ReservedCodeCacheSize=600M

        -XX:MinMetaspaceExpansion=3M

        -XX:MaxMetaspaceExpansion=18M

        -XX:MaxMetaspaceSize=500M

        -XX:MaxDirectMemorySize=288M

        -XX:CompressedClassSpaceSize=512M

        -XX:ParallelGCThreads=12

        -XX:ConcGCThreads=4

        -Dsun.rmi.dgc.server.gcInterval=86400000

        -Dsun.rmi.dgc.client.gcInterval=86400000

PS: Please don't mark this as duplicate. I have read many answers on stackoverflow, but nothing addressed or solved my issue.

Update

I started jetty with the following Java Options, and since then the memory usage is between 4.5g-4.8g (approximately 142 hours). It looks stable to me. I reserved around 2g of memory through java option flags(Xmx & MetaspaceSize), but additional 2.5g is used always. Is this normal behavior for Java 8 on a linux machine?

Java Options used:

-server
-XX:+UnlockDiagnosticVMOptions
-XX:+LogVMOutput
-XX:LogFile=../logs/jvm.log

-XX:+UseG1GC
-XX:MaxGCPauseMillis=75
-XX:ParallelGCThreads=12
-XX:ConcGCThreads=12
-XX:+PrintGCDetails
-XX:+PrintGCTimeStamps
-Xloggc:../logs/gc.log
-XX:NativeMemoryTracking=summary

-Xmx1500m
-Xss256k

-XX:MaxMetaspaceSize=512m

-XX:+HeapDumpOnOutOfMemoryError
-XX:HeapDumpPath=/opt/logs/jetty.hprof

-Dsun.rmi.dgc.server.gcInterval=86400000
-Dsun.rmi.dgc.client.gcInterval=86400000

Thanks for your time!

It seems to be an issue in Java8/9 that manifests itself in Jetty due to the annotations module that scans the jars and has a memory leak. Ref. https://github.com/eclipse/jetty.project/issues/575. A solution for me (because I do not use Jetty annotations) is to disable the annotations module by commenting out the lines in jetty/modules/annotations.mod. So tie file looks like this:

#
# Jetty Annotation Scanning Module
#

[depend]
# Annotations needs plus, and jndi features
plus

[lib]
# Annotations needs jetty annotation jars
lib/jetty-annotations-${jetty.version}.jar
# Need annotation processing jars too
#lib/annotations/*.jar

[xml]
# Enable annotation scanning webapp configurations
#etc/jetty-annotations.xml

Edit 1 - Alternative solution

Switching off all annotation scanning can be too drastic, it turns of jsp as well because it is dependent. The alternative is to provide a web application context which restricts the scanning using a pattern. Save this in an xml and deploy it in the webapps together with the war or include it in the war.

<Configure class="org.eclipse.jetty.webapp.WebAppContext">
   <Set name="contextPath">/[myApp]</Set>
   <Set name="war">/[DIRECTORY]/[myApp[.war</Set>
   <Call name="setAttribute">
      <Arg>org.eclipse.jetty.server.webapp.WebInfIncludeJarPattern</Arg>
      <Arg>SCAN-NO-JARS</Arg>
    </Call>
</Configure>