We have been having OutOfMemory in one of our Web Applications. This application is running on Tomcat on Aamzon EC2 behind an Elastic Load Balancer. From heap dump, most the memory was consumed by SocksSocketImpl objects. Yourkit shows 99% of these objects are in Pending Finalization state. However, somehow they are NOT being finalized and GC'ed. The same application with same setup (same tomcat version and JDK version) on other regions and doesn't have issues.
Has anyone seen this kind of behavior before? Any pointers will be helpful. Thanks for your time.
I had the exact same problem running Tomcat 6.0.39, memory issue started when JDK on Ubuntu server was upgraded with security patches to:
java version "1.6.0_30"
OpenJDK Runtime Environment (IcedTea6 1.13.1) (6b30-1.13.1-1ubuntu2~0.12.04.1)
OpenJDK Server VM (build 23.25-b01, mixed mode)
Profiled with Eclipse Memory Analyzer, traced to finalizer queue of objects for SSL Connections and Hibernate sessions consuming all memory. Problem resolved by upgrading to:
java version "1.7.0_51"
OpenJDK Runtime Environment (IcedTea 2.4.4) (7u51-2.4.4-0ubuntu0.12.04.2)
OpenJDK Server VM (build 24.45-b08, mixed mode)
The issue got resolved after upgrading to JVM 1.7.0_51. As suspected, it seems to be a JVM level issue either during socket clean up or GC. The problematic version was 1.6.0_30 OpenJDK.
Its caused by regression to the openjdk package:
https://bugs.launchpad.net/ubuntu/+source/openjdk-6/+bug/1295987
It has been fixed, but is waiting for a new openjdk6 package to be deployed.