java.lang.OutOfMemoryError: requested 1958536 byte

Question

We are facing the below problem at our production enviournment in unpredictable manner sometimes the server is down in a day or sometimes in a week, below is the exact error dump, below are the settings for the server.

JDK: jdk1.6.0_21
Server: Tomcat 7.0.2
OS: Red Hat Enterprise Linux Server release 5.5

In catalina.sh the following setting has been done:

JAVA_OPTS="-Xms1024M -Xmx1536M -XX:+HeapDumpOnOutOfMemoryError -XX:+AggressiveOpts 
-XX:-DisableExplicitGC  -XX:AdaptiveSizeThroughPutPolicy=0  
-XX:+UsePSAdaptiveSurvivorSizePolicy 
-XX:+UseAdaptiveGenerationSizePolicyAtMinorCollection  
-XX:+UseAdaptiveGenerationSizePolicyAtMajorCollection -XX:PermSize=768M 
-XX:MaxPermSize=768M    -XX:+PrintGCDetails -Xloggc:/tmp/gcLogs.txt"

export CATALINA_OPTS="-Dcom.sun.management.jmxremote.port=22222 
-Dcom.sun.management.jmxremote.authenticate=true -Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.password.file=/jakarta-tomcat7/apache-tomcat-7.0.2/conf
/jmxremote.password -Dcom.sun.management.jmxremote.access.file=/jakarta-tomcat7/apache-
tomcat-7.0.2/conf/jmxremote.access"

Error Trace:-

#
# A fatal error has been detected by the Java Runtime Environment:
#
# java.lang.OutOfMemoryError: requested 1958536 bytes for Chunk::new. Out of swap space?
#
#  Internal Error (allocation.cpp:215), pid=18658, tid=589781904
#  Error: Chunk::new
#
# JRE version: 6.0_21-b06
# Java VM: Java HotSpot(TM) Server VM (17.0-b16 mixed mode linux-x86 )
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x23787400):  JavaThread "CompilerThread0" daemon [_thread_in_native, id=18668, stack(0x231f5000,0x23276000)]

Stack: [0x231f5000,0x23276000],  sp=0x23272e70,  free space=1f723276000k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x6a9262]
V  [libjvm.so+0x2b277f]
V  [libjvm.so+0x12e03c]
V  [libjvm.so+0x12e536]
V  [libjvm.so+0x5d67d0]
V  [libjvm.so+0x2f809d]
V  [libjvm.so+0x4f65a9]
V  [libjvm.so+0x27b85f]
V  [libjvm.so+0x278043]
V  [libjvm.so+0x209767]
V  [libjvm.so+0x280f8c]
V  [libjvm.so+0x280839]
V  [libjvm.so+0x66feb6]
V  [libjvm.so+0x66959e]
V  [libjvm.so+0x57a89e]
C  [libpthread.so.0+0x5832]


Current CompileTask:
C2:3230  !   org.apache.jsp.com.common.press_jsp._jspService(Ljavax/servlet/http/HttpServletRequest;Ljavax/servlet/http/HttpServletResponse;)V (4433 bytes)


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x09a21400 JavaThread "http-8080-exec-904" daemon [_thread_in_native, id=17126, stack(SIGTERM: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004
SIGQUIT: [libjvm.so+0x57aaf0], sa_mask[0]=0x7ffbfeff, sa_flags=0x10000004


---------------  S Y S T E M  ---------------

OS:Red Hat Enterprise Linux Server release 5.5 (Tikanga)

uname:Linux 2.6.18-194.17.1.el5PAE #1 SMP Mon Sep 20 07:34:07 EDT 2010 i686
libc:glibc 2.5 NPTL 2.5 
rlimit: STACK 10240k, CORE 0k, NPROC 114688, NOFILE 1024, AS infinity
load average:0.39 0.54 0.38

CPU:total 2 (2 cores per cpu, 1 threads per core) family 6 model 15 stepping 11, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3

Memory: 4k page, physical 6228576k(225096k free), swap 6974456k(6974352k free)

vm_info: Java HotSpot(TM) Server VM (17.0-b16) for linux-x86 JRE (1.6.0_21-b06), built on Jun 22 2010 01:04:46 by "java_re" with gcc 3.2.1-7a (J2SE release)

time: Fri Dec 10 14:01:06 2010
elapsed time: 79552 seconds

Thanks in advance, Amit

Answer 1

For the record (and Google), this looks like both these bugs, which were fixed in the 1.6u22 release of sun's jdk. So, first thing to do is update your JVM. If it still happens, and always happens on a particular method, you can exclude that method from compilation (as long as you're aware of the performance implications of that) with the following jvm flag:

-XX:CompileCommand=exclude,org/apache/velocity/runtime/directive/Foreach,render

(as suggested here). But, first update your jvm.

Answer 2

You're running with a lot of JVM args that affect memory. Have you tried empirically removing each option to see which one is causing the OOM? This particular OOM is not coming from the Java heap, it's coming from the JVM's own C heap.

Answer 3

As stated by the other answers / comments, you are running out of memory. Given your JVM settings, I'd say that the root cause is 99% likely to be a memory leak.

If you have been doing a lot of hot loading in the Tomcat instance, this could just be caused by that. Hot loading is notorious for leaking memory, and there is not much you can do about it in practice ... except exit and restart your Tomcat more often.

The other possibility is that your application is leaking memory. If this is the case then you will need to use a memory profiler to track down the leak.

The fact that the OOME caused a JVM crash is interesting, but probably not significant. (It looks like the JVM was trying to JIT compile a class generated from a JSP when it ran out of memory. The chunk being requested is rather large, but that probably means you have a rather large / complicated JSP.)