JVM Crash Problematic Frame: Canonicalizer::do_If

问题:

Iam facing JVM Crash cosistently while enabling hotdeploy (USING below java options on starting up JAVA_OPTS -Xmx4096m -XX:MetaspaceSize=512m -XX:+HeapDumpOnOutOfMemoryError -XX:HeapDumpPath=crash -XX:ThreadStackSize=512 -XX:+UseConcMarkSweepGC -XX:ParallelGCThreads=5 -XX:NewRatio=2 -XX:+UnlockDiagnosticVMOptions -XX:-UseLoopPredicate -Xdebug -Xrunjdwp:transport=dt_socket,address=$DEBUG_PORT,server=y,suspend=n -XX:NewRatio=2 -Dspringloaded.synchronize=true JAVA_OPTS=`echo $JAVA_OPTS -Dspringloaded.synchronize=true -javaagent:springloaded-1.2.1.jar -noverify )

Environment : JDK 1.8 U 66, RHEL 6.7

    #
    # A fatal error has been detected by the Java Runtime Environment:
    #
    #  SIGSEGV (0xb) at pc=0x00007faee9a1e27c, pid=27208, tid=140379827795712
    #
    # JRE version: Java(TM) SE Runtime Environment (8.0_66-b17) (build 1.8.0_66-b17)
    # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.66-b17 mixed mode linux-amd64 )
    # Problematic frame:
    # V  [libjvm.so+0x35027c]  Canonicalizer::do_If(If*)+0x1c
    #
    # Core dump written. Default location:  core.27208
    #
    # An error report file with more information is saved as:
    # hs_err_pid27208.log
    # [ timer expired, abort... ]

回答1:

I've noticed both -javaagent and -noverify in Java options list.

It looks like springloaded agent generates invalid bytecode, while the bytecode verification is explicitly turned off. No surprise, this may lead to unpredictable results including JVM crash.

This is not a JVM problem, but most likely a bug in springloaded agent. Try to remove -noverify option.

-XX:-TieredCompilation may also work around this particular problem, but don't expect application to work correctly if the bytecode fails to pass verification. It's better to stay away from the buggy agent libraries.

回答2:

4.2.1 Crash in HotSpot Compiler Thread or Compiled Code

If the fatal error log indicates that the crash occurred in a compiler thread, then it is possible (but not always the case) that you have encountered a compiler bug. Similarly, if the crash is in compiled code then it is possible that the compiler has generated incorrect code.

In the case of the HotSpot Client VM (-client option), the compiler thread appears in the error log as CompilerThread0. With the HotSpot Server VM there are multiple compiler threads and these appear in the error log file as CompilerThread0, CompilerThread1, and AdapterThread.

Below is a fragment of an error log for a compiler bug that was encountered and fixed during the development of J2SE 5.0. The log file shows that the HotSpot Server VM is used and the crash occurred in CompilerThread1. In addition, the log file shows that the Current CompileTask was the compilation of the java.lang.Thread.setPriority method.

An unexpected error has been detected by HotSpot Virtual Machine: : Java VM: Java HotSpot(TM) Server VM (1.5-internal-debug mixed mode) : --------------- T H R E A D ---------------

Current thread (0x001e9350): JavaThread "CompilerThread1" daemon [_thread_in_vm, id=20]

Stack: [0xb2500000,0xb2580000), sp=0xb257e500, free space=505k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0xc3b13c] :

Current CompileTask: opto: 11 java.lang.Thread.setPriority(I)V (53 bytes)

--------------- P R O C E S S ---------------

Java Threads: ( => current thread ) 0x00229930 JavaThread "Low Memory Detector" daemon [_thread_blocked, id=21] =>0x001e9350 JavaThread "CompilerThread1" daemon [_thread_in_vm, id=20] :

In this case there are two potential workarounds:

The brute force approach: change the configuration so that the application is run with the -client option to specify the HotSpot

Client VM.

Assume that the bug only occurs during the compilation of the setPriority method and exclude this method from compilation.

The first approach (to use the -client option) might be trivial to configure in some environments. In others, it might be more difficult if the configuration is complex or if the command line to configure the VM is not readily accessible. In general, switching from the HotSpot Server VM to the HotSpot Client VM also reduces the peak performance of an application. Depending on the environment, this might be acceptable until the actual issue is diagnosed and fixed.

The second approach (exclude the method from compilation) requires creating the file .hotspot_compiler in the working directory of the application. Below is an example of this file:

exclude java/lang/Thread setPriority

In general the format of this file is exclude CLASS METHOD, where CLASS is the class (fully qualified with the package name) and METHOD is the name of the method. Constructor methods are specified as and static initializers are specified as .

Note - The .hotspot_compiler file is an unsupported interface. It is documented here solely for the purposes of troubleshooting and finding a temporary workaround.

Once the application is restarted, the compiler will not attempt to compile any of the methods listed as excluded in the .hotspot_compiler file. In some cases this can provide temporary relief until the root cause of the crash is diagnosed and the bug is fixed.

In order to verify that the HotSpot VM correctly located and processed the .hotspot_compiler file that is shown in the example above, look for the following log information at runtime. Note that the file name separator is a dot, not a slash.

Excluding compile: java.lang.Thread::setPriority

Source

回答3:

Agree with @apangin, In the program you are doing bytecode intrumentation (-agent) but specifies -noverify. When verification is turned off, you may end up such crashes.

You should not use -noverify or -Xverify:none during byte code intrumentation.

For those of you unfamiliar with bytecode verification, it is simply part of the JVM's classloading process that checks the code for certain dangerous and disallowed behavior. You can (but shouldn't) disable this protection on many JVMs by adding -Xverify:none or -noverify to the Java command line. https://blogs.oracle.com/buck/entry/never_disable_bytecode_verification_in