Bug ID: JDK-8214939 C2 Compiler process is stuck in MemNode::can_see_stored

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 8u191

Priority: P4
Status: Closed
Resolution: Incomplete
OS: linux
CPU: x86_64

Submitted: 2018-12-05
Updated: 2021-11-22
Resolved: 2018-12-17

ADDITIONAL SYSTEM INFORMATION :
OS: 
Linux 3.16.0-7-amd64 #1 SMP Debian 3.16.59-1 (2018-10-03) x86_64 GNU/Linux

JAVA:
java version "1.8.0_91"
Java(TM) SE Runtime Environment (build 1.8.0_91-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.91-b14, mixed mode)

JAVA_OPTIONS:
-Xmx9000m 
-XX:+UseG1GC 
-XX:ParallelGCThreads=4 
-XX:ConcGCThreads=4 
-Dcom.sun.management.jmxremote.ssl=false 
-Dcom.sun.management.jmxremote.authenticate=false 
-Dcom.sun.management.jmxremote.port=9999


 

A DESCRIPTION OF THE PROBLEM :
Our jetty server was occupying 100% CPU time of two cores for ~500h. The cause was found in two sub-processes of the main jvm process. Both processes were related to the C2Compiler and both have been found to be stuck in the MemNode::can_see_stored_value(Node*, PhaseTransform*):

This is the stack trace of one of them:

~~~
[Thread debugging using libthread_db enabled]
Using host libthread_db library "/lib/x86_64-linux-gnu/libthread_db.so.1".
0x00007fe4a30c8eaf in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#0  0x00007fe4a30c8eaf in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#1  0x00007fe4a30c9c74 in LoadNode::Value(PhaseTransform*) const () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#2  0x00007fe4a31ac87a in PhaseIterGVN::transform_old(Node*) () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#3  0x00007fe4a31a9fcc in PhaseIterGVN::optimize() () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#4  0x00007fe4a2ce51a9 in Compile::Optimize() () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#5  0x00007fe4a2ce6fea in Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool) () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#6  0x00007fe4a2c34818 in C2Compiler::compile_method(ciEnv*, ciMethod*, int) () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#7  0x00007fe4a2cf14da in CompileBroker::invoke_compiler_on_method(CompileTask*) () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#8  0x00007fe4a2cf2486 in CompileBroker::compiler_thread_loop() () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#9  0x00007fe4a32ae77f in JavaThread::thread_main_inner() () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#10 0x00007fe4a32ae8ac in JavaThread::run() () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#11 0x00007fe4a3161f78 in java_start(Thread*) () from /opt/java-oracle/jdk1.8/jre/lib/amd64/server/libjvm.so
#12 0x00007fe4a481a064 in start_thread (arg=0x7fe43f921700) at pthread_create.c:309
#13 0x00007fe4a454f62d in clone () at ../sysdeps/unix/sysv/linux/x86_64/clone.S:111
~~~

The main java process was running as linux daemon using jsvc 1.0.15-6+deb8u1 (http://commons.apache.org/proper/commons-daemon/jsvc.html)

For more information on the analysis please see https://dev.e-taxonomy.eu/redmine/issues/7938

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
not reproducible so far, still trying


FREQUENCY : rarely

Closing as Incomplete for now as there is no reproducer to make any progress on this.

17-12-2018

We received similar issue JDK-8212637, report claims that "C2 Stuck in C2Compiler::compile_method with 100%CPU usage for one week" From the thread dump we could see, Thread 1 (process 68): #0 0xf6abda90 in MemBarCPUOrderNode::Opcode() const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #1 0xf6e8462c in MemNode::can_see_stored_value(Node*, PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #2 0xf6e84afb in LoadNode::Value(PhaseTransform*) const () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #3 0xf6f5368a in PhaseIterGVN::transform_old(Node*) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #4 0xf6f51084 in PhaseIterGVN::optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #5 0xf6aece24 in Compile::Optimize() () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #6 0xf6aeea0a in Compile::Compile(ciEnv*, C2Compiler*, ciMethod*, int, bool, bool, bool) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #7 0xf6a43a4d in C2Compiler::compile_method(ciEnv*, ciMethod*, int) () from /usr/java/jdk1.8.0_152/jre/lib/i386/server/libjvm.so #8 0xf6af92b1 in CompileBroker::invoke_compiler_on_method(CompileTask*) () from Looks like it got stuck in call to "int opc = current->in(0)->Opcode();" or "while (current->is_Proj())" is never set to break as we don't hit into preceding if block.

06-12-2018

There is only one change happened in "memnode.cpp" in the method "MemNode::can_see_stored_value(Node* st, PhaseTransform* phase)" comparing with latest JDK code v/s JDK 8 JDK-8076188: Optimize arraycopy out for non escaping destination Requesting submitter to verify against JDK11/JDK12 version to confirm if it is the same issue.

06-12-2018