Bug ID: JDK-8035887 VM crashes trying to force inlining the recursive call

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 8	JDK 9
8u20Fixed	9 b06Fixed

Found in the wild:
 http://mail.openjdk.java.net/pipermail/jmh-dev/2014-February/000511.html

Minimal test case:

public class Test {

        public static void doCall(int claims) {
                if (claims == 0) return;
                doCall(claims - 1);
        }

        public static void main(String[] args) {
                for (int i = 0; i < 10000; i++) {
                        doCall(i);
                }
        }
}

Works fine on Linux x86_64, JDK 8b129 out-of-box, segfaults with forceful inlining via CompileCommand:

$ ~/Install/jdk8b129/bin/java -XX:CompileCommand=inline,Test.doCall -XX:+PrintCompilation  Test
CompilerOracle: inline Test.doCall
     68    1       3       java.lang.String::hashCode (55 bytes)
     69    2       3       java.lang.Object::<init> (1 bytes)
     70    3       3       java.lang.String::equals (81 bytes)
     73    4       3       java.lang.String::indexOf (70 bytes)
     73    6     n 0       java.lang.System::arraycopy (native)   (static)
     73    5       3       java.lang.String::charAt (29 bytes)
     74    7       3       java.lang.Math::min (11 bytes)
     74    8       3       java.lang.String::length (6 bytes)
     78    9       1       java.lang.Object::<init> (1 bytes)
     78    2       3       java.lang.Object::<init> (1 bytes)   made not entrant
     78   10       3       java.util.Arrays::copyOfRange (63 bytes)
     80   11       3       Test::doCall (12 bytes)
Segmentation fault (core dumped)

jdk9/hs-comp also affected, *BUT ONLY THE RELEASE BUILD*, fastdebug works fine.

noreg-long: provoking the crash can take very long time in some configurations.
27-02-2014
Review request: http://mail.openjdk.java.net/pipermail/hotspot-compiler-dev/2014-February/013599.html Proposed fix: http://cr.openjdk.java.net/~vlivanov/8035887/webrev.00/
27-02-2014
Top of the stack is irrelevant here - compiler can hit stack overflow at a random place. The problem is GraphBuilder::try_inline_full inlines a method unconditionally: src/share/vm/c1/c1_GraphBuilder.cpp:3773 } else if (callee->should_inline()) { print_inlining(callee, "force inline by CompileOracle"); } else {
27-02-2014
C2 doesn't have this issue: $ java -XX:CompileCommand=inline,Test.doCall -XX:+UnlockDiagnosticVMOptions -XX:-TieredCompilation -XX:+PrintInlining -XX:+PrintCompilation Test CompilerOracle: inline Test.doCall 59 1 Test::doCall (12 bytes) @ 8 Test::doCall (12 bytes) force inline by CompilerOracle @ 8 Test::doCall (12 bytes) recursive inlining is too deep
27-02-2014
Head of the coredump: Program terminated with signal 11, Segmentation fault. #0 0x00007fa44b175b1e in LinkResolver::resolve_method(methodHandle&, KlassHandle, Symbol, Symbol, KlassHandle, bool, bool, Thread*) ()
26-02-2014
ILW=HLL=P4 Impact: Crash Likelihood: Requires the user to forcefully inline via CompileCommand. This isn't normal operation Workaround: Don't forcefully inline the method Fix this later
26-02-2014