JDK-8021898 : Broken JIT compiler optimization for loop unswitching
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 7u25
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • Submitted: 2013-07-30
  • Updated: 2014-02-12
  • Resolved: 2013-08-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availabitlity Release.

To download the current JDK release, click here.
JDK 7 JDK 8 Other
7u60Fixed 8 b105Fixed hs25Fixed
Description
FULL PRODUCT VERSION :
java version  " 1.7.0_25 " 
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)


FULL OS VERSION :
Linux xerosv 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux peugeot 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Linux bellatrix 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
The JIT compiler optimization leads to a SIGSEGV or an NullPointerException at a place it must not happen.

The demo code works well while having the JIT disabled (-Djava.compiler=NONE or -Xint), but it starts crashing as soon as the JIT is enabled. In production we've a little different situation where the service doesn't crash, but it raises an NullPointerException at a place where it is normally impossible.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
You can use this project to reproduce it, it is not from us, but makes it easy to reproduce the issue:

https://github.com/rholder/jvm-loop-unswitching-bug

The same issue is as well reported at other places and that is what we've got in our production environment:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected that it neither crashs with a SIGV nor that throws a NoSuchElementException.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Two possible result, the above code produces a SIGV:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff799074b27, pid=41781, tid=140701181404928
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J  com.nokia.cmp.middleware.LoopUnswitchBugTest$1.run()V
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try  " ulimit -c unlimited "  before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#


We've installed disassembler and it crashes here:


  0x00007ff799074b18: mov    %r10d,0x10(%rsp)
  0x00007ff799074b1d: mov    %ebx,%r8d
  0x00007ff799074b20: inc    %r8d               ;*iadd
                                                ; - java.util.ArrayList$Itr::next@53 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b23: mov    %r8d,0xc(%r13)     ;*putfield cursor
                                                ; - java.util.ArrayList$Itr::next@54 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b27: mov    0x8(%r12,%r10,8),%r10d  ; implicit exception: dispatches to 0x00007ff799074f95
  0x00007ff799074b2c: cmp    $0xefc9a08f,%r10d  ;   {oop('org/apache/http/impl/cookie/BasicClientCookie')}
  0x00007ff799074b33: jne    0x00007ff799074f4b  ;*checkcast
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@44 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b39: mov    %r8d,0x10(%r13)    ;*putfield lastRet
                                                ; - java.util.ArrayList$Itr::next@61 (line 799)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)



In our production environment it leads to a NoSuchElementException at a place where it must not happen:


NokiaSSOService: FAILED due to an unexpected exception, failture put into local cache
exception:
java.util.NoSuchElementException
     at java.util.ArrayList$Itr.next(ArrayList.java:794)
   at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:189)
    at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
      at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:108)
   at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:174)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:452)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:910)
        at com.nokia.idm.noa.http.ApacheHttpClient.call(ApacheHttpClient.java:59)
     at com.nokia.idm.noa.api.AuthenticateImpl.retrieve(AuthenticateImpl.java:210)
 at com.nokia.idm.noa.api.DefaultNoaApi.retrieve(DefaultNoaApi.java:139)
       at com.nokia.cmp.upm.authentication.NokiaSSOService.authenticate(NokiaSSOService.java:213)
    at com.nokia.cmp.upm.Manager$2.onCall(Manager.java:296)
       at com.nokia.cmp.aio.Async$Cursor.invoke(Async.java:786)
      at com.nokia.cmp.aio.Async$Cursor.access$900(Async.java:645)
  at com.nokia.cmp.aio.Async$Worker.executeCursor(Async.java:1659)
      at com.nokia.cmp.aio.Async$Worker.executeNextGlobalCursor(Async.java:1719)
    at com.nokia.cmp.aio.Async$Worker.run(Async.java:1935)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
The source code that we used to produce the SEGV including the hotspot.log (full disassembled code) and more details can be downloaded from here:

http://middleware.tst.cmp.nokia.com/file/logs.tgz

In fact we just used the code from here:

https://github.com/rholder/jvm-loop-unswitching-bug

  To try to reproduce as we have rather a NoSuchElementException. However, as our code is too complex and large to simply reproduce the problem the above is much better suited. For us in production NoSuchElementException happens in this code,as reported as well here:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173


    public List<Header> formatCookies(final List<Cookie> cookies) {
        if (cookies == null) {
            throw new IllegalArgumentException( " List of cookie may not be null " );
        }
        int version = Integer.MAX_VALUE;
        boolean isSetCookie2 = true;
        for (Cookie cookie: cookies) {    // <---- this is line 189
            if (!(cookie instanceof SetCookie2)) {
                isSetCookie2 = false;
            }
            if (cookie.getVersion() < version) {
                version = cookie.getVersion();
            }
        }
        if (version > 0) {
            if (isSetCookie2) {
                return getStrict().formatCookies(cookies);
            } else {
                return getObsoleteStrict().formatCookies(cookies);
            }
        } else {
            return getCompat().formatCookies(cookies);
        }
    }


As you can see it must be impossible that at line 189 a NoSuchElementException is raised as its a compiled loop.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workaround that works for us is to disable JIT, which is terrible in a production environment.
Comments
Can not change ADLC since each kill projection can have different ideal register type (RegI, RegP, Flags). Will fix clone_projs_shared().
2013-08-09

Adding Niclas for education purpose.
2013-08-09

Classical zero day RA problem. First, RA separates a node which produces flag from user by inserting MSC and LoadCon nodes: B94: # B504 B95 <- N4030 Freq: 0.283323 4029 Region === 4029 1209 [[ 4029 1207 1307 1308 3128 ]] 3128 partialSubtypeCheck_vs_Zero === 4029 3122 597 [[ 3129 3130 3127 ]] NULL 3129 MachProj === 3128 [[]] #1 3130 MachProj === 3128 [[]] #2 5087 MachSpillCopy === _ 3131 [[ 3127 ]] 5088 loadConI0 === 778 [[ 5089 5090 ]] #0/0x00000000 5089 MachProj === 5088 [[]] #1 !orig=[873] 5090 MachSpillCopy === _ 5088 [[ 3127 ]] 3127 cmovI_reg === _ 3128 5087 5090 [[ 5091 ]] ne !jvms: BestMatchSpec::formatCookies @ bci:59 LoopUnswitchBugTest$1::run @ bci:53 then RA clones the node producing flag (def->rematerialize()) and place it near user split_Rematerialize() : B94: # B504 B95 <- N4030 Freq: 0.283323 4029 Region === 4029 1209 [[ 4029 1207 1307 1308 3128 6164 ]] 3128 partialSubtypeCheck_vs_Zero === 4029 3122 597 [[ 3129 3130 3127 ]] NULL 3129 MachProj === 3128 [[]] #1 3130 MachProj === 3128 [[]] #2 5087 MachSpillCopy === _ 3131 [[ 3127 ]] 5088 loadConI0 === 778 [[ 5089 5090 ]] #0/0x00000000 5089 MachProj === 5088 [[]] #1 !orig=[873] 5090 MachSpillCopy === _ 5088 [[ 3127 ]] 6164 partialSubtypeCheck_vs_Zero === 4029 3122 597 [[ 6165 ]] NULL !orig=3128 6165 MachProj === 6164 [[]] #1 !orig=3129 3127 cmovI_reg === _ 3128 5087 5090 [[ 5091 ]] ne !jvms: BestMatchSpec::formatCookies @ bci:59 LoopUnswitchBugTest$1::run @ bci:53 when a node is cloned RA also should clone related MachProj nodes which indicate KILLed flags and registers (effect(KILL rcx, KILL result)). The PROBLEM is RA clones only ONE projection: see PhaseChaitin::clone_projs_shared()!!! 2 ways to solve the problem: - fix adlc to create only one MachProj with combined reg mask for all killed flags and registers. - fix clone_projs_shared() to clone all MachProj nodes.
2013-08-09

Note, with -XX:-LoopUnswitching partialSubtypeCheck_vs_Zero() instruction is not matching. Instead partialSubtypeCheck() is matched and it produces result in register instead of flags. The generated code in such case does not have the problem: 156b movq rdi, [RSI + in_bytes(Klass::secondary_supers_offset())] movl rcx, [rdi + Array<Klass*>::length_offset_in_bytes()] # length to scan addq rdi, Array<Klass*>::base_offset_in_bytes() # Skip to start of data; set NZ in case count is zero repne scasq # Scan *rdi++ for a match with rax while rcx-- jne,s miss # Missed: rdi not-zero movq [RSI + in_bytes(Klass::secondary_super_cache_offset())], RAX # Hit: update cache xorq RDI, RDI Hit: rdi zero miss: 1599 testq RDI, RDI # ptr 159c je B40 P=0.500000 C=-1.000000 159c 15a2 B239: # B40 <- B238 Freq: 0.0708232 15a2 xorl R14, R14 # int 15a5 jmp B40
2013-08-07

The code example which destroys rdi contant: 15e2 xorl RDI, RDI # int 15e4 movq rdi, [RSI + in_bytes(Klass::secondary_supers_offset())] movl rcx, [rdi + Array<Klass*>::length_offset_in_bytes()] # length to scan addq rdi, Array<Klass*>::base_offset_in_bytes() # Skip to start of data; set NZ in case count is zero repne scasq # Scan *rdi++ for a match with rax while cx-- != 0 jne,s miss # Missed: flags nz movq [RSI + in_bytes(Klass::secondary_super_cache_offset())], RAX # Hit: update cache miss: 160f cmovlne R11, RDI # signed, int
2013-08-01

As I found it is nothing to do with with loop unswitching. Loop unswitching created particular code shape with which we hit this problem. RDI register is incorrectly marked as KILL instead of TEMP in partialSubtypeCheck_vs_Zero instruction. As result RA think that it can keep an other int value in RDI across partialSubtypeCheck_vs_Zero code. diff -r 8d77d02828d9 src/cpu/x86/vm/x86_64.ad --- a/src/cpu/x86/vm/x86_64.ad Mon Jul 29 16:32:38 2013 -0700 +++ b/src/cpu/x86/vm/x86_64.ad Wed Jul 31 18:07:17 2013 -0700 @@ -11106,7 +11106,7 @@ rFlagsReg cr) %{ match(Set result (PartialSubtypeCheck sub super)); - effect(KILL rcx, KILL cr); + effect(TEMP rcx, KILL cr); ins_cost(1100); // slightly larger than the next version format %{ "movq rdi, [$sub + in_bytes(Klass::secondary_supers_offset())]\n\t" @@ -11129,7 +11129,7 @@ rdi_RegP result) %{ match(Set cr (CmpP (PartialSubtypeCheck sub super) zero)); - effect(KILL rcx, KILL result); + effect(TEMP rcx, TEMP result); ins_cost(1000); format %{ "movq rdi, [$sub + in_bytes(Klass::secondary_supers_offset())]\n\t"
2013-08-01

release-note-YES Justification: We should mention that the user should use -XX:-LoopUnswitching.
2013-07-31

Adding test and .jar files needed to reproduce the problem.
2013-07-31

-XX:InlineSmallCode=2000 -XX:-CheckCompressedOops flags may increase frequency of the failure with debug VM.
2013-07-31

Also with 7u40: $ cat hs_err_pid18115.log # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0xfffffd7ff8e84216, pid=18115, tid=107 # # JRE version: Java(TM) SE Runtime Environment (7.0_40-b35) (build 1.7.0_40-ea-b35) # Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b54 mixed mode solaris-amd64 compressed oops) # Problematic frame: # J org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x0000000000805800): JavaThread "pool-1-thread-44" [_thread_in_Java, id=107, stack(0xfffffd7fe2345000,0xfffffd7fe2445000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x0000000700ebdf48 Registers: RAX=0x00000003fc2cd180, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x0000000000000001 RSP=0xfffffd7fe2444580, RBP=0x00000006acaedf18, RSI=0x00000003fc2c8398, RDI=0x00000003fc2c8394 R8 =0x0000000000000000, R9 =0x00000000e01d7be8, R10=0x0000000000000000, R11=0x0000000000000002 R12=0x0000000000000000, R13=0x0000000000000001, R14=0x0000000000000004, R15=0x0000000000805800 RIP=0xfffffd7ff8e84216, RFLAGS=0x0000000000010203 Top of Stack: (sp=0xfffffd7fe2444580) 0xfffffd7fe2444580: 00000006acaec1b8 0000000500000005 0xfffffd7fe2444590: 00000006bb1583e0 0000000ad595d8f4 0xfffffd7fe24445a0: fffffd7fd674e754 00000006acaedee8 0xfffffd7fe24445b0: 00000006acaedee8 00000006acaed948 0xfffffd7fe24445c0: 00000006acaed948 00000006acaec1b8 0xfffffd7fe24445d0: fffffd7fe2444620 00000003fc2c87b0 0xfffffd7fe24445e0: 00000003fc2e5160 00000003fc2c6c78 0xfffffd7fe24445f0: 0000000000000000 fffffd7fe2444610 0xfffffd7fe2444600: fffffd7fe2444670 fffffd7ff8e06823 0xfffffd7fe2444610: 00000006acaedc68 fffffd7ff8e06823 0xfffffd7fe2444620: 00000006acaec1b8 00000006bb1583e0 0xfffffd7fe2444630: fffffd7fe2444630 00000003fc2be6fd 0xfffffd7fe2444640: fffffd7fe2444698 00000003fc2be9f8 0xfffffd7fe2444650: 00000003fc2e3928 00000003fc2be710 0xfffffd7fe2444660: fffffd7fe2444620 fffffd7fe2444698 0xfffffd7fe2444670: fffffd7fe24446e0 fffffd7ff8e069fe 0xfffffd7fe2444680: 00000006acaed948 0000000000000005 0xfffffd7fe2444690: 00000006acaec1b8 00000006ab3477f8 0xfffffd7fe24446a0: fffffd7fe24446a0 00000003fc2c28f4 0xfffffd7fe24446b0: fffffd7fe24446f0 00000003fc2c2c10 0xfffffd7fe24446c0: 00000003fc2e8340 00000003fc2c2908 0xfffffd7fe24446d0: fffffd7fe2444698 fffffd7fe24446f0 0xfffffd7fe24446e0: fffffd7fe2444738 fffffd7ff8e06823 0xfffffd7fe24446f0: 00000006ab347828 fffffd7fe24446f8 0xfffffd7fe2444700: 00000003fc2c0b7a fffffd7fe2444778 0xfffffd7fe2444710: 00000003fc2c1770 00000003fc2e8590 0xfffffd7fe2444720: 00000003fc2c0c08 fffffd7fe24446f0 0xfffffd7fe2444730: fffffd7fe2444778 fffffd7fe24447c0 0xfffffd7fe2444740: fffffd7ff8e069fe 0000000000000000 0xfffffd7fe2444750: 0000000000000000 0000000000000000 0xfffffd7fe2444760: 0000000000000000 0000000000000000 0xfffffd7fe2444770: 00000006ab347828 00000006ab347808 Instructions: (pc=0xfffffd7ff8e84216) 0xfffffd7ff8e841f6: 41 3b d6 7c 09 8b da e9 c3 00 00 00 8b d3 89 55 0xfffffd7ff8e84206: 10 44 8b 4c 97 10 44 8b da 41 ff c3 44 89 5d 0c 0xfffffd7ff8e84216: 43 8b 5c cc 08 81 fb 73 90 85 7f 0f 85 b5 14 00 0xfffffd7ff8e84226: 00 44 89 5d 10 33 db 50 48 8b c0 48 8b 7e 30 8b Register to memory mapping: RAX=0x00000003fc2cd180 is an oop {instance class} - klass: {other class} RBX=0x0000000000000000 is an unknown value RCX=0x0000000000000000 is an unknown value RDX=0x0000000000000001 is an unknown value RSP=0xfffffd7fe2444580 is pointing into the stack for thread: 0x0000000000805800 RBP=0x00000006acaedf18 is an oop java.util.ArrayList$Itr - klass: 'java/util/ArrayList$Itr' RSI=0x00000003fc2c8398 is an oop {instance class} - klass: {other class} RDI=0x00000003fc2c8394 is pointing into object: 0x00000003fc2c8370 [[I - klass: 'java/lang/Object'[] - length: 5 R8 =0x0000000000000000 is an unknown value R9 =0x00000000e01d7be8 is an unknown value R10=0x0000000000000000 is an unknown value R11=0x0000000000000002 is an unknown value R12=0x0000000000000000 is an unknown value R13=0x0000000000000001 is an unknown value R14=0x0000000000000004 is an unknown value R15=0x0000000000805800 is a thread Stack: [0xfffffd7fe2345000,0xfffffd7fe2445000], sp=0xfffffd7fe2444580, free space=1021k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;
2013-07-31

I reproduced it with 64-bit latest jdk8 VM: $ /java/re/jdk/8/latest/binaries/solaris-amd64/fastdebug/bin/java -d64 -cp .:httpcore-4.2.4.jar:httpclient-4.2.5.jar:commons-lang-2.6.jar -XX:-TieredCompilation LoopUnswitchBugTest $ cat hs_err_pid18097.log # # A fatal error has been detected by the Java Runtime Environment: # # SIGSEGV (0xb) at pc=0xfffffd7ff8f2ea53, pid=18097, tid=89 # # JRE version: Java(TM) SE Runtime Environment (8.0-b100) (build 1.8.0-ea-fastdebug-b100) # Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b42-fastdebug mixed mode solaris-amd64 compressed oops) # Problematic frame: # J org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xfffffd7ff8f2ea53 [0xfffffd7ff8f2dea0+2995] # # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x0000000000914800): JavaThread "pool-1-thread-26" [_thread_in_Java, id=89, stack(0xfffffd7fc09d4000,0xfffffd7fc0ad4000)] siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffffebfff205fb0c Registers: RAX=0x00000003fae63770, RBX=0x0000000000000001, RCX=0x0000000000000000, RDX=0x00000006ada61898 RSP=0xfffffd7fc0ad3280, RBP=0x00000006ada63030, RSI=0x00000003fae631c0, RDI=0xfffffd7ffe40bf60 R8 =0xffffebfff205fb00, R9 =0x0000000000000000, R10=0x000000007fffffff, R11=0x0000000000000001 R12=0x0000000000000000, R13=0x0000000000000004, R14=0x0000000000000005, R15=0x0000000000914800 RIP=0xfffffd7ff8f2ea53, RFLAGS=0x0000000000010202 Top of Stack: (sp=0xfffffd7fc0ad3280) 0xfffffd7fc0ad3280: 0000000500000001 d61053710000000a 0xfffffd7fc0ad3290: 00000006bb1010d0 00000006ada612a8 0xfffffd7fc0ad32a0: fffffd7fd5b4c313 fffffd7ffe40c000 0xfffffd7fc0ad32b0: fffffd7ffe41d988 fffffd7ffe40aa80 0xfffffd7fc0ad32c0: 0000000000000000 fffffd7fc0ad32e0 0xfffffd7fc0ad32d0: fffffd7fc0ad3340 fffffd7ff8e06ac2 0xfffffd7fc0ad32e0: fffffd7ffe408c30 fffffd7ff8e06ac2 0xfffffd7fc0ad32f0: 00000006ada612a8 00000006bb1010d0 0xfffffd7fc0ad3300: fffffd7fc0ad3300 fffffd7ffe408a15 0xfffffd7fc0ad3310: fffffd7fc0ad3368 fffffd7ffe408ae0 0xfffffd7fc0ad3320: fffffd7ffe41cda0 fffffd7ffe408a30 0xfffffd7fc0ad3330: fffffd7fc0ad32f0 fffffd7fc0ad3368 0xfffffd7fc0ad3340: fffffd7fc0ad33b0 fffffd7ff8e06946 0xfffffd7fc0ad3350: 00000006ada62a58 0000000000000005 0xfffffd7fc0ad3360: 00000006ada612a8 00000006aec09110 0xfffffd7fc0ad3370: fffffd7fc0ad3370 fffffd7ffe32c484 0xfffffd7fc0ad3380: fffffd7fc0ad33c0 fffffd7ffe32c560 0xfffffd7fc0ad3390: fffffd7ffe3421e8 fffffd7ffe32c4b0 0xfffffd7fc0ad33a0: fffffd7fc0ad3368 fffffd7fc0ad33c0 0xfffffd7fc0ad33b0: fffffd7fc0ad3408 fffffd7ff8e06ac2 0xfffffd7fc0ad33c0: 00000006aec09140 fffffd7fc0ad33c8 0xfffffd7fc0ad33d0: fffffd7ffe32abea fffffd7fc0ad3448 0xfffffd7fc0ad33e0: fffffd7ffe32b600 fffffd7ffe341fe8 0xfffffd7fc0ad33f0: fffffd7ffe32ace0 fffffd7fc0ad33c0 0xfffffd7fc0ad3400: fffffd7fc0ad3448 fffffd7fc0ad3490 0xfffffd7fc0ad3410: fffffd7ff8e06946 0000000000000000 0xfffffd7fc0ad3420: 0000000000000000 0000000000000000 0xfffffd7fc0ad3430: 0000000000000000 0000000000000000 0xfffffd7fc0ad3440: 00000006aec09140 00000006aec09120 0xfffffd7fc0ad3450: fffffd7fc0ad3450 fffffd7ffe3072ff 0xfffffd7fc0ad3460: fffffd7fc0ad34e8 fffffd7ffe30cdd0 0xfffffd7fc0ad3470: fffffd7ffe342528 fffffd7ffe307480 Instructions: (pc=0xfffffd7ff8f2ea53) 0xfffffd7ff8f2ea33: 28 8b 0f 48 83 c7 08 48 85 c0 f2 48 af 58 0f 85 0xfffffd7ff8f2ea43: 04 00 00 00 48 89 46 20 45 0f 45 c8 4d 8d 04 fc 0xfffffd7ff8f2ea53: 45 8b 40 0c 45 3b c2 45 0f 4c d0 41 3b db 7d 08 0xfffffd7ff8f2ea63: 44 8b c3 e9 6e ff ff ff 41 3b dd 7c 13 44 8b c3 Register to memory mapping: RAX=0x00000003fae63770 is pointing into metadata RBX=0x0000000000000001 is an unknown value RCX=0x0000000000000000 is an unknown value RDX=0x00000006ada61898 is an oop [Ljava.lang.Object; - klass: 'java/lang/Object'[] - length: 10 - 0 : a 'org/apache/http/impl/cookie/BasicClientCookie' - 1 : a 'org/apache/http/impl/cookie/BasicClientCookie' - 2 : a 'org/apache/http/impl/cookie/BasicClientCookie' - 3 : a 'org/apache/http/impl/cookie/BasicClientCookie' - 4 : a 'org/apache/http/impl/cookie/BasicClientCookie' - 5 : NULL - 6 : NULL - 7 : NULL - 8 : NULL - 9 : NULL RSP=0xfffffd7fc0ad3280 is pointing into the stack for thread: 0x0000000000914800 RBP=0x00000006ada63030 is an oop java.util.ArrayList$Itr - klass: 'java/util/ArrayList$Itr' - ---- fields (total size 4 words): - 'cursor' 'I' @12 1 - 'lastRet' 'I' @16 0 - 'expectedModCount' 'I' @20 5 - final synthetic 'this$0' 'Ljava/util/ArrayList;' @24 a 'java/util/ArrayList' (d5b4c255 0) RSI=0x00000003fae631c0 is pointing into metadata RDI=0xfffffd7ffe40bf60 is pointing into metadata R8 =0xffffebfff205fb00 is an unknown value R9 =0x0000000000000000 is an unknown value R10=0x000000007fffffff is an unknown value R11=0x0000000000000001 is an unknown value R12=0x0000000000000000 is an unknown value R13=0x0000000000000004 is an unknown value R14=0x0000000000000005 is an unknown value R15=0x0000000000914800 is a thread Stack: [0xfffffd7fc09d4000,0xfffffd7fc0ad4000], sp=0xfffffd7fc0ad3280, free space=1020k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) J org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xfffffd7ff8f2ea53 [0xfffffd7ff8f2dea0+2995] And 32-bit VM: $ /java/re/jdk/8/latest/binaries/solaris-i586/fastdebug/bin/java -cp .:httpcore-4.2.4.jar:httpclient-4.2.5.jar:commons-lang-2.6.jar -XX:-TieredCompilation LoopUnswitchBugTest $ cat hs_err_pid18082.log # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (/tmp/workspace/2-build-solaris-i586/jdk8/5205/hotspot/src/share/vm/runtime/handles.cpp:48), pid=18082, tid=144 # assert(obj->is_oop()) failed: sanity check # # JRE version: Java(TM) SE Runtime Environment (8.0-b100) (build 1.8.0-ea-fastdebug-b100) # Java VM: Java HotSpot(TM) Server VM (25.0-b42-fastdebug mixed mode solaris-x86 ) # Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again # # If you would like to submit a bug report, please visit: # http://bugreport.sun.com/bugreport/crash.jsp # --------------- T H R E A D --------------- Current thread (0x08564000): JavaThread "pool-1-thread-81" [_thread_in_Java, id=144, stack(0xb209f000,0xb20ef000)] Stack: [0xb209f000,0xb20ef000], sp=0xb20ed3c0, free space=312k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x1106abf] void VMError::report(outputStream*)+0x95f V [libjvm.so+0x1107fa9] void VMError::report_and_die()+0x5e9 V [libjvm.so+0x731b58] void report_vm_error(const char*,int,const char*,const char*)+0x68 V [libjvm.so+0x8e2d06] oop*HandleArea::allocate_handle(oop)+0x156 V [libjvm.so+0xfb0bb8] StackValue*StackValue::create_stack_value(const frame*,const RegisterMap*,ScopeValue*)+0x2c8 V [libjvm.so+0x10ff875] StackValueCollection*compiledVFrame::expressions()const+0xd5 V [libjvm.so+0x10faa9f] void vframeArrayElement::fill_in(compiledVFrame*)+0x72f V [libjvm.so+0x10fc5f1] void vframeArray::fill_in(JavaThread*,int,GrowableArray<compiledVFrame*>*,const RegisterMap*)+0xc1 V [libjvm.so+0x10fc4e6] vframeArray*vframeArray::allocate(JavaThread*,int,GrowableArray<compiledVFrame*>*,RegisterMap*,frame,frame,frame)+0x116 V [libjvm.so+0x75a804] vframeArray*Deoptimization::create_vframeArray(JavaThread*,frame,RegisterMap*,GrowableArray<compiledVFrame*>*)+0x484 V [libjvm.so+0x755775] Deoptimization::UnrollBlock*Deoptimization::fetch_unroll_info_helper(JavaThread*)+0xd35 V [libjvm.so+0x75e4dc] Deoptimization::UnrollBlock*Deoptimization::uncommon_trap(JavaThread*,int)+0x2c Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) v blob 0xf96aaae3 J org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xf96f6054 [0xf96f3e00+8788] j LoopUnswitchBugTest$1.run()V+53 j java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4 j java.util.concurrent.FutureTask.run()V+42 j java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95 j java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5 j java.lang.Thread.run()V+11 v ~StubRoutines::call_stub
2013-07-31

7u40-defer-request Justification: There is a workaround for this, requesting a deferral.
2013-07-30

Both links show that they know about workaround: -XX:-LoopUnswitching. https://github.com/rholder/jvm-loop-unswitching-bug https://issues.apache.org/jira/browse/HTTPCLIENT-1173 Also they know about this for long time. Apache bug shows that the issue was reported on 14/Mar/12 and workaround was found on 14/Feb/13. I will try to find the cause and fix it in the time we have before 7u40.
2013-07-30

Moving from hotspot/runtime -> hotspot/compiler for initial eval.
2013-07-30