United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-8021898 : Broken JIT compiler optimization for loop unswitching

Details
Type:
Bug
Submit Date:
2013-07-30
Status:
Resolved
Updated Date:
2013-10-19
Project Name:
JDK
Resolved Date:
2013-08-27
Component:
hotspot
OS:
linux
Sub-Component:
compiler
CPU:
Priority:
P3
Resolution:
Fixed
Affected Versions:
7u25
Fixed Versions:

Related Reports
Backport:
Backport:

Sub Tasks

Description
FULL PRODUCT VERSION :
java version  " 1.7.0_25 " 
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) 64-Bit Server VM (build 23.25-b01, mixed mode)


FULL OS VERSION :
Linux xerosv 3.2.0-49-generic #75-Ubuntu SMP Tue Jun 18 17:39:32 UTC 2013 x86_64 x86_64 x86_64 GNU/Linux
Linux peugeot 2.6.18-238.el5 #1 SMP Thu Jan 13 15:51:15 EST 2011 x86_64 x86_64 x86_64 GNU/Linux
Linux bellatrix 2.6.32-71.el6.x86_64 #1 SMP Fri May 20 03:51:51 BST 2011 x86_64 x86_64 x86_64 GNU/Linux
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
The JIT compiler optimization leads to a SIGSEGV or an NullPointerException at a place it must not happen.

The demo code works well while having the JIT disabled (-Djava.compiler=NONE or -Xint), but it starts crashing as soon as the JIT is enabled. In production we've a little different situation where the service doesn't crash, but it raises an NullPointerException at a place where it is normally impossible.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
You can use this project to reproduce it, it is not from us, but makes it easy to reproduce the issue:

https://github.com/rholder/jvm-loop-unswitching-bug

The same issue is as well reported at other places and that is what we've got in our production environment:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173

EXPECTED VERSUS ACTUAL BEHAVIOR :
Expected that it neither crashs with a SIGV nor that throws a NoSuchElementException.
ERROR MESSAGES/STACK TRACES THAT OCCUR :
Two possible result, the above code produces a SIGV:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00007ff799074b27, pid=41781, tid=140701181404928
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.25-b01 mixed mode linux-amd64 compressed oops)
# Problematic frame:
# J  com.nokia.cmp.middleware.LoopUnswitchBugTest$1.run()V
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try  " ulimit -c unlimited "  before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#


We've installed disassembler and it crashes here:


  0x00007ff799074b18: mov    %r10d,0x10(%rsp)
  0x00007ff799074b1d: mov    %ebx,%r8d
  0x00007ff799074b20: inc    %r8d               ;*iadd
                                                ; - java.util.ArrayList$Itr::next@53 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b23: mov    %r8d,0xc(%r13)     ;*putfield cursor
                                                ; - java.util.ArrayList$Itr::next@54 (line 798)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b27: mov    0x8(%r12,%r10,8),%r10d  ; implicit exception: dispatches to 0x00007ff799074f95
  0x00007ff799074b2c: cmp    $0xefc9a08f,%r10d  ;   {oop('org/apache/http/impl/cookie/BasicClientCookie')}
  0x00007ff799074b33: jne    0x00007ff799074f4b  ;*checkcast
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@44 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)
  0x00007ff799074b39: mov    %r8d,0x10(%r13)    ;*putfield lastRet
                                                ; - java.util.ArrayList$Itr::next@61 (line 799)
                                                ; - org.apache.http.impl.cookie.BestMatchSpec::formatCookies@39 (line 189)
                                                ; - com.nokia.cmp.middleware.LoopUnswitchBugTest$1::run@53 (line 46)



In our production environment it leads to a NoSuchElementException at a place where it must not happen:


NokiaSSOService: FAILED due to an unexpected exception, failture put into local cache
exception:
java.util.NoSuchElementException
     at java.util.ArrayList$Itr.next(ArrayList.java:794)
   at org.apache.http.impl.cookie.BestMatchSpec.formatCookies(BestMatchSpec.java:189)
    at org.apache.http.client.protocol.RequestAddCookies.process(RequestAddCookies.java:196)
      at org.apache.http.protocol.ImmutableHttpProcessor.process(ImmutableHttpProcessor.java:108)
   at org.apache.http.protocol.HttpRequestExecutor.preProcess(HttpRequestExecutor.java:174)
      at org.apache.http.impl.client.DefaultRequestDirector.execute(DefaultRequestDirector.java:452)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:820)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:941)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:919)
        at org.apache.http.impl.client.AbstractHttpClient.execute(AbstractHttpClient.java:910)
        at com.nokia.idm.noa.http.ApacheHttpClient.call(ApacheHttpClient.java:59)
     at com.nokia.idm.noa.api.AuthenticateImpl.retrieve(AuthenticateImpl.java:210)
 at com.nokia.idm.noa.api.DefaultNoaApi.retrieve(DefaultNoaApi.java:139)
       at com.nokia.cmp.upm.authentication.NokiaSSOService.authenticate(NokiaSSOService.java:213)
    at com.nokia.cmp.upm.Manager$2.onCall(Manager.java:296)
       at com.nokia.cmp.aio.Async$Cursor.invoke(Async.java:786)
      at com.nokia.cmp.aio.Async$Cursor.access$900(Async.java:645)
  at com.nokia.cmp.aio.Async$Worker.executeCursor(Async.java:1659)
      at com.nokia.cmp.aio.Async$Worker.executeNextGlobalCursor(Async.java:1719)
    at com.nokia.cmp.aio.Async$Worker.run(Async.java:1935)


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
The source code that we used to produce the SEGV including the hotspot.log (full disassembled code) and more details can be downloaded from here:

http://middleware.tst.cmp.nokia.com/file/logs.tgz

In fact we just used the code from here:

https://github.com/rholder/jvm-loop-unswitching-bug

  To try to reproduce as we have rather a NoSuchElementException. However, as our code is too complex and large to simply reproduce the problem the above is much better suited. For us in production NoSuchElementException happens in this code,as reported as well here:

https://issues.apache.org/jira/browse/HTTPCLIENT-1173


    public List<Header> formatCookies(final List<Cookie> cookies) {
        if (cookies == null) {
            throw new IllegalArgumentException( " List of cookie may not be null " );
        }
        int version = Integer.MAX_VALUE;
        boolean isSetCookie2 = true;
        for (Cookie cookie: cookies) {    // <---- this is line 189
            if (!(cookie instanceof SetCookie2)) {
                isSetCookie2 = false;
            }
            if (cookie.getVersion() < version) {
                version = cookie.getVersion();
            }
        }
        if (version > 0) {
            if (isSetCookie2) {
                return getStrict().formatCookies(cookies);
            } else {
                return getObsoleteStrict().formatCookies(cookies);
            }
        } else {
            return getCompat().formatCookies(cookies);
        }
    }


As you can see it must be impossible that at line 189 a NoSuchElementException is raised as its a compiled loop.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workaround that works for us is to disable JIT, which is terrible in a production environment.
                                    

Comments
URL:   http://hg.openjdk.java.net/jdk8/jdk8/hotspot/rev/4b2838704fd5
User:  amurillo
Date:  2013-08-27 20:04:58 +0000

                                     
2013-08-27
Classical zero day RA problem.

First, RA separates a node which produces flag from user by inserting MSC and LoadCon nodes:

B94: #  B504 B95 <- N4030  Freq: 0.283323
 4029   Region  ===  4029  1209  [[ 4029  1207  1307  1308  3128 ]] 
 3128   partialSubtypeCheck_vs_Zero     ===  4029  3122  597  [[ 3129  3130  3127 ]] NULL
 3129   MachProj        ===  3128  [[]] #1
 3130   MachProj        ===  3128  [[]] #2

 5087   MachSpillCopy   === _  3131  [[ 3127 ]] 
 5088   loadConI0       ===  778  [[ 5089  5090 ]] #0/0x00000000
 5089   MachProj        ===  5088  [[]] #1 !orig=[873]
 5090   MachSpillCopy   === _  5088  [[ 3127 ]] 

 3127   cmovI_reg       === _  3128  5087  5090  [[ 5091 ]] ne !jvms: BestMatchSpec::formatCookies @ bci:59 LoopUnswitchBugTest$1::run @ bci:53

then RA clones the node producing flag (def->rematerialize()) and place it near user split_Rematerialize() :

B94: #  B504 B95 <- N4030  Freq: 0.283323
 4029   Region  ===  4029  1209  [[ 4029  1207  1307  1308  3128  6164 ]] 
 3128   partialSubtypeCheck_vs_Zero     ===  4029  3122  597  [[ 3129  3130  3127 ]] NULL
 3129   MachProj        ===  3128  [[]] #1
 3130   MachProj        ===  3128  [[]] #2
 5087   MachSpillCopy   === _  3131  [[ 3127 ]] 
 5088   loadConI0       ===  778  [[ 5089  5090 ]] #0/0x00000000
 5089   MachProj        ===  5088  [[]] #1 !orig=[873]
 5090   MachSpillCopy   === _  5088  [[ 3127 ]] 

 6164   partialSubtypeCheck_vs_Zero     ===  4029  3122  597  [[ 6165 ]] NULL !orig=3128
 6165   MachProj        ===  6164  [[]] #1 !orig=3129
 3127   cmovI_reg       === _  3128  5087  5090  [[ 5091 ]] ne !jvms: BestMatchSpec::formatCookies @ bci:59 LoopUnswitchBugTest$1::run @ bci:53

when a node is cloned RA also should clone related MachProj nodes which indicate KILLed flags and registers (effect(KILL rcx, KILL result)).

The PROBLEM is RA clones only ONE projection: see PhaseChaitin::clone_projs_shared()!!!

2 ways to solve the problem:
- fix adlc to create only one MachProj with combined reg mask for all killed flags and registers.
- fix clone_projs_shared() to clone all MachProj nodes.

                                     
2013-08-09
Can not change ADLC since each kill projection can have different ideal register type (RegI, RegP, Flags).
Will fix clone_projs_shared().

                                     
2013-08-09
Adding Niclas for education purpose.

                                     
2013-08-09
Note, with -XX:-LoopUnswitching partialSubtypeCheck_vs_Zero() instruction is not matching. Instead partialSubtypeCheck() is matched and it produces result in register instead of flags. The generated code in such case does not have the problem:

156b    movq    rdi, [RSI + in_bytes(Klass::secondary_supers_offset())]
        movl    rcx, [rdi + Array<Klass*>::length_offset_in_bytes()]    # length to scan
        addq    rdi, Array<Klass*>::base_offset_in_bytes()      # Skip to start of data; set NZ in case count is zero
        repne   scasq   # Scan *rdi++ for a match with rax while rcx--
        jne,s   miss            # Missed: rdi not-zero
        movq    [RSI + in_bytes(Klass::secondary_super_cache_offset())], RAX    # Hit: update cache
        xorq    RDI, RDI                 Hit: rdi zero
        miss:   
1599    testq   RDI, RDI        # ptr
159c    je     B40  P=0.500000 C=-1.000000
159c
15a2   B239: #  B40 <- B238  Freq: 0.0708232
15a2    xorl    R14, R14        # int
15a5    jmp     B40

                                     
2013-08-07
As I found it is nothing to do with with loop unswitching. Loop unswitching created particular code shape with which we hit this problem. RDI register is incorrectly marked as KILL instead of TEMP in partialSubtypeCheck_vs_Zero instruction. As result RA think that it can keep an other int value in RDI across partialSubtypeCheck_vs_Zero code.

diff -r 8d77d02828d9 src/cpu/x86/vm/x86_64.ad
--- a/src/cpu/x86/vm/x86_64.ad	Mon Jul 29 16:32:38 2013 -0700
+++ b/src/cpu/x86/vm/x86_64.ad	Wed Jul 31 18:07:17 2013 -0700
@@ -11106,7 +11106,7 @@
                              rFlagsReg cr)
 %{
   match(Set result (PartialSubtypeCheck sub super));
-  effect(KILL rcx, KILL cr);
+  effect(TEMP rcx, KILL cr);
 
   ins_cost(1100);  // slightly larger than the next version
   format %{ "movq    rdi, [$sub + in_bytes(Klass::secondary_supers_offset())]\n\t"
@@ -11129,7 +11129,7 @@
                                      rdi_RegP result)
 %{
   match(Set cr (CmpP (PartialSubtypeCheck sub super) zero));
-  effect(KILL rcx, KILL result);
+  effect(TEMP rcx, TEMP result);
 
   ins_cost(1000);
   format %{ "movq    rdi, [$sub + in_bytes(Klass::secondary_supers_offset())]\n\t"

                                     
2013-08-01
The code example which destroys rdi contant: 

15e2    xorl    RDI, RDI        # int
15e4    movq    rdi, [RSI + in_bytes(Klass::secondary_supers_offset())]
        movl    rcx, [rdi + Array<Klass*>::length_offset_in_bytes()]    # length to scan
        addq    rdi, Array<Klass*>::base_offset_in_bytes()      # Skip to start of data; set NZ in case count is zero
        repne   scasq   # Scan *rdi++ for a match with rax while cx-- != 0
        jne,s   miss            # Missed: flags nz
        movq    [RSI + in_bytes(Klass::secondary_super_cache_offset())], RAX    # Hit: update cache
        miss:   
160f    cmovlne R11, RDI        # signed, int

                                     
2013-08-01
I reproduced it with 64-bit latest jdk8 VM:

$ /java/re/jdk/8/latest/binaries/solaris-amd64/fastdebug/bin/java -d64 -cp .:httpcore-4.2.4.jar:httpclient-4.2.5.jar:commons-lang-2.6.jar -XX:-TieredCompilation LoopUnswitchBugTest

$ cat hs_err_pid18097.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfffffd7ff8f2ea53, pid=18097, tid=89
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b100) (build 1.8.0-ea-fastdebug-b100)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (25.0-b42-fastdebug mixed mode solaris-amd64 compressed oops)
# Problematic frame:
# J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xfffffd7ff8f2ea53 [0xfffffd7ff8f2dea0+2995]
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000914800):  JavaThread "pool-1-thread-26" [_thread_in_Java, id=89, stack(0xfffffd7fc09d4000,0xfffffd7fc0ad4000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0xffffebfff205fb0c

Registers:
RAX=0x00000003fae63770, RBX=0x0000000000000001, RCX=0x0000000000000000, RDX=0x00000006ada61898
RSP=0xfffffd7fc0ad3280, RBP=0x00000006ada63030, RSI=0x00000003fae631c0, RDI=0xfffffd7ffe40bf60
R8 =0xffffebfff205fb00, R9 =0x0000000000000000, R10=0x000000007fffffff, R11=0x0000000000000001
R12=0x0000000000000000, R13=0x0000000000000004, R14=0x0000000000000005, R15=0x0000000000914800
RIP=0xfffffd7ff8f2ea53, RFLAGS=0x0000000000010202

Top of Stack: (sp=0xfffffd7fc0ad3280)
0xfffffd7fc0ad3280:   0000000500000001 d61053710000000a
0xfffffd7fc0ad3290:   00000006bb1010d0 00000006ada612a8
0xfffffd7fc0ad32a0:   fffffd7fd5b4c313 fffffd7ffe40c000
0xfffffd7fc0ad32b0:   fffffd7ffe41d988 fffffd7ffe40aa80
0xfffffd7fc0ad32c0:   0000000000000000 fffffd7fc0ad32e0
0xfffffd7fc0ad32d0:   fffffd7fc0ad3340 fffffd7ff8e06ac2
0xfffffd7fc0ad32e0:   fffffd7ffe408c30 fffffd7ff8e06ac2
0xfffffd7fc0ad32f0:   00000006ada612a8 00000006bb1010d0
0xfffffd7fc0ad3300:   fffffd7fc0ad3300 fffffd7ffe408a15
0xfffffd7fc0ad3310:   fffffd7fc0ad3368 fffffd7ffe408ae0
0xfffffd7fc0ad3320:   fffffd7ffe41cda0 fffffd7ffe408a30
0xfffffd7fc0ad3330:   fffffd7fc0ad32f0 fffffd7fc0ad3368
0xfffffd7fc0ad3340:   fffffd7fc0ad33b0 fffffd7ff8e06946
0xfffffd7fc0ad3350:   00000006ada62a58 0000000000000005
0xfffffd7fc0ad3360:   00000006ada612a8 00000006aec09110
0xfffffd7fc0ad3370:   fffffd7fc0ad3370 fffffd7ffe32c484
0xfffffd7fc0ad3380:   fffffd7fc0ad33c0 fffffd7ffe32c560
0xfffffd7fc0ad3390:   fffffd7ffe3421e8 fffffd7ffe32c4b0
0xfffffd7fc0ad33a0:   fffffd7fc0ad3368 fffffd7fc0ad33c0
0xfffffd7fc0ad33b0:   fffffd7fc0ad3408 fffffd7ff8e06ac2
0xfffffd7fc0ad33c0:   00000006aec09140 fffffd7fc0ad33c8
0xfffffd7fc0ad33d0:   fffffd7ffe32abea fffffd7fc0ad3448
0xfffffd7fc0ad33e0:   fffffd7ffe32b600 fffffd7ffe341fe8
0xfffffd7fc0ad33f0:   fffffd7ffe32ace0 fffffd7fc0ad33c0
0xfffffd7fc0ad3400:   fffffd7fc0ad3448 fffffd7fc0ad3490
0xfffffd7fc0ad3410:   fffffd7ff8e06946 0000000000000000
0xfffffd7fc0ad3420:   0000000000000000 0000000000000000
0xfffffd7fc0ad3430:   0000000000000000 0000000000000000
0xfffffd7fc0ad3440:   00000006aec09140 00000006aec09120
0xfffffd7fc0ad3450:   fffffd7fc0ad3450 fffffd7ffe3072ff
0xfffffd7fc0ad3460:   fffffd7fc0ad34e8 fffffd7ffe30cdd0
0xfffffd7fc0ad3470:   fffffd7ffe342528 fffffd7ffe307480 

Instructions: (pc=0xfffffd7ff8f2ea53)
0xfffffd7ff8f2ea33:   28 8b 0f 48 83 c7 08 48 85 c0 f2 48 af 58 0f 85
0xfffffd7ff8f2ea43:   04 00 00 00 48 89 46 20 45 0f 45 c8 4d 8d 04 fc
0xfffffd7ff8f2ea53:   45 8b 40 0c 45 3b c2 45 0f 4c d0 41 3b db 7d 08
0xfffffd7ff8f2ea63:   44 8b c3 e9 6e ff ff ff 41 3b dd 7c 13 44 8b c3 

Register to memory mapping:

RAX=0x00000003fae63770 is pointing into metadata
RBX=0x0000000000000001 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x00000006ada61898 is an oop
[Ljava.lang.Object; 
 - klass: 'java/lang/Object'[]
 - length: 10
 -   0 : a 'org/apache/http/impl/cookie/BasicClientCookie'
 -   1 : a 'org/apache/http/impl/cookie/BasicClientCookie'
 -   2 : a 'org/apache/http/impl/cookie/BasicClientCookie'
 -   3 : a 'org/apache/http/impl/cookie/BasicClientCookie'
 -   4 : a 'org/apache/http/impl/cookie/BasicClientCookie'
 -   5 : NULL
 -   6 : NULL
 -   7 : NULL
 -   8 : NULL
 -   9 : NULL
RSP=0xfffffd7fc0ad3280 is pointing into the stack for thread: 0x0000000000914800
RBP=0x00000006ada63030 is an oop
java.util.ArrayList$Itr 
 - klass: 'java/util/ArrayList$Itr'
 - ---- fields (total size 4 words):
 - 'cursor' 'I' @12  1
 - 'lastRet' 'I' @16  0
 - 'expectedModCount' 'I' @20  5
 - final synthetic 'this$0' 'Ljava/util/ArrayList;' @24  a 'java/util/ArrayList' (d5b4c255 0)
RSI=0x00000003fae631c0 is pointing into metadata
RDI=0xfffffd7ffe40bf60 is pointing into metadata
R8 =0xffffebfff205fb00 is an unknown value
R9 =0x0000000000000000 is an unknown value
R10=0x000000007fffffff is an unknown value
R11=0x0000000000000001 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000004 is an unknown value
R14=0x0000000000000005 is an unknown value
R15=0x0000000000914800 is a thread


Stack: [0xfffffd7fc09d4000,0xfffffd7fc0ad4000],  sp=0xfffffd7fc0ad3280,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xfffffd7ff8f2ea53 [0xfffffd7ff8f2dea0+2995]




And 32-bit VM:

$ /java/re/jdk/8/latest/binaries/solaris-i586/fastdebug/bin/java -cp .:httpcore-4.2.4.jar:httpclient-4.2.5.jar:commons-lang-2.6.jar -XX:-TieredCompilation LoopUnswitchBugTest

$ cat hs_err_pid18082.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/tmp/workspace/2-build-solaris-i586/jdk8/5205/hotspot/src/share/vm/runtime/handles.cpp:48), pid=18082, tid=144
#  assert(obj->is_oop()) failed: sanity check
#
# JRE version: Java(TM) SE Runtime Environment (8.0-b100) (build 1.8.0-ea-fastdebug-b100)
# Java VM: Java HotSpot(TM) Server VM (25.0-b42-fastdebug mixed mode solaris-x86 )
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x08564000):  JavaThread "pool-1-thread-81" [_thread_in_Java, id=144, stack(0xb209f000,0xb20ef000)]

Stack: [0xb209f000,0xb20ef000],  sp=0xb20ed3c0,  free space=312k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x1106abf]  void VMError::report(outputStream*)+0x95f
V  [libjvm.so+0x1107fa9]  void VMError::report_and_die()+0x5e9
V  [libjvm.so+0x731b58]  void report_vm_error(const char*,int,const char*,const char*)+0x68
V  [libjvm.so+0x8e2d06]  oop*HandleArea::allocate_handle(oop)+0x156
V  [libjvm.so+0xfb0bb8]  StackValue*StackValue::create_stack_value(const frame*,const RegisterMap*,ScopeValue*)+0x2c8
V  [libjvm.so+0x10ff875]  StackValueCollection*compiledVFrame::expressions()const+0xd5
V  [libjvm.so+0x10faa9f]  void vframeArrayElement::fill_in(compiledVFrame*)+0x72f
V  [libjvm.so+0x10fc5f1]  void vframeArray::fill_in(JavaThread*,int,GrowableArray<compiledVFrame*>*,const RegisterMap*)+0xc1
V  [libjvm.so+0x10fc4e6]  vframeArray*vframeArray::allocate(JavaThread*,int,GrowableArray<compiledVFrame*>*,RegisterMap*,frame,frame,frame)+0x116
V  [libjvm.so+0x75a804]  vframeArray*Deoptimization::create_vframeArray(JavaThread*,frame,RegisterMap*,GrowableArray<compiledVFrame*>*)+0x484
V  [libjvm.so+0x755775]  Deoptimization::UnrollBlock*Deoptimization::fetch_unroll_info_helper(JavaThread*)+0xd35
V  [libjvm.so+0x75e4dc]  Deoptimization::UnrollBlock*Deoptimization::uncommon_trap(JavaThread*,int)+0x2c

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
v  blob 0xf96aaae3
J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List; @ 0xf96f6054 [0xf96f3e00+8788]
j  LoopUnswitchBugTest$1.run()V+53
j  java.util.concurrent.Executors$RunnableAdapter.call()Ljava/lang/Object;+4
j  java.util.concurrent.FutureTask.run()V+42
j  java.util.concurrent.ThreadPoolExecutor.runWorker(Ljava/util/concurrent/ThreadPoolExecutor$Worker;)V+95
j  java.util.concurrent.ThreadPoolExecutor$Worker.run()V+5
j  java.lang.Thread.run()V+11
v  ~StubRoutines::call_stub


                                     
2013-07-31
release-note-YES Justification:  We should mention that the user should use -XX:-LoopUnswitching.
                                     
2013-07-31
Adding test and .jar files needed to reproduce the problem.
                                     
2013-07-31
-XX:InlineSmallCode=2000 -XX:-CheckCompressedOops flags may increase frequency of the failure with debug VM.
                                     
2013-07-31
Also with 7u40:

$ cat hs_err_pid18115.log
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfffffd7ff8e84216, pid=18115, tid=107
#
# JRE version: Java(TM) SE Runtime Environment (7.0_40-b35) (build 1.7.0_40-ea-b35)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (24.0-b54 mixed mode solaris-amd64 compressed oops)
# Problematic frame:
# J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;
#
# Failed to write core dump. Core dumps have been disabled. To enable core dumping, try "ulimit -c unlimited" before starting Java again
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000805800):  JavaThread "pool-1-thread-44" [_thread_in_Java, id=107, stack(0xfffffd7fe2345000,0xfffffd7fe2445000)]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=2 (SEGV_ACCERR), si_addr=0x0000000700ebdf48

Registers:
RAX=0x00000003fc2cd180, RBX=0x0000000000000000, RCX=0x0000000000000000, RDX=0x0000000000000001
RSP=0xfffffd7fe2444580, RBP=0x00000006acaedf18, RSI=0x00000003fc2c8398, RDI=0x00000003fc2c8394
R8 =0x0000000000000000, R9 =0x00000000e01d7be8, R10=0x0000000000000000, R11=0x0000000000000002
R12=0x0000000000000000, R13=0x0000000000000001, R14=0x0000000000000004, R15=0x0000000000805800
RIP=0xfffffd7ff8e84216, RFLAGS=0x0000000000010203

Top of Stack: (sp=0xfffffd7fe2444580)
0xfffffd7fe2444580:   00000006acaec1b8 0000000500000005
0xfffffd7fe2444590:   00000006bb1583e0 0000000ad595d8f4
0xfffffd7fe24445a0:   fffffd7fd674e754 00000006acaedee8
0xfffffd7fe24445b0:   00000006acaedee8 00000006acaed948
0xfffffd7fe24445c0:   00000006acaed948 00000006acaec1b8
0xfffffd7fe24445d0:   fffffd7fe2444620 00000003fc2c87b0
0xfffffd7fe24445e0:   00000003fc2e5160 00000003fc2c6c78
0xfffffd7fe24445f0:   0000000000000000 fffffd7fe2444610
0xfffffd7fe2444600:   fffffd7fe2444670 fffffd7ff8e06823
0xfffffd7fe2444610:   00000006acaedc68 fffffd7ff8e06823
0xfffffd7fe2444620:   00000006acaec1b8 00000006bb1583e0
0xfffffd7fe2444630:   fffffd7fe2444630 00000003fc2be6fd
0xfffffd7fe2444640:   fffffd7fe2444698 00000003fc2be9f8
0xfffffd7fe2444650:   00000003fc2e3928 00000003fc2be710
0xfffffd7fe2444660:   fffffd7fe2444620 fffffd7fe2444698
0xfffffd7fe2444670:   fffffd7fe24446e0 fffffd7ff8e069fe
0xfffffd7fe2444680:   00000006acaed948 0000000000000005
0xfffffd7fe2444690:   00000006acaec1b8 00000006ab3477f8
0xfffffd7fe24446a0:   fffffd7fe24446a0 00000003fc2c28f4
0xfffffd7fe24446b0:   fffffd7fe24446f0 00000003fc2c2c10
0xfffffd7fe24446c0:   00000003fc2e8340 00000003fc2c2908
0xfffffd7fe24446d0:   fffffd7fe2444698 fffffd7fe24446f0
0xfffffd7fe24446e0:   fffffd7fe2444738 fffffd7ff8e06823
0xfffffd7fe24446f0:   00000006ab347828 fffffd7fe24446f8
0xfffffd7fe2444700:   00000003fc2c0b7a fffffd7fe2444778
0xfffffd7fe2444710:   00000003fc2c1770 00000003fc2e8590
0xfffffd7fe2444720:   00000003fc2c0c08 fffffd7fe24446f0
0xfffffd7fe2444730:   fffffd7fe2444778 fffffd7fe24447c0
0xfffffd7fe2444740:   fffffd7ff8e069fe 0000000000000000
0xfffffd7fe2444750:   0000000000000000 0000000000000000
0xfffffd7fe2444760:   0000000000000000 0000000000000000
0xfffffd7fe2444770:   00000006ab347828 00000006ab347808 

Instructions: (pc=0xfffffd7ff8e84216)
0xfffffd7ff8e841f6:   41 3b d6 7c 09 8b da e9 c3 00 00 00 8b d3 89 55
0xfffffd7ff8e84206:   10 44 8b 4c 97 10 44 8b da 41 ff c3 44 89 5d 0c
0xfffffd7ff8e84216:   43 8b 5c cc 08 81 fb 73 90 85 7f 0f 85 b5 14 00
0xfffffd7ff8e84226:   00 44 89 5d 10 33 db 50 48 8b c0 48 8b 7e 30 8b 

Register to memory mapping:

RAX=0x00000003fc2cd180 is an oop
{instance class} 
 - klass: {other class}
RBX=0x0000000000000000 is an unknown value
RCX=0x0000000000000000 is an unknown value
RDX=0x0000000000000001 is an unknown value
RSP=0xfffffd7fe2444580 is pointing into the stack for thread: 0x0000000000805800
RBP=0x00000006acaedf18 is an oop
java.util.ArrayList$Itr 
 - klass: 'java/util/ArrayList$Itr'
RSI=0x00000003fc2c8398 is an oop
{instance class} 
 - klass: {other class}
RDI=0x00000003fc2c8394 is pointing into object: 0x00000003fc2c8370
[[I 
 - klass: 'java/lang/Object'[]
 - length: 5
R8 =0x0000000000000000 is an unknown value
R9 =0x00000000e01d7be8 is an unknown value
R10=0x0000000000000000 is an unknown value
R11=0x0000000000000002 is an unknown value
R12=0x0000000000000000 is an unknown value
R13=0x0000000000000001 is an unknown value
R14=0x0000000000000004 is an unknown value
R15=0x0000000000805800 is a thread


Stack: [0xfffffd7fe2345000,0xfffffd7fe2445000],  sp=0xfffffd7fe2444580,  free space=1021k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
J  org.apache.http.impl.cookie.BestMatchSpec.formatCookies(Ljava/util/List;)Ljava/util/List;


                                     
2013-07-31
Moving from hotspot/runtime -> hotspot/compiler for initial eval.
                                     
2013-07-30
Both links show that they know about workaround: -XX:-LoopUnswitching.

https://github.com/rholder/jvm-loop-unswitching-bug 
https://issues.apache.org/jira/browse/HTTPCLIENT-1173

Also they know about this for long time. Apache bug shows that the issue was reported on 14/Mar/12 and workaround was found on 14/Feb/13.

I will try to find the cause and fix it in the time we have before 7u40.

                                     
2013-07-30
7u40-defer-request Justification:  There is a workaround for this, requesting a deferral.
                                     
2013-07-30



Hardware and Software, Engineered to Work Together