Hello,

I spent yesterday tracing a bug in our software, and the trail points to
Hotspot.

Extremely short version:

cadd_cmpLTMask doesnt handle 64-bit register as tmp, and generates
subtly broken code if it receives one.

Slightly longer version:

After extensive inlining and elision of the full-check, the interesting
part of

@Override
public int size() {
   if (full) {
       path = 0;
       return capacity();
   }

   final int result;
   if (last >= first) {
           result = last - first;
   } else {
           result = last - first + capacity();
   }
   return result;
}

compiles down to something like

11d     subl    R10, RDX        # cadd_cmpLTMask1
          sbbl    R11, R11
          andl    R11, RBX
          addl    R10, R11
127     movl    [rsp + #40], R10        # spill

where initially R10 is last, RBX is first, R8 is whatever and RCX is
result of capacity(). However, looking at the code that is actually run
with GDB, the disassembly is rather different:

0x2aaaac856d7d:    sub    %edx,%r10d
0x2aaaac856d80:    sbb    %ebx,%ebx
0x2aaaac856d82:    and    %ebx,%ebx
0x2aaaac856d84:    add    %ebx,%r11d
0x2aaaac856d87:    mov    %r10d,0x28(%rsp)

Both and and add operate on odd registers, and the code also produces
wrong result into [RSP+40]. Looking at the binary code, it's as follows:

(gdb) x/15xb 0x2aaaac856d7d
0x2aaaac856d7d:    0x44    0x2b    0xd2    0x1b    0xdb    0x23   
0xdb    0x44
0x2aaaac856d85:    0x03    0xdb    0x44    0x89    0x54    0x24    0x28

Notably the SBB instruction is missing REX.RB prefix, which changes it
from SBBL R11 R11 into SBBL EBX EBX and clobbers the actual EBX. Also,
the prefix of the ADD is only REX.R, whereas it probably should be
REX.RB, and in any case the destination register is wrong.

Looking at x86_64.ad, it's clear that the code that generates this
section (cadd_cmpLTMask) is not prepared to handle 64-bit tmp register.
This explain missing prefixes, and also wrong destination register of
ADD, since the high bit of reg clobbers the low bit or r/m in emit_rm().

Now, where do I report this bug? Presumably there is a bug tracker
somewhere.

-- 

Tuure Laurinolli
Software Engineer
Indagon Ltd.