JDK-8160651 : StubRoutines::_dtan does not restore callee save register rbx
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 9
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • CPU: x86
  • Submitted: 2016-06-30
  • Updated: 2017-08-07
  • Resolved: 2016-07-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 9
9 b131Fixed
Related Reports
Duplicate :  
Duplicate :  
Relates :  
Description
StubRoutines::_dtan overwrites the callee save register $rbx but does not restore the original value before returning. 

In the caller method we execute: 
 0x7fe406be496f: mov %rsi,%rbx // Save 'this' to $rbx 
 0x7fe406be4972: mov %rdx,%r13 
 0x7fe406be4975: vmovq %rdx,%xmm0 
 0x7fe406be497a: movabs $0x7fe405003c7b,%r10 
 0x7fe406be4984: callq *%r10 // Call StubRoutines::_dtan (overwrites rbx with 0x8000) 
 0x7fe406be4987: vucomisd %xmm0,%xmm0 
 0x7fe406be498b: jp 0x7fe406be498f 
 0x7fe406be498d: je 0x7fe406be49c0 
 0x7fe406be498f: movabs $0x7ff8000000000000,%r9 
 0x7fe406be4999: movabs $0x71de35b60,%rdx 
 0x7fe406be49a3: mov %rbx,%rsi // Restore 'this' 
 0x7fe406be49a6: mov %r13,%rcx 
 0x7fe406be49a9: mov %rbp,%r8 
 0x7fe406be49ac: nop 
 0x7fe406be49ad: nop 
 0x7fe406be49ae: nop 
 0x7fe406be49af: callq 0x7fe40661dea0 // Call test_engine.print_result 

We crash in test_engine.print_result() while trying to increment an integer field because 'this' is invalid. 

The problem is in MacroAssembler::fast_tan() introduced by JDK-8152907: 
http://hg.openjdk.java.net/jdk9/hs-comp/hotspot/rev/33f10a35ce20#l16.1052 

We miss a 'pop(rbx);' at the end of the stub.
Comments
Verified by manual testing.
07-08-2017

Fix: http://cr.openjdk.java.net/~thartmann/8160651/webrev.00/
30-06-2016

ILW = Crash due to invalid register contents, only with Math.tan intrinsic, disable intrinsic = HML = P2
30-06-2016