FULL PRODUCT VERSION :
This error is in the Hotspot 1.5 and 1.6, as far as I see in the sources.

FULL OS VERSION :
Microsoft Windows [Version 5.2.3790]
x64 (amd64)


A DESCRIPTION OF THE PROBLEM :
This bug occurs in SAP's Java VM 5, which is based on the 1.5 Hotspot sources.

We found a bug which caused under rare circumstances a crash on Windows AMD64. In theory, this bug might happen on all AMD64 platforms. It also seems to affect 1.6 and 1.7 as well, according to the sources.

----

The following explanation and the source line numbers refer to cpu/amd64/runtime_amd64.cpp as of update release 13.

The offending function is

void SharedRuntime::generate_deopt_blob().


That function generates a CodeBlob which handles deoptimization. Inside that function (lines 395ff) Interpreter frames are pushed in a loop onto the stack.

Then, after repushing the deopt frame itself, the function Deoptimization::unpack_frames() is called (line 432). Deoptimization::unpack_frames() is a "normal" C++ function, not generated at runtime.

The problem is that at the point of that call, the stackpointer is not guaranteed to be 16 byte aligned. The alignment is pretty random - depending on the number of interpreter frames pushed and of their size.
This violates the ABI for Windows x64, but normally does not cause an error, only when 128bit instructions are used.

This exactly happened at SAP. We did add some tracing code to Deoptimization::unpack_frames() and recompiled, and the C++ compiler decided as part of an optimization to
use SSE instructions. In particular it generated code like this:

movaps [rsp + ...0h], xmm0

which relies on rsp being 16 byte aligned. This was not the case and the VM crashed.

We think that if C++ code gets called from within generated code, it should follow the ABI exactly.

----

I fixed the problem in our own (SAPs) VM by nesting the call to Deoptimization::unpack_frames() into a little wrapper subroutine, where I align the stackpointer before the call into the runtime.
If you are interested in this fix, we can give it to you (I did not find any way to submit a bug fix to 1.5 on your website).


THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: No

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Just debug the VM and check the stackpointer inside Deoptimization::unpack_frames. Sometimes it will be 16-byte-aligned, sometimes not. If not, the code under it might crash.

Also, you could simply emit an SSE instruction just before the call to Deoptimization::unpack_frames(). Something like this:

; make room for one xmm register while preserving alignment
sub  rsp, 0x10

; next instruction will crash if done with unaligned rsp:
movaps  [rsp], xmm0

; restore stack pointer
add rsp, 0x10

This will crash very soon, proving that alignment is random at this point.

REPRODUCIBILITY :
This bug can be reproduced rarely.