A java thread T potentially continues execution after it was suspended by a JVMTI agent during the
attempt to process a handshake operation. This can lead to undefined behaviour and crashes.
Problematic execution sequence:
- Handshake operation H is executed for java thread T
- T calls ThreadSafepointState::handle_polling_page_exception()
- T arrives in HandshakeState::process_self_inner()
- T transitions from _thread_in_Java to _thread_in_vm
- but too late: the VM thread already claimed H
- T calls _semaphore.wait_with_safepoint_check()
- T transitions from _thread_in_vm to _thread_blocked
- The VM thread completes H and calls _semaphore.signal()
- Before T can transition back to _thread_in_vm, the VM thread schedules another safepoint and
executes VM_ThreadSuspend on behalf of a JVMTI agent that wants to suspend T
- After the safepoint T transitions back to _thread_in_vm and then to _thread_in_Java.
- T continues executing an undefined number of bytecodes ...
Similar if T returns from a native method call.
The VM could crash because of this, e.g. because T's stack is not walkable after the suspend.