JDK-8270010 : Handling of failed reallocation of scalar-replaced objects skips some exception handlers
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 8u40,11,16,17,18
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: generic
  • CPU: generic
  • Submitted: 2021-07-07
  • Updated: 2022-10-07
  • Resolved: 2022-03-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 19
19Resolved
Related Reports
Duplicate :  
JDK-8227309 - Handle OOM when rematerializing objects less surprising
Relates :  
JDK-6898462 - The escape analysis with G1 cause crash assertion src/share/vm/runtime/vframeArray.cpp:94
Relates :  
JDK-8227309 - Handle OOM when rematerializing objects less surprising
Description
When an nmethod is deoptimized, its scalar-replaced objects are reallocated on the heap. When this reallocation fails due to lack of memory, an OOME is raised, and the nmethod's corresponding interpreter frames are popped, and their held locks are released, but *without* running their exception handlers. Because we allow recovery from OOME, execution may resume in some caller, in apparent violation of ยง14.20 of the JLS.

This behavior was introduced by JDK-6898462.

A possible solution might be to pop *all* frames in the thread and to trigger the thread's uncaught exception handler with the OOME. A mechanism for freeing native resources might be needed as well.

Another possible solution is to ensure this situation never arises by reserving heap space in some emergency region that would ensure, on nmethod-entry, that there will be enough heap memory in the case of reallocation.
Comments
Checked with [~neliasso]: > Yes, we can merge them, and when we have something in the future, we can create subtask if we need to address them differently Closing as duplicate of JDK-8227309.
01-03-2022
This sounds like a duplicate of JDK-8227309.
06-08-2021
So this is an old super annoying issue that has been successfully dodged over the years. It has not been solved because it is rather difficult to solve. In the solution domain, here is an idea for a solution that I believe might actually work, after discussions with [~fparain] and [~thartmann]. Annoying to do, but just might work. The idea is to have a two step fallback mode for scalarized objects. First scalarize them, speculatively hoping they won't need to materialize. When materializing them, speculatively hope they can be allocated in the heap (*almost* always works). Failure of materializing speculatively scalarized objects, is partially dealt with for various parts of the VM with the EscapeBarrier work already, dealing with serviceability APIs where they can't be materialized again. At this particular point however, where we are in the deopt handler and entering a frame whose scalarized objects can't be materialized on the heap, we would instead materialize all scalarized objects on the stack and enter a degenerated mode stuck in the interpreter, as long as there are stack allocated objects around (determined with a thread-local counter, counted up when such frames are created due to deopt, and decremented when returning from such frames). We would speculate that the scalarized objects are stack local, until an interpreter store publishes a stack object to a heap object. Such stores would throw out of memory exception. This is rather similar to partial escape analysis in spirit; assume the objects are local and non-escaping until proven otherwise. And at the point where it escapes (heap to stack store), try materializing again, or throw OOM if that failed. The main point of this dance would be to move the problem to a place where we are actually allowed to throw. Moreover, it would solve probably most cases where something unexpected happens to the user. It's a not totally uncommon pattern for example for GC tests to catch OOM and recover. Such code today has workarounds due to this unsound behaviour. When stack banging a compiled frame, we know the size of the interpreter frames, and also the size of the scalarized objects. So we can make materialization on the stack always succeed, however be less efficient as it forces interpreted execution to stick around until we can get rid of the stack object, which we really don't want to pass around in compiled methods. This would require "heap" accessing bytecodes in the interpreter to check if the base is stack allocated, and then not run any IN_HEAP barriers. It would also require changing the layout of interpreted frames to accomodate stack allocated objects. I expect this would be a bit messy and complicated, but actually possible to fix, if we want to fix this. I think we do want to fix this. Frame root parsing code would also need to identify such frames with stack objects, and iterate over their oops, that are now roots. Serviceability code that expose objects that might not be heap materialized currently use the EscapeBarrier mechanism. This mechanism would be extended to materialize stack allocated objects in the heap as well. Annoying to implement (like most things in the interpreter), but solves an annoying corner case that has been a known issue since a very long time ago, with impact hopefully contained mostly in the interpreter code.
19-07-2021
ILW = Incorrect behavior of deoptimized code after OOME exception, never observed in testing, run with large enough memory = HLM = P3
07-07-2021