Bug ID: JDK-8325003 Incorrect OOM thrown from the UncaughtExceptionHandler after deoptimization

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 23

Priority: P4
Status: Open
Resolution: Unresolved

Submitted: 2024-01-30
Updated: 2024-07-10

Other
tbdUnresolved

When running new test TestNestedRelockAtDeopt.java from JDK-8324174 with default flags, it exits early with OOM exception thrown:

> java -Xmx128m TestNestedRelockAtDeopt
OOM caught in test1
Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"

It is incorrect since the test should catch all OOM exception:

> java -Xmx128m -Xint TestNestedRelockAtDeopt
OOM caught in test1
OOM caught in test2
OOM caught in test3
OOM caught in test4

OOM could be thrown during deoptimization when scalarized object is re-allocated and re-locked. But we should handle such case gracefully after fix JDK-6898462.

Other JDK releases may be also affected but without JDK-8324174 fix it is hard to tell.

ILW = Incorrectly throwing OOM exception, edge case, possibly use -XX:-DoEscapeAnalysis = MLM = P4
05-02-2024
Based on performance data this change is NO GO. SPECjvm2008-MonteCarlo regressed about 75% with Parallel GC on x64 and aarch64. And 60% with G1 (default) GC.
01-02-2024
I am working on disabling EA if bytecode has `catch(SomeException e)` by checking `if (exc_table.catch_type_index(i) > 0) ` in ciMethod. I don't know other way to avoid skipping such `catch` during deoptimization. It may be too conservative (try/catch blocks could be not related to scalarized allocation) but I think it is more safe. I will run performance testing.
01-02-2024
On other hand if OOM is caught by `catch` then following NPE could be expected. May be we should not scalarize object if there is try/catch in compiled code. Because we can NOT handle it correctly during deoptimization. If code (including inlined method) have try/catch/final and object is referenced by safepoint (in debuginfo) do not scalarize it. If we follow it then current code in deoptimize.cpp will correct work - it can pop all frames since there will be no try/catch/final there.
31-01-2024
After more thinking popping only part of frames could be unsafe if scalarized object is returned from inlined methods: static void test() { Test o = null; try { o = test1(); } catch (OutOfMemoryError oom) { print(); } if (o.f) foo(); // `o` will be scalarized when test1() is inlined } static Test test1() { Test o = new Test(); synchronized (o) { arr.add(new byte[CHUNK]); } return o; }
31-01-2024
I think VM tries to print calls stacks and fail again and again because it does not have heap memory for it. Nothing I can do about it. What I can do for this bug is to track which frames have failed re-allocations and pop only them. And do unlocking also only for them. It is in hope that caller frame can free resources and catch exception. I am not sure what to do for next case. It requires bytecode analysis to see if we can execute `catch`: static void test() { synchronized (new TestNestedLockEliminatedAtDeopt()) { try { synchronized (new TestNestedLockEliminatedAtDeopt()) { arr.add(new byte[CHUNK]); } // unlock } catch (OutOfMemoryError oom) { arr = null; print(); } } // unlock } Without outer `synchronized` it is safe to jump to `catch`. With outer `synchronized` it is not. May be it is safer to pop whole frame.
31-01-2024
Without `arr = null; ` I got `java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main"` because there was no memory to print call stack: [4.027s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b508}: Java heap space: failed reallocation of scalar replaced objects> thrown in interpreter method <{method} {0x000000011b4005e0} 'main' '([Ljava/lang/String;)V' in 'TestNestedLockEliminatedAtDeopt'> at bci 0 for thread 0x00007fac79820e10 (main) [4.027s][info][exceptions] Found matching handler for exception of type "java.lang.OutOfMemoryError" in method "main" at BCI: 6 [5.271s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b230}> (0x00000007f957b230) thrown [/Users/vkozlov/work/leyden/src/hotspot/share/gc/shared/memAllocator.cpp, line 137] for thread 0x00007fac79820e10 [5.271s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b230}: Java heap space> thrown in interpreter method <{method} {0x000000011b4005e0} 'main' '([Ljava/lang/String;)V' in 'TestNestedLockEliminatedAtDeopt'> at bci 14 for thread 0x00007fac79820e10 (main) [6.488s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b520}> (0x00000007f957b520) thrown [/Users/vkozlov/work/leyden/src/hotspot/share/gc/shared/memAllocator.cpp, line 137] for thread 0x00007fac79820e10 [6.488s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b520}: Java heap space> thrown in interpreter method <{method} {0x000000080002c768} 'uncaughtException' '(Ljava/lang/Thread;Ljava/lang/Throwable;)V' in 'java/lang/ThreadGroup'> at bci 41 for thread 0x00007fac79820e10 (main) [6.488s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b520}: Java heap space> thrown in interpreter method <{method} {0x000000080002c768} 'uncaughtException' '(Ljava/lang/Thread;Ljava/lang/Throwable;)V' in 'java/lang/ThreadGroup'> at bci 13 for thread 0x00007fac79820e10 (main) [6.488s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b520}: Java heap space> thrown in interpreter method <{method} {0x0000000800012ae0} 'dispatchUncaughtException' '(Ljava/lang/Throwable;)V' in 'java/lang/Thread'> at bci 6 for thread 0x00007fac79820e10 (main) Exception: java.lang.OutOfMemoryError thrown from the UncaughtExceptionHandler in thread "main" [7.726s][info][exceptions] Exception <a 'java/lang/OutOfMemoryError'{0x00000007f957b248}> (0x00000007f957b248) thrown [/Users/vkozlov/work/leyden/src/hotspot/share/gc/shared/memAllocator.cpp, line 137] for thread 0x00007fac5980ea10
31-01-2024
After adding catch to main() and freeing memory there I got correct output: public static void main(String[] args) { try { test(); } catch (Error ex) { arr = null; // Free memory System.out.println("Exception caught in main: " + ex.getClass().getCanonicalName()); ex.printStackTrace(); } } Exception caught in main: java.lang.OutOfMemoryError java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects at TestNestedLockEliminatedAtDeopt.test(TestNestedLockEliminatedAtDeopt.java:27) at TestNestedLockEliminatedAtDeopt.main(TestNestedLockEliminatedAtDeopt.java:12)
31-01-2024
It fail to catch when we have only one method with try/catch and synchronization because we popped this frame: static void test() { try { synchronized (new TestNestedLockEliminatedAtDeopt()) { arr.add(new byte[CHUNK]); } } catch (OutOfMemoryError oom) { print(); } }
31-01-2024
It is from original changes JDK-6898462: https://github.com/openjdk/jdk/blob/master/src/hotspot/share/runtime/deoptimization.cpp#L1717
31-01-2024
We pop too many frames when returning into Interpreter. We pop all frames in compiled method which may inline other method. We should pop only frames where we can't re-allocate scalarized objects. In next test we pop both Interpreter frames when test1() is inlined and as result we missed `catch`: static void test() { try { test1(); } catch (OutOfMemoryError oom) { print(); } } static void test1() { synchronized (new TestNestedLockEliminatedAtDeopt()) { arr.add(new byte[CHUNK]); } }
31-01-2024
OOM is thrown by deoptimizer during failed re-allocaiton. It should be "failed reallocation of scalar replaced objects" as Dean pointed. But it was not caught. If you move code from main() to new test_main() method it will fail on the first test. Frames are correctly deoptimized to bytecode " 42 newarray byte". So Interpreter should catch it (why it works with -XX:-TierdCompialtion?). Continue investigation. May be JDK-6898462 does not work for this case.
31-01-2024
I agree, if the OOM being thrown is "java.lang.OutOfMemoryError: Java heap space: failed reallocation of scalar replaced objects", then it sounds like a test bug.
31-01-2024
Are we sure this is not simply a test bug? The test fills up the Java heap, then catches the OOM and continues. From that point on, anything can happen, especially hitting another OOM just below. I added a comment to https://github.com/openjdk/jdk/pull/17600
31-01-2024