Bug ID: JDK-8324743 C2 can't compile method with nested locks

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

Other
tbdUnresolved

There are 2 problems: broken bailout message and C2 can't compile.

JDK 22 and latest 23 produces strange message:

java -XX:CICompilerCount=1 -XX:-TieredCompilation X:+PrintCompilation -Xbatch -Xmx128m TestNestedLocks
154    1    b        java.util.ArrayList::add (25 bytes)
159    2    b        java.util.ArrayList::add (23 bytes)
162    3   !b        TestNestedLocks::test (102 bytes)
162    3   !b        TestNestedLocks::test (102 bytes)   COMPILE SKIPPED: ??????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????????
182    4 % !b        TestNestedLocks::main @ 18 (39 bytes)
184    5   !b        TestNestedLocks::main (39 bytes)
637    4 % !         TestNestedLocks::main @ 18 (39 bytes)   made not entrant
OOM caught in test

It looks like something was broken recently with bailout messages.

JDK 11u produces:

    259    4   !b        TestNestedLocks::test (102 bytes)   COMPILE SKIPPED: cannot parse method (not retryable)

JDK 21 has more detailed info:

    426    3   !b        TestNestedLocks::test (102 bytes)   COMPILE SKIPPED: cannot parse method: not compilable (unbalanced monitors) (retry at different tier)

We need to fix message in JDK 22 and 23.

And find why C2 think it is unbalanced.

Actually C1 is also can't compile it:
     28    6   !b  1       TestNestedLocks::test (102 bytes)   COMPILE SKIPPED: invalid parsing (retry at different tier)

It works fine if we put inner 2 locks into separate method and inline it.

Just a cross reference. [~stefank] noted this while working on the legacy locking vs. lightweight locking regressions. https://bugs.openjdk.org/browse/JDK-8316921?focusedId=14619546&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-14619546
27-02-2024
Converted to RFE.
01-02-2024
Just filed JDK-8325095, to deal with the broken bailout message. Do we convert this issue here to an RFE, since it is intended behaviour, given the comments?
01-02-2024
I suggest to file separate bug to fix bailout message and keep this bug for compilation failure.
31-01-2024
There are two problems here. The first is that GenerateOopMap::do_monitorenter() does not handle recursive locking. This has also been noticed by [~coleenp] and [~stefank] recently. The comment says: // Bail out when we get repeated locks on an identical monitor. This case // isn't too hard to handle and can be made to work if supporting nested // redundant synchronized statements becomes a priority. // // See also "Note" in do_monitorexit(), below. We can file a separate RFE if we decide to fix this. The weird bailout message seems to be a regression from JDK-8303951. It does record_method_not_compilable(ss.as_string()); which means the string is resource-allocated. Then when CompileBroker tries to access the string again through the ci_env, it's not clear the memory is still live. [~epeter] could you please take a look at this?
27-01-2024
ILW = Broken bailout messages and unable to C2 compile, single test, no workaround = MLH = P4
26-01-2024

Relates :	JDK-8303951 - Add asserts before record_method_not_compilable where possible
Relates :	JDK-8325095 - C2: bailout message broken: ResourceArea allocated string used after free