JDK-8257594 : C2 compiled checkcast of non-null object triggers endless deoptimization/recompilation cycle
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 11,16
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2020-12-02
  • Updated: 2021-01-06
  • Resolved: 2020-12-03
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 16
11.0.11-oracleFixed 16 b28Fixed
Description
When C2 determines at compile time that there is no subtype relation between the objects involved in a checkcast, it inserts a null assert because the checkcast can only pass if the value is null. This will trigger deoptimization and recompilation if the object is non-null:

   1151  636   !   4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   made not entrant
   1151  638   !b  4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
   1152  638   !   4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   made not entrant
   1152  639   !b  4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
   1152  639   !   4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   made not entrant
   1153  640   !b  4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)

[...]

   1475 1030   !   4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   made not entrant
   1475 1031   !b  4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)
made not compilable on level 4  compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   decompile_count > PerMethodRecompilationCutoff
   1476 1031   !   4       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)   made not entrant
   1476 1032   !b  1       compiler.uncommontrap.TestNullAssertAtCheckCast::test (17 bytes)

However, we can do better here if we already know that the object is non-null (i.e. the check always fails) or if we hit too_many_traps_or_recompiles (which suggests that the object is often null) to avoid an endless deoptimization/recompilation cycle until we hit the PerMethodRecompilationCutoff and starve at lower tiers.

In the former case we can simply throw right away and in the latter case we should fall back to a full check in case there is an exception handler in compiled code that handles the failing (non-null) case. That way we avoid deoptimization.

I've spotted this with our test framework for inline types in project Valhalla, while investigating an intermittent failure where several test methods suddenly became non-compilable at tier 4.
Comments
Fix Request (11u) This backports the fix for C2 peformance problem, and keeps codebases in sync (I see 11.0.11-oracle). Patch applies cleanly to 11u, passes tier{1,2} tests. New test fails without the VM changes, passes with them.
04-01-2021

Changeset: 129c3770 Author: Tobias Hartmann <thartmann@openjdk.org> Date: 2020-12-03 13:42:19 +0000 URL: https://git.openjdk.java.net/jdk/commit/129c3770
03-12-2020

ILW = Repeated deoptimization -> recompilation cycle leads to performance drop, edge case but easy to reproduce, no workaround but disable compilation of affected method = MMM = P3
02-12-2020