JDK-8339526 : JVM crashes (after some warm-up phase) running Java 21 (and above)
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 21.0.4-oracle,22,23,24
  • Priority: P3
  • Status: Closed
  • Resolution: Incomplete
  • OS: windows
  • CPU: x86_64
  • Submitted: 2024-08-26
  • Updated: 2024-11-18
  • Resolved: 2024-11-18
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 24
24Resolved
Related Reports
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Windows Server and Windows 11 / Desktop (both 64 bit) running Java 21 (and above, but not below) using the JVMs from Oracle (and other vendors, like amazon or adoptium).

A DESCRIPTION OF THE PROBLEM :
JVM crashes (after some warm-up phase) running Java 21 (and above, but not below) using the JVMs from Oracle (and other vendors, like amazon or adoptium).
The code to reproduce the issue is non-shareable / closed-source (sorry for that), but it could quite stably reproduce the issue locally (server and desktop – using windows 64 bit).
The application is a webapp using tomcat application-server and the triggering “test” is single-threaded and is doing some SQL changes (including string creation for the prepared statements) for “translations” in a loop.
After a while (some hundred iterations running stable) it looks like hitting a C2 (hotspot) issue, because GraalVM (using a different C2) is working and non-working versions (like 21.0.4) are working (quite slow), if C2 is disabled (using -XX:TieredStopAtLevel=3).
Another important thing to know is, that the issue is aligned to the disabled or not available “compressed oops” mode (System.getProperty("java.vm.compressedOopsMode") returns null). 
This enables a workaround, to configure a max heap size 32256m (and below – for Java21) or using -XX:ObjectAlignmentInBytes=16/32… (to use more memory with “compressed oops”).

Thank you very much for checking and hopefully fixing the issue. 
All you hard work is very much appreciated! 
Btw. this is my first reported crash, so the JVM normally runs perfect.
In addition, I know, that the C2 optimizations are quite hard to maintain (a reason for the new GraalVM).

REGRESSION : Last worked in version 11.0.24

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
The code to reproduce the issue is non-shareable / closed source (sorry for that), but it could quite stably reproduce the issue locally (server and desktop – using windows 64 bit).
The application is a webapp using tomcat application-server and the triggering “test” is single-threaded and is doing some SQL changes (including string creation for the prepared statements) for “translations” in a loop.

ACTUAL -
Crashes (see “hs_err_pid files”) and “strange” exceptions (after a warm-up phase and right before the crash, where magically something got null), like:

java.lang.StringIndexOutOfBoundsException: Range [67, 67 + 0) out of bounds for length 48
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:55) ~[?:?]
	at java.base/jdk.internal.util.Preconditions$1.apply(Preconditions.java:52) ~[?:?]
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:213) ~[?:?]
	at java.base/jdk.internal.util.Preconditions$4.apply(Preconditions.java:210) ~[?:?]
	at java.base/jdk.internal.util.Preconditions.outOfBounds(Preconditions.java:98) ~[?:?]
	at java.base/jdk.internal.util.Preconditions.outOfBoundsCheckFromIndexSize(Preconditions.java:118) ~[?:?]
	at java.base/jdk.internal.util.Preconditions.checkFromIndexSize(Preconditions.java:397) ~[?:?]
	at java.base/java.lang.String.checkBoundsOffCount(String.java:4853) ~[?:?]
	at java.base/java.lang.StringUTF16.checkBoundsOffCount(StringUTF16.java:1625) ~[?:?]
	at java.base/java.lang.StringUTF16.inflate(StringUTF16.java:1459) ~[?:?]
	at java.base/java.lang.StringLatin1.inflate(StringLatin1.java:771) ~[?:?]
	at java.base/java.lang.String.getBytes(String.java:4728) ~[?:?]
	at java.base/java.lang.StringConcatHelper.prepend(StringConcatHelper.java:324) ~[?:?]
	at java.base/java.lang.StringConcatHelper.prepend(StringConcatHelper.java:343) ~[?:?]

java.lang.NullPointerException: Cannot invoke "Object.toString()" because "value" is null
	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:467) ~[?:?]
	at com.webfair.domain.dynamicresources.TranslationItem.toString(TranslationItem.java:77) ~[launchnet-backend.jar:?]
	at java.base/java.lang.String.valueOf(String.java:4465) ~[?:?]
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:173) ~[?:?]
	at java.base/java.util.AbstractCollection.toString(AbstractCollection.java:459) ~[?:?]
	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:467) ~[?:?]
	at java.base/java.lang.StringConcatHelper.simpleConcat(StringConcatHelper.java:422) ~[?:?]

The TranslationItem.toString method is a simple autogenerated toString method using fixed strings and basic (int, String, Integer) properties.

java.lang.NullPointerException: Cannot read the array length because "this.value" is null
	at java.base/java.lang.String.length(String.java:1519) ~[?:?]
	at java.base/java.lang.StringConcatHelper.mix(StringConcatHelper.java:123) ~[?:?]
	at com.webfair.domain.dynamicresources.TranslationItem.toString(TranslationItem.java:77) ~[launchnet-backend.jar:?]
	at java.base/java.lang.String.valueOf(String.java:4465) ~[?:?]
	at java.base/java.lang.StringBuilder.append(StringBuilder.java:173) ~[?:?]
	at java.base/java.util.AbstractCollection.toString(AbstractCollection.java:459) ~[?:?]
	at java.base/java.lang.StringConcatHelper.stringOf(StringConcatHelper.java:467) ~[?:?]
	at java.base/java.lang.StringConcatHelper.simpleConcat(StringConcatHelper.java:422) ~[?:?]

All were using Java 21.0.4 (to find the lines in the source correctly), but there are similar issues in other non-working versions too.

---------- BEGIN SOURCE ----------
Sorry code to trigger the issue is non-shareable / closed source, so sharing the “test case” code (isolated) would make no sense.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
- Use „older“ Java version < 21
- Use GraalVM (even with Java21)
- Disable C2 (using -XX:TieredStopAtLevel=3) even if it’s then quite slow (and may not acceptable)
- Keep “compressed oops” mode active (System.getProperty("java.vm.compressedOopsMode") returns not null)

FREQUENCY : often



Comments
Okay, closing as Incomplete.
18-11-2024

[~pnarayanaswa] Any further information from the reporter?
12-11-2024

Thanks for the additional information. > How to get and use a "debug build"? I don't think we provide debug builds for the Oracle JDK, so the only way would be to build the JDK yourself. > Do you have more options to check? Unfortunately, I don't have any other options to check at this point. We would need a reproducer to investigate further. Maybe the reporter could attempt to extract a standalone reproducer? The relevant parts seems to be the "(including string creation for the prepared statements) for “translations” in a loop" mentioned in the description. > How could I get other / "older" builds to check (w/o have to build them by myself)? We don't provide any older builds of the Oracle JDK, so the only way would be to build yourself.
15-10-2024

Additional Information from submitter: ============================================ Here are the answers for the latest questions: @thartmann -XX:-OptimizeStringConcat and -XX:-CompactStrings crashes too / does fix the problems (so are not valid workarounds) How to get and use a "debug build"? I tried the Adoptium nightly build (https://adoptium.net/temurin/nightly/) OpenJDK21U-jdk_x64_windows_hotspot_21.0.5_8-ea.zip + OpenJDK21U-debugimage_x64_windows_hotspot_21.0.5_8-ea.zip (using -ea), but I think this crashed the same way (hs_err file attached - please keep it confidential) Do you have more options to check? How could I get other / "older" builds to check (w/o have to build them by myself)?
15-10-2024

[~pnarayanaswa] Any further information from the reporter? Otherwise, I would suggest to close this as incomplete.
14-10-2024

[~pnarayanaswa] you posted this reply already above. Did you mean to post a new reply?
13-09-2024

Additional Information from Submitter: =============================== @thartmann I tried both options (-XX:-DoEscapeAnalysis and -XX:-EliminateAllocations) and they both (isolated and used together) "fixes" the issue (so are valid workarounds) :-) I addition (also for @dholmes) here are the versions (working and non-working) I tested so far: Working versions (values taken from java -version): OpenJDK 64-Bit Server VM Temurin-11.0.24+8 (build 11.0.24+8, mixed mode) OpenJDK 64-Bit Server VM Temurin-17.0.12+7 (build 17.0.12+7, mixed mode, sharing) OpenJDK 64-Bit Server VM Temurin-20.0.2+9 (build 20.0.2+9, mixed mode, sharing) Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.4+8.1 (build 21.0.4+8-LTS-jvmci-23.1-b41, mixed mode, sharing) Non-working versions (values taken from hs_err_pid files): OpenJDK 64-Bit Server VM (21+35-2513, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (21.0.3+9-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) Java HotSpot(TM) 64-Bit Server VM (21.0.4+8-LTS-274, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (21.0.4+7-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Corretto-21.0.4.7.1 (21.0.4+7-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-22.0.2+9 (22.0.2+9, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM (23+36-2368, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM (24-ea+10-1076, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) It looks like the "issue" was added in the Java 21 version (from the first released version) and is still in. Feel free to give me more VM options to trace the issue, because I could reproduce it quite easy, but could find a "small sharable sample".
13-09-2024

Pre-ILW = Incorrect result or crash (probably with C2 compiled code), reproducible in customer environment only, -XX:-EliminateAllocations or enable compressed oops = HLM = P3
11-09-2024

Unfortunately, this is still not enough information to further investigate the issue. The fact that '-XX:-EliminateAllocations' helps, points to a C2 bug but it's also a quite invasive setting, so it might just hide the issue. To debug this, we would need a reproducer. Here are a few more things to try: - Run with a debug build of the JVM and see if any asserts trigger - Run with -XX:-OptimizeStringConcat - Run with -XX:-CompactStrings Most importantly, it would help us if the reporter could further narrow down the JDK 21 build that introduced the issue. Since old builds are not available anymore publicly, the reporter would need to build the JVM manually. Would that be possible?
11-09-2024

Re-opening now we have additional information.
09-09-2024

Adding additional information received from submitter ================================== I tried to add the new information via the upload form (via the link in the "Update Notification: Bug Report" email - "To provide more information about this issue, click here."), but this does not work (even if I got positive feedback after filling and submitting the form)... Here is my additional information: @tohartma I tried both options (-XX:-DoEscapeAnalysis and -XX:-EliminateAllocations) and they both (isolated and used together) "fixes" the issue (so are valid workarounds) :-) I addition (also for @dholmes) here are the versions (working and non-working) I tested so far: Working versions (values taken from java -version): OpenJDK 64-Bit Server VM Temurin-11.0.24+8 (build 11.0.24+8, mixed mode) OpenJDK 64-Bit Server VM Temurin-17.0.12+7 (build 17.0.12+7, mixed mode, sharing) OpenJDK 64-Bit Server VM Temurin-20.0.2+9 (build 20.0.2+9, mixed mode, sharing) Java HotSpot(TM) 64-Bit Server VM Oracle GraalVM 21.0.4+8.1 (build 21.0.4+8-LTS-jvmci-23.1-b41, mixed mode, sharing) Non-working versions (values taken from hs_err_pid files): OpenJDK 64-Bit Server VM (21+35-2513, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-21.0.3+9 (21.0.3+9-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) Java HotSpot(TM) 64-Bit Server VM (21.0.4+8-LTS-274, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-21.0.4+7 (21.0.4+7-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Corretto-21.0.4.7.1 (21.0.4+7-LTS, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM Temurin-22.0.2+9 (22.0.2+9, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM (23+36-2368, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) OpenJDK 64-Bit Server VM (24-ea+10-1076, mixed mode, sharing, tiered, compressed class ptrs, g1 gc, windows-amd64) It looks like the "issue" was added in the Java 21 version (from the first released version) and is still in. Feel free to give me more VM options to trace the issue, because I could reproduce it quite easy, but could find a "small sharable sample". Regards
09-09-2024

Resolving as Incomplete until we wait for more information from the reporter.
05-09-2024

As already mentioned, this will be difficult to investigate without a reproducer. Here are a few things that would be helpful: - Try with a debug build of the JVM and see if any asserts trigger - Try with previous versions of JDK 21 and below to narrow down when the issue was introduced (ideally, perform binary search on the builds) - Try with -XX:-DoEscapeAnalysis - Try with -XX:-EliminateAllocations
04-09-2024

I checked a few of the hs_err files provided by the reporter and I think they all have the same root cause which is an invalid (or unexpectedly null) oop. [~dholmes] I think the stack trace is missing because we crash when executing a virtual call (probably the frame states are screwed up). Seems similar to JDK-8338656 which I currently struggle to reproduce.
04-09-2024

Mailed submitter ============ Hi Lars, A reproducer would be very helpful, Without a reproducer this will be difficult to assess. Could you share the complete java version > 21 have been tested? Thanks Raja
04-09-2024

Without a reproducer this will be difficult to assess. It is interesting that none of the hs_err files actually show a stack trace, other than an apparent native entry point - but perhaps that is a C2 limitation. Can the submitter advise exactly which versions > 21 have been tested?
04-09-2024