JDK-8267703 : runtime/cds/appcds/cacheObject/HeapFragmentationTest.java crashed with OutOfMemory
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 17
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2021-05-25
  • Updated: 2022-01-11
  • Resolved: 2021-06-07
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 17
17 b26Fixed
Related Reports
Relates :  
JDK-8279241 - runtime/cds/appcds/cacheObject/HeapFragmentationTest.java failed with "fatal error: OutOfMemory encountered: Java heap space"
Relates :  
JDK-8214455 - Relocate CDS archived regions to the top of the G1 heap
Description
The following test failed in the JDK17 CI:

runtime/cds/appcds/cacheObject/HeapFragmentationTest.java

Here's a snippet from the log file:

0.820s][info][gc,heap] GC(4) Eden regions: 0->0(1)
[0.820s][info][gc,heap] GC(4) Survivor regions: 0->0(5)
[0.820s][info][gc,heap] GC(4) Old regions: 1->1
[0.820s][info][gc,heap] GC(4) Archive regions: 0->0
[0.820s][info][gc,heap] GC(4) Humongous regions: 0->0
[0.820s][info][gc     ] GC(4) Pause Full (G1 Humongous Allocation) 0M->0M(8M) 90.869ms
[0.833s][info][gc     ] GC(1) Concurrent Undo Cycle 415.336ms
Aborting due to java.lang.OutOfMemoryError: Java heap space
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=/debug.cpp:364
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/opt/mach5/mesos/work_dir/slaves/3c846bae-ce30-4a97-93ee-9fef4497ccb6-S79213/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/3d136cf9-b484-41b0-a4b4-5ded75b927ce/runs/990bcb78-f7c1-4ada-8b7a-5a7801a37528/workspace/open/src/hotspot/share/utilities/debug.cpp:364), pid=32046, tid=32049
#  fatal error: OutOfMemory encountered: Java heap space
#
# JRE version: Java(TM) SE Runtime Environment (17.0+24) (fastdebug build 17-ea+24-LTS-2133)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 17-ea+24-LTS-2133, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/3c846bae-ce30-4a97-93ee-9fef4497ccb6-S79213/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2c4217ec-0c42-4111-a920-a26f6eeef273/runs/52d08d41-3773-43fb-bd87-ef71779f483b/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/2/core.32046)
#
# An error report file with more information is saved as:
# /opt/mach5/mesos/work_dir/slaves/3c846bae-ce30-4a97-93ee-9fef4497ccb6-S79213/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/2c4217ec-0c42-4111-a920-a26f6eeef273/runs/52d08d41-3773-43fb-bd87-ef71779f483b/testoutput/test-support/jtreg_open_test_hotspot_jtreg_hotspot_runtime/scratch/2/hs_err_pid32046.log
];
 stderr: [java version "17-ea" 2021-09-14 LTS
Java(TM) SE Runtime Environment (fastdebug build 17-ea+24-LTS-2133)
Java HotSpot(TM) 64-Bit Server VM (fastdebug build 17-ea+24-LTS-2133, mixed mode)
]
 exitValue = 134

java.lang.RuntimeException: 'array.length = 1048576000' missing from stdout/stderr 

	at jdk.test.lib.process.OutputAnalyzer.shouldContain(OutputAnalyzer.java:206)
	at jdk.test.lib.cds.CDSTestUtils.checkMatches(CDSTestUtils.java:483)
	at jdk.test.lib.cds.CDSTestUtils$Result.assertNormalExit(CDSTestUtils.java:194)
	at HeapFragmentationTest.main(HeapFragmentationTest.java:80)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:77)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:568)
	at com.sun.javatest.regtest.agent.MainActionHelper$AgentVMRunnable.run(MainActionHelper.java:298)
	at java.base/java.lang.Thread.run(Thread.java:831)

JavaTest Message: Test threw exception: java.lang.RuntimeException
JavaTest Message: shutting down test

result: Failed. Execution failed: `main' threw exception: java.lang.RuntimeException: 'array.length = 1048576000' missing from stdout/stderr


Here's the crashing thread's stack:

---------------  T H R E A D  ---------------

Current thread (0x00007f9ab4039900):  JavaThread "main" [_thread_in_vm, id=32049, stack(0x00007f9abb7d7000,0x00007f9abb8d8000)]

Stack: [0x00007f9abb7d7000,0x00007f9abb8d8000],  sp=0x00007f9abb8d62e0,  free space=1020k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x198a0f0]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x1a0
V  [libjvm.so+0xa81ceb]  report_fatal(VMErrorType, char const*, int, char const*, ...)+0x12b
V  [libjvm.so+0xa81f9d]  report_java_out_of_memory(char const*)+0xed
V  [libjvm.so+0x141114a]  MemAllocator::Allocation::check_out_of_memory()+0xda
V  [libjvm.so+0x14139f1]  MemAllocator::allocate() const+0x191
V  [libjvm.so+0x18e05a6]  TypeArrayKlass::allocate_common(int, bool, JavaThread*)+0x2e6
V  [libjvm.so+0x154ce3f]  oopFactory::new_typeArray(BasicType, int, JavaThread*)+0xff
V  [libjvm.so+0xe565bf]  InterpreterRuntime::newarray(JavaThread*, BasicType, int)+0x10f
j  HeapFragmentationApp.main([Ljava/lang/String;)V+20
v  ~StubRoutines::call_stub
V  [libjvm.so+0xe704b4]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*)+0x504
V  [libjvm.so+0xfa9275]  jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) [clone .constprop.1]+0x375
V  [libjvm.so+0xfac8b5]  jni_CallStaticVoidMethod+0x1c5
C  [libjli.so+0x47d7]  JavaMain+0xd37
C  [libjli.so+0x7ce9]  ThreadJavaMain+0x9

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  HeapFragmentationApp.main([Ljava/lang/String;)V+20
v  ~StubRoutines::call_stub
Comments
Changeset: 204b4929 Author: Stefan Johansson <sjohanss@openjdk.org> Date: 2021-06-07 06:22:11 +0000 URL: https://git.openjdk.java.net/jdk/commit/204b4929f7251feec6736aed9489ff82cd5dd744
07-06-2021
Just had a sighting in Tier2 so I've bumped the priority from P4 -> P3.
02-06-2021
There are probably a few reasons this has not showed up more, you have to be a bit "unlucky". First of all, if the machine has a lot of memory so the initial heap sizes is larger than 1004M or so, there won't be any fragmentation to "solve". But in this case the initial heap size is 746M, so the array won't fit either below or above the initially allocated region that will be at region #746 (since the max is 1400M). G1 never actively try to place regions to fit a specific humongous allocation, but the Full GC tries to compact the regions to lower region numbers so that we get free regions at higher region numbers. After the Full GC was made Parallel, there is no longer a guarantee that after a Full GC all regions except humongous regions will be at the bottom. The reason is that the GC workers compete for regions to compact and if a worker is stalled or started slower than the others it might not get any "low" regions to compact into. In this scenario, when the Full GC is kicked off there is a single region around index #746 that needs to be compacted and in most cases it will be compacted to a region index that is way below #400 which is what is required to fit the array.
26-05-2021
This test was checked in as part of JDK-8214455 on 2021-04-07, and this has been the only such failure.
25-05-2021
NOTE: CDS is not enabled in the VM that crashed. # JRE version: Java(TM) SE Runtime Environment (17.0+24) (fastdebug build 17-ea+24-LTS-2133) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 17-ea+24-LTS-2133, mixed mode, tiered, compressed oops, compressed class ptrs, g1 gc, linux-amd64) The test configures a 1400m heap and asserts that it must be able to allocate a byte array that has 1000m elements: // We run with a 1400m heap, so we should be able to allocate a 1000m buffer, regardless // of the heap size chosen at dump time. String dumpTimeHeapSize = "-Xmx800m"; String runTimeHeapSize = "-Xmx1400m"; String BUFF_SIZE = Integer.toString(1000 * 1024 * 1024); To GC team, is this assumption correct? From the hs_err file, heap starts at 0x600000000, but the old region is placed in the middle at 0x626100000. That prevents the allocation of a humongous region that can fit the 1000m array. | 608|0x0000000626000000, 0x0000000626000000, 0x0000000626100000| 0%| F| |TAMS 0x0000000626000000, 0x0000000626000000| Untracked | 609|0x0000000626100000, 0x000000062619bd88, 0x0000000626200000| 60%| O| |TAMS 0x0000000626100000, 0x0000000626100000| Untracked | 610|0x0000000626200000, 0x0000000626200000, 0x0000000626300000| 0%| F| |TAMS 0x0000000626200000, 0x0000000626200000| Untracked Question: shouldn't G1 relocate the old region to make space for the humongous allocation?
25-05-2021