JDK-8266585 : dynamicArchive/ParallelLambdaLoadTest.java crashes with memory stomping error
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 17
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • Submitted: 2021-05-05
  • Updated: 2021-05-17
  • Resolved: 2021-05-09
Related Reports
Duplicate :  
JDK-8266765 - [BACKOUT] JDK-8255493 Support for pre-generated java.lang.invoke classes in CDS dynamic archive
Relates :  
JDK-8266764 - [REDO] JDK-8255493 Support for pre-generated java.lang.invoke classes in CDS dynamic archive
Relates :  
JDK-8266576 - dynamicArchive/ParallelLambdaLoadTest.java crashes with "assert(is_instance(java_string)) failed: must be java_string"
Relates :  
JDK-8266594 - dynamicArchive/ParallelLambdaLoadTest.java fails with "Archiving hidden ..." missing from stdout/stderr
Relates :  
JDK-8255493 - Support for pre-generated java.lang.invoke classes in CDS dynamic archive
Description
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\\os.cpp:652
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (t:\\workspace\\open\\src\\hotspot\\share\\runtime\\os.cpp:652), pid=23504, tid=11588
#  fatal error: memory stomping error
#
Comments
JDK-8266765 [BACKOUT] JDK-8255493 Support for pre-generated java.lang.invoke classes in CDS dynamic archive was integrated in jdk-17+22-1908.
17-05-2021
The jdk-17+21-1848-tier6 crash in Dan's comment above happened with -XX:ArchiveClassesAtExit. It's likely a side effect of the memory corruption that happened inside LambdaFormInvokers::append(). ================== Command Line: -XX:MaxRAMPercentage=4 -Djava.io.tmpdir=t:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_hotspot_runtime\tmp -XX:+CreateCoredumpOnCrash -XX:+UseSerialGC -XX:ArchiveClassesAtExit=T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_hotspot_runtime\scratch\0\appcds-02h15m45s410.jsa -Xshare:auto -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -Xlog:cds+dynamic=info -Xbootclasspath/a:T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_hotspot_runtime\scratch\0\WhiteBox.jar ParallelLambdaLoad --------------- T H R E A D --------------- Current thread (0x000001e21da32a80): JavaThread "Thread-2" [_thread_in_vm, id=46348, stack(0x0000008043d00000,0x0000008043e00000)] Stack: [0x0000008043d00000,0x0000008043e00000], sp=0x0000008043dfd578, free space=1013k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x193fa] AccessInternal::PostRuntimeDispatch<EpsilonBarrierSet::AccessBarrier<548964,EpsilonBarrierSet>,2,548964>::oop_access_barrier+0xa (access.inline.hpp:85) Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j java.lang.invoke.MethodHandleNatives.resolve(Ljava/lang/invoke/MemberName;Ljava/lang/Class;IZ)Ljava/lang/invoke/MemberName;+0 java.base@17-ea j java.lang.invoke.MemberName$Factory.resolve(BLjava/lang/invoke/MemberName;Ljava/lang/Class;IZ)Ljava/lang/invoke/MemberName;+36 java.base@17-ea j java.lang.invoke.MemberName$Factory.resolveOrFail(BLjava/lang/invoke/MemberName;Ljava/lang/Class;ILjava/lang/Class;)Ljava/lang/invoke/MemberName;+31 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.resolveOrFail(BLjava/lang/Class;Ljava/lang/String;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MemberName;+44 java.base@17-ea
07-05-2021
When the following is integrated: JDK-8266765 [BACKOUT] JDK-8255493 Support for pre-generated java.lang.invoke classes in CDS dynamic archive and we have verified that the failures associated with this bug stop, we can close this issue as a duplicate of JDK-8266765.
07-05-2021
Here's snippets from the hs_err_pid for the jdk-17+21-1848-tier6 sighting: # EXCEPTION_ACCESS_VIOLATION (0xc0000005) at pc=0x00007ffdea2993fa, pid=5608, tid=46348 # # JRE version: Java(TM) SE Runtime Environment (17.0+21) (fastdebug build 17-ea+21-LTS-1848) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 17-ea+21-LTS-1848, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, serial gc, windows-amd64) # Problematic frame: # V [jvm.dll+0x193fa] AccessInternal::PostRuntimeDispatch<EpsilonBarrierSet::AccessBarrier<548964,EpsilonBarrierSet>,2,548964>::oop_access_barrier+0xa # # Core dump will be written. Default location: T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_hotspot_runtime\scratch\0\hs_err_pid5608.mdmp --------------- T H R E A D --------------- Current thread (0x000001e21da32a80): JavaThread "Thread-2" [_thread_in_vm, id=46348, stack(0x0000008043d00000,0x0000008043e00000)] Stack: [0x0000008043d00000,0x0000008043e00000], sp=0x0000008043dfd578, free space=1013k Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0x193fa] AccessInternal::PostRuntimeDispatch<EpsilonBarrierSet::AccessBarrier<548964,EpsilonBarrierSet>,2,548964>::oop_access_barrier+0xa (access.inline.hpp:85) Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j java.lang.invoke.MethodHandleNatives.resolve(Ljava/lang/invoke/MemberName;Ljava/lang/Class;IZ)Ljava/lang/invoke/MemberName;+0 java.base@17-ea j java.lang.invoke.MemberName$Factory.resolve(BLjava/lang/invoke/MemberName;Ljava/lang/Class;IZ)Ljava/lang/invoke/MemberName;+36 java.base@17-ea j java.lang.invoke.MemberName$Factory.resolveOrFail(BLjava/lang/invoke/MemberName;Ljava/lang/Class;ILjava/lang/Class;)Ljava/lang/invoke/MemberName;+31 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.resolveOrFail(BLjava/lang/Class;Ljava/lang/String;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MemberName;+44 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.findStatic(Ljava/lang/Class;Ljava/lang/String;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MethodHandle;+6 java.base@17-ea j java.lang.invoke.ClassSpecializer$Factory.findFactory(Ljava/lang/Class;Ljava/util/List;)Ljava/lang/invoke/MethodHandle;+30 java.base@17-ea j java.lang.invoke.ClassSpecializer$Factory.findFactories(Ljava/lang/Class;Ljava/util/List;)Ljava/util/List;+10 java.base@17-ea j java.lang.invoke.ClassSpecializer$Factory.linkSpeciesDataToCode(Ljava/lang/invoke/ClassSpecializer$SpeciesData;Ljava/lang/Class;)V+24 java.base@17-ea j java.lang.invoke.ClassSpecializer$Factory.loadSpecies(Ljava/lang/invoke/ClassSpecializer$SpeciesData;)Ljava/lang/invoke/ClassSpecializer$SpeciesData;+85 java.base@17-ea j java.lang.invoke.ClassSpecializer.findSpecies(Ljava/lang/Object;)Ljava/lang/invoke/ClassSpecializer$SpeciesData;+53 java.base@17-ea j java.lang.invoke.BoundMethodHandle$SpeciesData.extendWith(B)Ljava/lang/invoke/BoundMethodHandle$SpeciesData;+46 java.base@17-ea j java.lang.invoke.LambdaFormEditor.newSpeciesData(Ljava/lang/invoke/LambdaForm$BasicType;)Ljava/lang/invoke/BoundMethodHandle$SpeciesData;+9 java.base@17-ea j java.lang.invoke.LambdaFormEditor.bindArgumentForm(I)Ljava/lang/invoke/LambdaForm;+78 java.base@17-ea j java.lang.invoke.LambdaFormEditor.bindArgumentL(Ljava/lang/invoke/BoundMethodHandle;ILjava/lang/Object;)Ljava/lang/invoke/BoundMethodHandle;+44 java.base@17-ea j java.lang.invoke.BoundMethodHandle.bindArgumentL(ILjava/lang/Object;)Ljava/lang/invoke/BoundMethodHandle;+7 java.base@17-ea j java.lang.invoke.MethodHandles.insertArguments(Ljava/lang/invoke/MethodHandle;I[Ljava/lang/Object;)Ljava/lang/invoke/MethodHandle;+84 java.base@17-ea j java.lang.invoke.StringConcatFactory.generateMHInlineCopy(Ljava/lang/invoke/MethodType;Ljava/util/List;)Ljava/lang/invoke/MethodHandle;+195 java.base@17-ea j java.lang.invoke.StringConcatFactory.makeConcatWithConstants(Ljava/lang/invoke/MethodHandles$Lookup;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/String;[Ljava/lang/Object;)Ljava/lang/invoke/CallSite;+220 java.base@17-ea j java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+18 java.base@17-ea j java.lang.invoke.DelegatingMethodHandle$Holder.delegate(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+21 java.base@17-ea j java.lang.invoke.Invokers$Holder.invokeExact_MT(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)Ljava/lang/Object;+26 java.base@17-ea j java.lang.invoke.BootstrapMethodInvoker.invoke(Ljava/lang/Class;Ljava/lang/invoke/MethodHandle;Ljava/lang/String;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Class;)Ljava/lang/Object;+157 java.base@17-ea j java.lang.invoke.CallSite.makeSite(Ljava/lang/invoke/MethodHandle;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/Object;Ljava/lang/Class;)Ljava/lang/invoke/CallSite;+8 java.base@17-ea j java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(Ljava/lang/Class;Ljava/lang/invoke/MethodHandle;Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/invoke/MemberName;+6 java.base@17-ea j java.lang.invoke.MethodHandleNatives.linkCallSite(Ljava/lang/Object;ILjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;[Ljava/lang/Object;)Ljava/lang/invoke/MemberName;+46 java.base@17-ea v ~StubRoutines::call_stub j LambdaVerification.verifyCallerIsArchivedLambda(Z)V+9 j DoSomething.run()V+33 j ParallelLambdaLoad$$Lambda$15+0x0000000801006000.run()V+4 j ParallelLambdaLoad.lambda$main$2()V+26 j ParallelLambdaLoad$$Lambda$3+0x0000000801001000.run()V+0 j java.lang.Thread.run()V+11 java.base@17-ea v ~StubRoutines::call_stub siginfo: EXCEPTION_ACCESS_VIOLATION (0xc0000005), reading address 0xffffffffffffffff While this is not the same as the "memory stomping error" failure mode, it looks to me like this is closely related: - crashed due to EXCEPTION_ACCESS_VIOLATION - # Problematic frame: # V [jvm.dll+0x193fa] AccessInternal::PostRuntimeDispatch<EpsilonBarrierSet::AccessBarrier<548964,EpsilonBarrierSet>,2,548964>::oop_access_barrier+0xa
07-05-2021
Here's snippets from the hs_err_pid for the jdk-17+21-1866-tier6 sighting: # Internal Error (t:\workspace\open\src\hotspot\share\services/nmtCommon.hpp:63), pid=76700, tid=42724 # assert(flag_is_valid(flag)) failed: Invalid flag # # JRE version: Java(TM) SE Runtime Environment (17.0+21) (fastdebug build 17-ea+21-LTS-1866) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 17-ea+21-LTS-1866, mixed mode, sharing, tiered, compressed oops, compressed class ptrs, g1 gc, windows-amd64) # Core dump will be written. Default location: T:\testoutput\test-support\jtreg_open_test_hotspot_jtreg_hotspot_runtime\scratch\2\hs_err_pid76700.mdmp --------------- T H R E A D --------------- Current thread (0x00000205742949b0): JavaThread "Thread-6" [_thread_in_vm, id=42724, stack(0x000000d803800000,0x000000d803900000)] Stack: [0x000000d803800000,0x000000d803900000] Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code) V [jvm.dll+0xacddf1] os::platform_print_native_stack+0xf1 (os_windows_x86.cpp:235) V [jvm.dll+0xcf06b7] VMError::report+0xfc7 (vmError.cpp:739) V [jvm.dll+0xcf203e] VMError::report_and_die+0x7fe (vmError.cpp:1549) V [jvm.dll+0xcf2754] VMError::report_and_die+0x64 (vmError.cpp:1330) V [jvm.dll+0x4ce667] report_vm_error+0xb7 (debug.cpp:282) V [jvm.dll+0x9fa0ce] MallocTracker::record_free+0xfe (mallocTracker.cpp:182) V [jvm.dll+0xabbe7b] os::free+0x8b (os.cpp:798) V [jvm.dll+0x8e8a6e] LambdaFormInvokers::append+0x1de (lambdaFormInvokers.cpp:79) V [jvm.dll+0x7a08c6] JVM_LogLambdaFormInvoker+0x2e6 (jvm.cpp:3667) C 0x000002055c56f861 Java frames: (J=compiled Java code, j=interpreted, Vv=VM code) j jdk.internal.misc.CDS.logLambdaFormInvoker(Ljava/lang/String;)V+0 java.base@17-ea j jdk.internal.misc.CDS.traceLambdaFormInvoker(Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;Ljava/lang/String;)V+47 java.base@17-ea j java.lang.invoke.MethodHandleStatics.traceLambdaForm(Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/Class;Ljava/lang/invoke/MemberName;)V+92 java.base@17-ea j java.lang.invoke.InvokerBytecodeGenerator.resolveFrom(Ljava/lang/String;Ljava/lang/invoke/MethodType;Ljava/lang/Class;)Ljava/lang/invoke/MemberName;+31 java.base@17-ea j java.lang.invoke.InvokerBytecodeGenerator.lookupPregenerated(Ljava/lang/invoke/LambdaForm;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MemberName;+288 java.base@17-ea j java.lang.invoke.InvokerBytecodeGenerator.generateCustomizedCode(Ljava/lang/invoke/LambdaForm;Ljava/lang/invoke/MethodType;)Ljava/lang/invoke/MemberName;+2 java.base@17-ea j java.lang.invoke.LambdaForm.compileToBytecode()V+69 java.base@17-ea j java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/LambdaForm;+792 java.base@17-ea j java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Ljava/lang/invoke/MethodType;I)Ljava/lang/invoke/LambdaForm;+17 java.base@17-ea j java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Ljava/lang/invoke/MemberName;Z)Ljava/lang/invoke/LambdaForm;+163 java.base@17-ea j java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Ljava/lang/invoke/MemberName;)Ljava/lang/invoke/LambdaForm;+2 java.base@17-ea j java.lang.invoke.DirectMethodHandle.makeAllocator(Ljava/lang/invoke/MemberName;)Ljava/lang/invoke/DirectMethodHandle;+84 java.base@17-ea j java.lang.invoke.DirectMethodHandle.make(Ljava/lang/invoke/MemberName;)Ljava/lang/invoke/DirectMethodHandle;+8 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.getDirectConstructorCommon(Ljava/lang/Class;Ljava/lang/invoke/MemberName;Z)Ljava/lang/invoke/MethodHandle;+61 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.getDirectConstructorNoSecurityManager(Ljava/lang/Class;Ljava/lang/invoke/MemberName;)Ljava/lang/invoke/MethodHandle;+6 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.getDirectMethodForConstant(BLjava/lang/Class;Ljava/lang/invoke/MemberName;)Ljava/lang/invoke/MethodHandle;+44 java.base@17-ea j java.lang.invoke.MethodHandles$Lookup.linkMethodHandleConstant(BLjava/lang/Class;Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/invoke/MethodHandle;+153 java.base@17-ea j java.lang.invoke.MethodHandleNatives.linkMethodHandleConstant(Ljava/lang/Class;ILjava/lang/Class;Ljava/lang/String;Ljava/lang/Object;)Ljava/lang/invoke/MethodHandle;+38 java.base@17-ea v ~StubRoutines::call_stub j java.util.stream.Collectors.toList()Ljava/util/stream/Collector;+4 java.base@17-ea j LambdaVerification.lambda$getCallerClass$0(ILjava/util/stream/Stream;)Ljava/util/List;+10 j LambdaVerification$$Lambda$21+0x0000000801006f30.apply(Ljava/lang/Object;)Ljava/lang/Object;+8 j java.lang.StackStreamFactory$StackFrameTraverser.consumeFrames()Ljava/lang/Object;+25 java.base@17-ea j java.lang.StackStreamFactory$AbstractStackWalker.doStackWalk(JIIII)Ljava/lang/Object;+78 java.base@17-ea v ~StubRoutines::call_stub j java.lang.StackStreamFactory$AbstractStackWalker.callStackWalk(JIII[Ljava/lang/Object;)Ljava/lang/Object;+0 java.base@17-ea j java.lang.StackStreamFactory$AbstractStackWalker.beginStackWalk()Ljava/lang/Object;+31 java.base@17-ea j java.lang.StackStreamFactory$AbstractStackWalker.walk()Ljava/lang/Object;+8 java.base@17-ea j java.lang.StackWalker.walk(Ljava/util/function/Function;)Ljava/lang/Object;+10 java.base@17-ea j LambdaVerification.getCallerClass(I)Ljava/lang/Class;+20 j DoSomething.run()V+1 j ParallelLambdaLoad$$Lambda$12+0x0000000801002880.run()V+4 j ParallelLambdaLoad.lambda$main$6()V+27 j ParallelLambdaLoad$$Lambda$7+0x00000008010018a0.run()V+0 j java.lang.Thread.run()V+11 java.base@17-ea v ~StubRoutines::call_stub While this is not the same as the "memory stomping error" failure mode, it looks to me like this is closely related: V [jvm.dll+0x4ce667] report_vm_error+0xb7 (debug.cpp:282) V [jvm.dll+0x9fa0ce] MallocTracker::record_free+0xfe (mallocTracker.cpp:182) V [jvm.dll+0xabbe7b] os::free+0x8b (os.cpp:798) V [jvm.dll+0x8e8a6e] LambdaFormInvokers::append+0x1de (lambdaFormInvokers.cpp:79) as we're in os::free() and cause NMT to fail an assertion: # Internal Error (t:\workspace\open\src\hotspot\share\services/nmtCommon.hpp:63), pid=76700, tid=42724 # assert(flag_is_valid(flag)) failed: Invalid flag
07-05-2021
Evaluation from [~iklam] The memory stomping happens in here: void LambdaFormInvokers::append(char* line) { if (_lambdaform_lines == NULL) { _lambdaform_lines = new GrowableArrayCHeap<char*, mtClassShared>(150); } _lambdaform_lines->append(line); } The function should be protected by a lock.
05-05-2021