JDK-8296955 : Kitchensink.java failed with "double free or corruption (!prev): "
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 20
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2022-11-14
  • Updated: 2022-12-19
  • Resolved: 2022-12-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 20 JDK 21
20 b28Fixed 21Fixed
Related Reports
Duplicate :  
JDK-8295357 - Kitchensink8H.java failed with SIGSEGV in MallocTracker::record_free
Relates :  
JDK-8290012 - RenaissanceStressTest.java and RenaissanceStressTest24H.java misc failures
Description
The following test failed in the JDK20 CI:

applications/kitchensink/Kitchensink.java

Here's a snippet from the log file:

[stress.process.err] Iteration start: Monitor at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: MemAccess at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: MBean at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Inflater at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: SpecJbb2005 at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Reference at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: LockDeflation at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Jcmd at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Jni at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Jvmti at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Jstat at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: SpecJvm2008Batch at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: ThreadDumper at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Jfr at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Perfmon at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: NMT at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Exception at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration start: Allocator at Sat Nov 12 04:13:14 UTC 2022
[stress.process.err] Iteration done: MBean at Sat Nov 12 04:13:16 UTC 2022
[stress.process.err] Iteration done: ThreadDumper at Sat Nov 12 04:13:16 UTC 2022
[stress.process.err] *** Error in `/opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/bin/java': double free or corruption (!prev): 0x00007fcaf0009a70 ***
[stress.process.err] ======= Backtrace: =========
[stress.process.err] /lib64/libc.so.6(+0x81329)[0x7fcb74161329]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0xfea973)[0x7fcb72dc4973]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x16097ae)[0x7fcb733e37ae]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x1687ca3)[0x7fcb73461ca3]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x174af8b)[0x7fcb73524f8b]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0xe5f027)[0x7fcb72c39027]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x1bd6c90)[0x7fcb739b0c90]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x1a7e070)[0x7fcb73858070]
[stress.process.err] /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so(+0x1714e43)[0x7fcb734eee43]
[stress.process.err] /lib64/libpthread.so.0(+0x7ea5)[0x7fcb746b9ea5]
[stress.process.err] /lib64/libc.so.6(clone+0x6d)[0x7fcb741deb2d]
[stress.process.err] ======= Memory map: ========
[stress.process.err] 519800000-785800000 rw-p 00000000 00:00 0 
[stress.process.err] 785800000-7ff000000 ---p 00000000 00:00 0 
[stress.process.err] 7ff000000-7ff700000 rw-p 00000000 00:00 0 

<snip>

[stress.process.err] 7ffe872a8000-7ffe872ab000 r--p 00000000 00:00 0                          [vvar]
[stress.process.err] 7ffe872ab000-7ffe872ad000 r-xp 00000000 00:00 0                          [vdso]
[stress.process.err] ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]
Stress process is started and all modules initialized.
Dumping stress process VM options.
 stdout: [23488:
];
 stderr: [java.io.IOException: Connection reset by peer
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.read(Native Method)
	at jdk.attach/sun.tools.attach.VirtualMachineImpl$SocketInputStream.read(VirtualMachineImpl.java:265)
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.readInt(HotSpotVirtualMachine.java:332)
	at jdk.attach/sun.tools.attach.VirtualMachineImpl.execute(VirtualMachineImpl.java:197)
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.executeCommand(HotSpotVirtualMachine.java:313)
	at jdk.attach/sun.tools.attach.HotSpotVirtualMachine.executeJCmd(HotSpotVirtualMachine.java:295)
	at jdk.jcmd/sun.tools.jcmd.JCmd.executeCommandForPid(JCmd.java:124)
	at jdk.jcmd/sun.tools.jcmd.JCmd.main(JCmd.java:97)
]
 exitValue = 1

Got an exception, so I'll kill the stress process process
[Sat Nov 12 04:14:20 UTC 2022] Unexpected errors during GlueModule initialization: Expected to get exit value of [0], exit value is: [1]
java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]
	at jdk.test.lib.process.OutputAnalyzer.shouldHaveExitValue(OutputAnalyzer.java:490)
	at applications.kitchensink.process.glue.Main.dumpOptions(Main.java:442)
	at applications.kitchensink.process.glue.Main.execute(Main.java:333)
	at applications.kitchensink.process.glue.Main.main(Main.java:217)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:125)
	at java.base/java.lang.Thread.run(Thread.java:1591)

JavaTest Message: Test threw exception: java.lang.RuntimeException: Expected to get exit value of [0], exit value is: [1]
Comments
Changeset: d4531903 Author: Coleen Phillimore <coleenp@openjdk.org> Date: 2022-12-13 00:49:16 +0000 URL: https://git.openjdk.org/jdk20/commit/d4531903007dbe8dcdd163e423d23e8cefba61c8
13-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk20/pull/10 Date: 2022-12-09 12:27:42 +0000
09-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/11594 Date: 2022-12-08 14:22:15 +0000
08-12-2022
Yes, this looks like an accurate diagnosis. In the core file, there are several threads doing CodeCacheUnloadingTask.
07-12-2022
InstanceKlass::get_jmethod_id has a couple of places where it thinks it is single-threaded because it is either during bootstrapping or in a safepoint. But as the stack from the core dump in the comment from 2022-11-14 shows, it is being called from G1ParallelCleaningTask. So that single-threaded assumption looks to be wrong. One of the places this can go wrong is in the call to get_jmethod_id_fetch_or_update, which has an out parameter for an array of jmethodID's that is to be deallocated. I think what is happening is that two worker threads get that same array and both try to deallocate it. Likely culprit is JDK-8290025: Remove the Sweeper. Everything else in the relevant neighborhood looks to have been last changed 3+ years ago.
06-12-2022
I see the following stack in the core dump: Thread 1 (Thread 0x7fcb7052f700 (LWP 23492)): #0 0x00007fcb74116387 in raise () from /lib64/libc.so.6 #1 0x00007fcb74117a78 in abort () from /lib64/libc.so.6 #2 0x00007fcb74158f67 in __libc_message () from /lib64/libc.so.6 #3 0x00007fcb74161329 in _int_free () from /lib64/libc.so.6 #4 0x00007fcb72dc4973 in InstanceKlass::get_jmethod_id(methodHandle const&) () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #5 0x00007fcb733e37ae in Method::jmethod_id() () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #6 0x00007fcb73461ca3 in nmethod::unlink() [clone .part.0] () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #7 0x00007fcb73524f8b in CodeCacheUnloadingTask::work(unsigned int) () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #8 0x00007fcb72c39027 in G1ParallelCleaningTask::work(unsigned int) () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #9 0x00007fcb739b0c90 in WorkerThread::run() () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #10 0x00007fcb73858070 in Thread::call_run() () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #11 0x00007fcb734eee43 in thread_native_entry(Thread*) () from /opt/mach5/mesos/work_dir/jib-master/install/jdk-20+24-1714/linux-x64-debug.jdk/jdk-20/fastdebug/lib/server/libjvm.so #12 0x00007fcb746b9ea5 in start_thread () from /lib64/libpthread.so.0 #13 0x00007fcb741deb2d in clone () from /lib64/libc.so.6 ----------------------------------------
14-11-2022
There's another BigApps bug that has a "double free or corruption (!prev):" failure mode: JDK-8290012 RenaissanceStressTest.java and RenaissanceStressTest24H.java misc failures
14-11-2022