JDK-8286345 : runtime/NMT/ThreadedMallocTestType.java failed with java.lang.RuntimeException, we also see "exitValue = 134" and time out
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 19
  • Priority: P3
  • Status: Closed
  • Resolution: Cannot Reproduce
  • OS: os_x
  • CPU: aarch64
  • Submitted: 2022-05-06
  • Updated: 2024-01-25
  • Resolved: 2023-12-12
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 23
23Resolved
Related Reports
Blocks :  
CODETOOLS-7903217 - jtreg could try killing descendants of stuck test, before timing out the test
Relates :  
JDK-8299777 - Test runtime/NMT/BaselineWithParameter.java timed out
Relates :  
JDK-8299555 - Missing timeout info
Relates :  
CODETOOLS-7903217 - jtreg could try killing descendants of stuck test, before timing out the test
Relates :  
JDK-8250806 - serviceability/dcmd/gc/HeapDumpCompressedTest.java#id2 test timed out
Relates :  
JDK-8289162 - runtime/NMT/ThreadedMallocTestType.java should print out memory allocations to help debug
Relates :  
JDK-8291060 - OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE
Description
The following test failed in the JDK19 CI:

runtime/NMT/ThreadedMallocTestType.java

Here's a snippet from the log file:

#section:main
----------messages:(7/554)----------
command: main -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:NativeMemoryTracking=detail ThreadedMallocTestType
reason: User specified action: run main/othervm -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:NativeMemoryTracking=detail ThreadedMallocTestType 
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.base,java.management --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
Timeout information:
--- Timeout information end.
elapsed time (seconds): 502.215
----------configuration:(4/111)----------
Boot Layer
  add modules: java.base java.management   
  add exports: java.base/jdk.internal.misc ALL-UNNAMED

----------System.out:(4/242)----------
[2022-05-06T20:21:45.450275Z] Gathering output for process 99935
Timeout refired 480 times
[2022-05-06T20:30:02.868454Z] Waiting for completion for process 99935
[2022-05-06T20:30:02.868568Z] Waiting for completion finished for process 99935
----------System.err:(19/907)----------
 stdout: [99932:
];
 stderr: []
 exitValue = 134

java.lang.RuntimeException: 'Test (reserved=896KB, committed=896KB)' missing from stdout/stderr 

	at jdk.test.lib.process.OutputAnalyzer.shouldContain(OutputAnalyzer.java:221)
	at ThreadedMallocTestType.main(ThreadedMallocTestType.java:67)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:578)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:828)

JavaTest Message: Test threw exception: java.lang.RuntimeException: 'Test (reserved=896KB, committed=896KB)' missing from stdout/stderr 

JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: 'Test (reserved=896KB, committed=896KB)' missing from stdout/stderr
----------rerun:(38/7224)*----------
Comments
Correct, jcmd was a child process launched by the test and that jcmd process itself got stuck and blocked the parent process from successfully terminating. The enhancement in https://bugs.openjdk.org/browse/CODETOOLS-7903217 proposes to terminate any such child process before timing out the target/parent process, thus flushing stderr/stdout of the stuck child processes, and allowing the parent process to wrap up. Hopefully doing so will provide clues in the test's log as to what was going on at the time of the timeout. https://github.com/openjdk/jtreg/pull/97
28-07-2022
Looking at following output: ----------System.out:(4/242)---------- [2022-05-06T20:21:45.450275Z] Gathering output for process 99935 Timeout refired 480 times [2022-05-06T20:30:02.868454Z] Waiting for completion for process 99935 [2022-05-06T20:30:02.868568Z] Waiting for completion finished for process 99935 ----------System.err:(19/907)---------- I believe the test failure was due to fail or timeout on launching jcmd process.
28-07-2022
I found a simple enhancement to jtreg to help it handle stuck processes by trying to kill the descendants (which if stuck themselves will block the parent) first before timing out: https://bugs.openjdk.org/browse/CODETOOLS-7903217
24-06-2022
Here is a simple test I tried: ' fprintf(stderr, "[stderr SECRET]\n");' ' fprintf(stdout, "[stdout SECRET]\n");' ' fprintf(stderr, "[stderr SECRET2]"); fflush(stderr);' ' fprintf(stdout, "[stdout SECRET2]"); fflush(stdout);' ' fprintf(stderr, "[stderr SECRET3]");' ' fprintf(stdout, "[stdout SECRET3]");' ' abort();' On macOS we see: ' [stderr SECRET]' ' [stdout SECRET]' ' [stderr SECRET2][stdout SECRET2][stderr SECRET3][stdout SECRET3][1] 38919 abort ./jcmd_bad' on Linux (ubuntu) we see: ' [stderr SECRET]' ' [stdout SECRET]' ' [stderr SECRET2][stdout SECRET2][stderr SECRET3]Aborted (core dumped)' So it does look like on macOS the "abort" flushes, but on Linux it does not. On both oses the stderr is unbuffered, so it shows immediately. The stdout buffered, but it is automatically line flushed.
21-06-2022
Interesting note from https://man7.org/linux/man-pages/man3/abort.3.html Up until glibc 2.26, if the abort() function caused process termination, all open streams were closed and flushed (as with fclose(3)). However, in some cases this could result in deadlocks and data corruption. Therefore, starting with glibc 2.27, abort() terminates the process without flushing streams. POSIX.1 permits either possible behavior, saying that abort() "may include an attempt to effect fclose() on all open streams".
03-06-2022
I'm going to investigate whether this could be caused by reaching the macOS limit for file descriptors when running tests as part of test framework. Also, I will continue to think what we could check here inside the test to help us debug similar issues in the future.
03-06-2022
The question here then is what causes "jcmd" to get stuck trying to talk back to the test via "Java_sun_tools_attach_VirtualMachineImpl_read" The memory allocation probably succeeds, and is unrelated to this issue, however, I'd like to see the test print out the values from the malloc. This again, reminds me of the "stuck" process issue seen in our Mach5 testing frameworks.
03-06-2022
Even though I can't explain yet 100% what we see, the big picture right now is this: 1. test is started by jtreg harness. 2. test mallocs some memory. 3. test starts "jcmd", which runs as a separate process. 4. "jcmd" tries to talk back to the test via "Java_sun_tools_attach_VirtualMachineImpl_read" API to get the memory status. 5. "jcmd" seems stuck at "Java_sun_tools_attach_VirtualMachineImpl_read" and gets the test itself stuck, which is waiting for the output. 6. "jtreg" fires a timeout handler, which runs "jstack" as a separate process to report the back traces of all java threads, however, we see no output other than: Timeout information: --- Timeout information end. 7. "jstack" seems to exit, with no reported error, but also no output, other than above 8. "jcmd" receives SIGABORT (134 == 128 + 6) signal (from "jtreg"?) and exits, though without output other than: stdout: [99932: ]; stderr: [] exitValue = 134 9. test gets unblocked, but there is no output whatsoever and so the test throws a runtime exception and fails. So in this scenario we have: - timeout (of "jcmd" not the test itself) - crash and core file (from "jcmd", not the test itself) - runtime exception from the test marking the run as a failure
03-06-2022
Looking at the jtreg code, however, (assuming jtreg/src/share/classes/com/sun/javatest/regtest/exec/DefaultTimeoutHandler.java is used) it looks like we should at least see: 1. Timeout information: 2. Running jstack on process PID 3. --- Timeout information end. or in case things went wrong, some error messages and stack traces from jtreg itself at the point of error. But we only see 1 and 3 - there is no 2, which I can't explain looking at the src code right now.
03-06-2022
Similarly for the timeout info we have: Timeout information: --- Timeout information end. but what we should see coming from "jtreg" itself is something like: Timeout information: Running jstack on process 73902 2022-06-03 13:21:32 Full thread dump OpenJDK 64-Bit Server VM (19-internal-adhoc.gerard.jdk mixed mode, sharing): Threads class SMR info: _java_thread_list=0x00006000026bee90, length=16, elements={ 0x000000014f80da10, 0x000000014f8a3810, 0x000000014f8a6610, 0x000000014f8a6010, 0x000000014f8a7810, 0x000000014f8a9c10, 0x000000014f8aa810, 0x000000012f808210, 0x0000000158028810, 0x000000014f8b3e10, 0x0000000158026610, 0x000000014f8acc10, 0x000000014e80c810, 0x000000013e80ec10, 0x000000013e808210, 0x000000013e80c410 } _java_thread_list_alloc_cnt=19, _java_thread_list_free_cnt=17, _java_thread_list_max=16, _nested_thread_list_max=0 _tlh_cnt=41, _tlh_times=0, avg_tlh_time=0.00, _tlh_time_max=0 _deleted_thread_cnt=1, _deleted_thread_times=0, avg_deleted_thread_time=0.00, _deleted_thread_time_max=0 _delete_lock_wait_cnt=0, _delete_lock_wait_max=0 _to_delete_list_cnt=0, _to_delete_list_max=1 "main" #1 prio=5 os_prio=31 cpu=86.37ms elapsed=120.17s tid=0x000000014f80da10 nid=7171 in Object.wait() [0x000000016b602000] java.lang.Thread.State: WAITING (on object monitor) Thread: 0x000000014f80da10 [0x1c03] State: _at_safepoint _at_poll_safepoint 0 JavaThread state: _thread_blocked at java.lang.Object.wait(java.base@19-internal/Native Method) - waiting on <0x000000070fc21b00> (a java.lang.Thread) at java.lang.Thread.join(java.base@19-internal/Thread.java:1299) - locked <0x000000070fc21b00> (a java.lang.Thread) at java.lang.Thread.join(java.base@19-internal/Thread.java:1367) at com.sun.javatest.regtest.agent.MainWrapper.main(MainWrapper.java:74) ... "Attach Listener" #20 daemon prio=9 os_prio=31 cpu=0.54ms elapsed=119.92s tid=0x000000013e80c410 nid=39427 waiting on condition [0x0000000000000000] java.lang.Thread.State: RUNNABLE Thread: 0x000000013e80c410 [0x9a03] State: _at_safepoint _at_poll_safepoint 0 JavaThread state: _thread_blocked "VM Thread" os_prio=31 cpu=3.58ms elapsed=120.15s tid=0x000000014f007210 nid=15363 runnable "GC Thread#0" os_prio=31 cpu=0.21ms elapsed=120.17s tid=0x000000014e708a00 nid=18691 runnable "G1 Main Marker" os_prio=31 cpu=0.02ms elapsed=120.17s tid=0x000000014e709350 nid=18179 runnable "G1 Conc#0" os_prio=31 cpu=0.01ms elapsed=120.17s tid=0x000000014e709a70 nid=17667 runnable "G1 Refine#0" os_prio=31 cpu=0.01ms elapsed=120.17s tid=0x000000014e70bc70 nid=12035 runnable "G1 Service" os_prio=31 cpu=18.66ms elapsed=120.17s tid=0x000000014e70c430 nid=16643 runnable "VM Periodic Task Thread" os_prio=31 cpu=71.53ms elapsed=120.11s tid=0x000000014f00a820 nid=25603 waiting on condition JNI global refs: 9, weak refs: 0 --- Timeout information end.
03-06-2022
Where we have: stdout: [99932: ]; stderr: [] exitValue = 134 There should be an entire output from "jcmd", something like: stdout: [69126: Native Memory Tracking: (Omitting categories weighting less than 1KB) Total: reserved=5819680KB, committed=396208KB malloc: 29100KB #99534 mmap: reserved=5790580KB, committed=367108KB - Java Heap (reserved=4194304KB, committed=262144KB) (mmap: reserved=4194304KB, committed=262144KB) - Class (reserved=173KB, committed=173KB) (classes #1179) ( instance classes #1027, array classes #152) (malloc=173KB #1716) ( Metadata: ) ( reserved=65536KB, committed=8064KB) ( used=7948KB) ( waste=116KB =1.44%) ( Class space:) ( reserved=1048576KB, committed=768KB) ( used=707KB) ( waste=61KB =7.89%) - Thread (reserved=45634KB, committed=45634KB) (thread #23) (stack: reserved=45572KB, committed=45572KB) (malloc=37KB #142) (arena=25KB #44) - Code (reserved=248768KB, committed=8656KB) (malloc=1072KB #27634) (mmap: reserved=247696KB, committed=7584KB) - GC (reserved=205942KB, committed=60022KB) (malloc=17110KB #645) (mmap: reserved=188832KB, committed=42912KB) - GCCardSet (reserved=32KB, committed=32KB) (malloc=32KB #387) - Compiler (reserved=213KB, committed=213KB) (malloc=17KB #76) (arena=197KB #5) - Internal (reserved=623KB, committed=623KB) (malloc=591KB #23091) (mmap: reserved=32KB, committed=32KB) - Other (reserved=2KB, committed=2KB) (malloc=2KB #1) - Symbol (reserved=2717KB, committed=2717KB) (malloc=1462KB #38668) (arena=1255KB #1) - Native Memory Tracking (reserved=1888KB, committed=1888KB) (malloc=332KB #3266) (tracking overhead=1555KB) - Arena Chunk (reserved=3984KB, committed=3984KB) (malloc=3984KB) - Test (reserved=896KB, committed=896KB) (malloc=896KB #3) - Module (reserved=167KB, committed=167KB) (malloc=167KB #1215) - Safepoint (reserved=32KB, committed=32KB) (mmap: reserved=32KB, committed=32KB) - Synchronization (reserved=162KB, committed=162KB) (malloc=162KB #2427) - Serviceability (reserved=17KB, committed=17KB) (malloc=17KB #9) - Metaspace (reserved=65548KB, committed=8076KB) (malloc=12KB #12) (mmap: reserved=65536KB, committed=8064KB) - String Deduplication (reserved=1KB, committed=1KB) (malloc=1KB #8) - Object Monitors (reserved=1KB, committed=1KB) (malloc=1KB #6) - Unknown (reserved=1048576KB, committed=768KB) (mmap: reserved=1048576KB, committed=768KB) ]; stderr: [] exitValue = 0
31-05-2022
This core file / backtrace is not from the test itself, but the jcmd, which was launched by the test itself to check whether its memory allocation worked (i.e. "jcmd PID VM.native_memory summary"). This jcmd process appears itself to be stuck here: thread #3 frame #0: 0x00000001aa127614 libsystem_kernel.dylib`read + 8 frame #1: 0x000000010091384c libattach.dylib`Java_sun_tools_attach_VirtualMachineImpl_read + 88 frame #2: 0x0000000113075cd4 frame #3: 0x0000000113072168 frame #4: 0x0000000113072168 frame #5: 0x0000000113072168 frame #6: 0x0000000113072080 frame #7: 0x0000000113072080 frame #8: 0x0000000113072080 frame #9: 0x00000001130722c8 frame #10: 0x000000011306c1bc frame #11: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #12: 0x0000000102c8a650 libjvm.dylib`jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) + 448 frame #13: 0x0000000102c8f10c libjvm.dylib`jni_CallStaticVoidMethod + 312 frame #14: 0x000000010068d000 libjli.dylib`JavaMain + 2524 frame #15: 0x000000010068f2f4 libjli.dylib`ThreadJavaMain + 12 frame #16: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148
31-05-2022
I found the core file and opening it on a M1 machine produces a more exhaustive BT: * thread #1 * frame #0: 0x00000001aa126890 libsystem_kernel.dylib`mach_msg_trap + 8 frame #1: 0x00000001aa126d00 libsystem_kernel.dylib`mach_msg + 76 frame #2: 0x00000001aa2312b0 CoreFoundation`__CFRunLoopServiceMachPort + 372 frame #3: 0x00000001aa22f760 CoreFoundation`__CFRunLoopRun + 1180 frame #4: 0x00000001aa22eb24 CoreFoundation`CFRunLoopRunSpecific + 600 frame #5: 0x000000010068ec14 libjli.dylib`CreateExecutionEnvironment + 400 frame #6: 0x000000010068b14c libjli.dylib`JLI_Launch + 1228 frame #7: 0x000000010055bb8c jcmd`main + 400 frame #8: 0x000000010093108c dyld`start + 520 thread #2 frame #0: 0x00000001aa128814 libsystem_kernel.dylib`__ulock_wait + 8 frame #1: 0x00000001aa1665a0 libsystem_pthread.dylib`_pthread_join + 444 frame #2: 0x000000010068f2ac libjli.dylib`CallJavaMainInNewThread + 148 frame #3: 0x000000010068e188 libjli.dylib`ContinueInNewThread + 144 frame #4: 0x000000010068c0b0 libjli.dylib`JLI_Launch + 5168 frame #5: 0x000000010055bb8c jcmd`main + 400 frame #6: 0x000000010068fb6c libjli.dylib`apple_main + 88 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #3 frame #0: 0x00000001aa127614 libsystem_kernel.dylib`read + 8 frame #1: 0x000000010091384c libattach.dylib`Java_sun_tools_attach_VirtualMachineImpl_read + 88 frame #2: 0x0000000113075cd4 frame #3: 0x0000000113072168 frame #4: 0x0000000113072168 frame #5: 0x0000000113072168 frame #6: 0x0000000113072080 frame #7: 0x0000000113072080 frame #8: 0x0000000113072080 frame #9: 0x00000001130722c8 frame #10: 0x000000011306c1bc frame #11: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #12: 0x0000000102c8a650 libjvm.dylib`jni_invoke_static(JNIEnv_*, JavaValue*, _jobject*, JNICallType, _jmethodID*, JNI_ArgumentPusher*, JavaThread*) + 448 frame #13: 0x0000000102c8f10c libjvm.dylib`jni_CallStaticVoidMethod + 312 frame #14: 0x000000010068d000 libjli.dylib`JavaMain + 2524 frame #15: 0x000000010068f2f4 libjli.dylib`ThreadJavaMain + 12 frame #16: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #4 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #5 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x0000000102a44758 libjvm.dylib`G1ConcurrentMarkThread::wait_for_next_cycle() + 108 frame #5: 0x0000000102a44524 libjvm.dylib`G1ConcurrentMarkThread::run_service() + 68 frame #6: 0x00000001028c542c libjvm.dylib`ConcurrentGCThread::run() + 36 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #6 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #7 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x0000000102a477b0 libjvm.dylib`G1PrimaryConcurrentRefineThread::wait_for_completed_buffers() + 128 frame #3: 0x0000000102a4719c libjvm.dylib`G1ConcurrentRefineThread::run_service() + 64 frame #4: 0x00000001028c542c libjvm.dylib`ConcurrentGCThread::run() + 36 frame #5: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #6: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #8 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x0000000102ac6194 libjvm.dylib`G1ServiceThread::wait_for_task() + 244 frame #5: 0x0000000102ac6420 libjvm.dylib`G1ServiceThread::run_service() + 40 frame #6: 0x00000001028c542c libjvm.dylib`ConcurrentGCThread::run() + 36 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #9 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x000000010341e40c libjvm.dylib`VMThread::wait_for_operation() + 572 frame #5: 0x000000010341d398 libjvm.dylib`VMThread::loop() + 192 frame #6: 0x000000010341d174 libjvm.dylib`VMThread::run() + 192 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #10 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a5330 libjvm.dylib`Monitor::wait(long long) + 452 frame #4: 0x0000000102d094d4 libjvm.dylib`JVM_WaitForReferencePendingList + 156 frame #5: 0x0000000113075cd4 frame #6: 0x00000001130722c8 frame #7: 0x00000001130722c8 frame #8: 0x000000011306c1bc frame #9: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #10: 0x0000000102bd489c libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 504 frame #11: 0x0000000102bd4a60 libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 112 frame #12: 0x0000000102d06814 libjvm.dylib`thread_entry(JavaThread*, JavaThread*) + 364 frame #13: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #14: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #15: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #16: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #11 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310d84c libjvm.dylib`os::PlatformEvent::park() + 488 frame #3: 0x00000001030dd350 libjvm.dylib`ObjectMonitor::wait(long, bool, JavaThread*) + 2444 frame #4: 0x000000010330a5b4 libjvm.dylib`ObjectSynchronizer::wait(Handle, long, JavaThread*) + 124 frame #5: 0x0000000102ce9e20 libjvm.dylib`JVM_MonitorWait + 740 frame #6: 0x0000000113075cd4 frame #7: 0x00000001130722c8 frame #8: 0x0000000113072080 frame #9: 0x0000000113072080 frame #10: 0x000000011306c1bc frame #11: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #12: 0x0000000102bd489c libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 504 frame #13: 0x0000000102bd4a60 libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 112 frame #14: 0x0000000102d06814 libjvm.dylib`thread_entry(JavaThread*, JavaThread*) + 364 frame #15: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #16: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #17: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #18: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #12 frame #0: 0x00000001aa12870c libsystem_kernel.dylib`__workq_kernreturn + 8 thread #13 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010321e4ac libjvm.dylib`os::signal_wait() + 128 frame #3: 0x00000001030fe964 libjvm.dylib`signal_thread_entry(JavaThread*, JavaThread*) + 360 frame #4: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #5: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #6: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #14 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001031f938c libjvm.dylib`ServiceThread::service_thread_entry(JavaThread*, JavaThread*) + 192 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #15 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x000000010308dcbc libjvm.dylib`MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*) + 172 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #16 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a5330 libjvm.dylib`Monitor::wait(long long) + 452 frame #4: 0x00000001028a3434 libjvm.dylib`CompileQueue::get(CompilerThread*) + 424 frame #5: 0x00000001028a8328 libjvm.dylib`CompileBroker::compiler_thread_loop() + 496 frame #6: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #17 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a5330 libjvm.dylib`Monitor::wait(long long) + 452 frame #4: 0x00000001028a3434 libjvm.dylib`CompileQueue::get(CompilerThread*) + 424 frame #5: 0x00000001028a8328 libjvm.dylib`CompileBroker::compiler_thread_loop() + 496 frame #6: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #18 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001032fbd6c libjvm.dylib`NMethodSweeper::sweeper_loop() + 260 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #19 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001030c9a1c libjvm.dylib`NotificationThread::notification_thread_entry(JavaThread*, JavaThread*) + 200 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #20 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001030c91d0 libjvm.dylib`WatcherThread::sleep() const + 232 frame #5: 0x00000001030c9314 libjvm.dylib`WatcherThread::run() + 228 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #21 frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310da88 libjvm.dylib`os::PlatformEvent::park(long) + 528 frame #3: 0x00000001030dd314 libjvm.dylib`ObjectMonitor::wait(long, bool, JavaThread*) + 2384 frame #4: 0x000000010330a5b4 libjvm.dylib`ObjectSynchronizer::wait(Handle, long, JavaThread*) + 124 frame #5: 0x0000000102ce9e20 libjvm.dylib`JVM_MonitorWait + 740 frame #6: 0x0000000113075cd4 frame #7: 0x00000001130722c8 frame #8: 0x0000000113072080 frame #9: 0x0000000113072808 frame #10: 0x00000001130722c8 frame #11: 0x000000011306c1bc frame #12: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #13: 0x0000000102bd489c libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 504 frame #14: 0x0000000102bd4a60 libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 112 frame #15: 0x0000000102d06814 libjvm.dylib`thread_entry(JavaThread*, JavaThread*) + 364 frame #16: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #17: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #18: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #19: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #22 frame #0: 0x00000001aa12ee64 libsystem_kernel.dylib`__accept + 8 frame #1: 0x000000010266af14 libjvm.dylib`BsdAttachListener::dequeue() + 96 frame #2: 0x000000010266b2c0 libjvm.dylib`AttachListener::dequeue() + 172 frame #3: 0x0000000102669378 libjvm.dylib`attach_listener_thread_entry(JavaThread*, JavaThread*) + 356 frame #4: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #5: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #6: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #23 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #24 frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 It looks like the other BT log that I found was produced on x64 machine that was unable to reproduce the correct symbols from aarch64 core. The lesson to learn here is to use the same CPU architecture when working with core files.
31-05-2022
I used the Xcode profiler to check the CPU core usage, and when only this test is being run, only performance cores are used. Though when ran as part of jtreg framework, all cores are probably used, including the efficient ones. I checked the src code where we create pthreads and I do not see the new QoS APIs used.
31-05-2022
Researching __ulock_wait led me to https://opensource.apple.com/source/xnu/xnu-3789.41.3/bsd/kern/sys_ulock.c.auto.html , which lead me to https://developer.apple.com/library/archive/documentation/Performance/Conceptual/power_efficiency_guidelines_osx/PrioritizeWorkAtTheTaskLevel.html (pthread QoS). Combine this with the fact that we are only observing those lockups on aarch64, which has asymmetrical CPU architecture, and the comment from sys_ulock.c.auto.html: /* * How ulock promotion works: * * There’s a requested policy field on every thread called ‘promotions’, which * expresses which ulock promotions are happening to this thread. * The promotion priority saturates until the promotion count goes to 0. * * We also track effective promotion qos, which is the qos before clamping. * This value is used for promoting a thread that another thread is waiting on, * so that the lock owner reinflates to the right priority after unclamping. * * This also works for non-QoS threads, which can donate base priority to QoS * and non-QoS threads alike. * * ulock wait applies a promotion to the owner communicated through * UL_UNFAIR_LOCK as waiters block, and that promotion is saturated as long as * there is still an owner. In ulock wake, if the waker is still the owner, * then it clears its ownership and drops the boost. It does NOT transfer * ownership/priority boost to the new thread. Instead, it selects the * waiting thread with the highest base priority to be woken next, and * relies on that thread to carry the torch for the other waiting threads. */ makes me suspect QoS related deadlock. I will try to see if I can determine the QoS of hotspot VM threads to see if they all share the same QoS level or not...
18-05-2022
Threads #2 and #3 seem very suspicious, too bad there is no back trace for them: thread #2, stop reason = signal SIGSTOP frame #0: 0x00000001aa128814 libsystem_kernel.dylib`__ulock_wait + 8 thread #3, stop reason = signal SIGSTOP frame #0: 0x00000001aa127614 libsystem_kernel.dylib`read + 8 The "libsystem_kernel.dylib`read" symbol I see popping up all over the place, including the MACH5 hosts lockups.
13-05-2022
I can't seem to be able to find the core file, but I located this log, showing all threads: ---------------------------------------- [2022-05-06 20:30:23] [/System/Volumes/Data/mesos/work_dir/jib-master/install/jpg/infra/builddeps/devkit-macosx/Xcode12.4+1.0/devkit-macosx-Xcode12.4+1.0.tar.gz/Xcode.app/Contents/Developer/usr/bin/lldb, --core, /System/Volumes/Data/mesos/work_dir/slaves/779adf21-f3e5-4e6a-a889-8cc0f9bc6fbb-S47386/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/301e55f4-1738-4211-98e4-0ddaf8706909/runs/b6525d22-66c7-47f2-a5dc-44872ae865fa/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/runtime/NMT/ThreadedMallocTestType/core.99935, /System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-19+22-1520/macosx-aarch64-debug.jdk/jdk-19/fastdebug/bin/java, -o, thread backtrace all, -o, quit] timeout=20000 ---------------------------------------- (lldb) target create "/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-19+22-1520/macosx-aarch64-debug.jdk/jdk-19/fastdebug/bin/java" --core "/System/Volumes/Data/mesos/work_dir/slaves/779adf21-f3e5-4e6a-a889-8cc0f9bc6fbb-S47386/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/301e55f4-1738-4211-98e4-0ddaf8706909/runs/b6525d22-66c7-47f2-a5dc-44872ae865fa/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/runtime/NMT/ThreadedMallocTestType/core.99935" Core file '/System/Volumes/Data/mesos/work_dir/slaves/779adf21-f3e5-4e6a-a889-8cc0f9bc6fbb-S47386/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/301e55f4-1738-4211-98e4-0ddaf8706909/runs/b6525d22-66c7-47f2-a5dc-44872ae865fa/testoutput/test-support/jtreg_open_test_hotspot_jtreg_tier1_runtime/runtime/NMT/ThreadedMallocTestType/core.99935' (arm64) was loaded. (lldb) thread backtrace all * thread #1, stop reason = signal SIGSTOP * frame #0: 0x00000001aa126890 libsystem_kernel.dylib`mach_msg_trap + 8 thread #2, stop reason = signal SIGSTOP frame #0: 0x00000001aa128814 libsystem_kernel.dylib`__ulock_wait + 8 thread #3, stop reason = signal SIGSTOP frame #0: 0x00000001aa127614 libsystem_kernel.dylib`read + 8 thread #4, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 thread #5, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 thread #6, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 thread #7, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 thread #8, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 thread #9, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 thread #10, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 thread #11, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 thread #12, stop reason = signal SIGSTOP frame #0: 0x00000001aa12870c libsystem_kernel.dylib`__workq_kernreturn + 8 thread #13, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010321e4ac libjvm.dylib`os::signal_wait() + 128 frame #3: 0x00000001030fe964 libjvm.dylib`signal_thread_entry(JavaThread*, JavaThread*) + 360 frame #4: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #5: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #6: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #14, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001031f938c libjvm.dylib`ServiceThread::service_thread_entry(JavaThread*, JavaThread*) + 192 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #15, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x000000010308dcbc libjvm.dylib`MonitorDeflationThread::monitor_deflation_thread_entry(JavaThread*, JavaThread*) + 172 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #16, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a5330 libjvm.dylib`Monitor::wait(long long) + 452 frame #4: 0x00000001028a3434 libjvm.dylib`CompileQueue::get(CompilerThread*) + 424 frame #5: 0x00000001028a8328 libjvm.dylib`CompileBroker::compiler_thread_loop() + 496 frame #6: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #17, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a5330 libjvm.dylib`Monitor::wait(long long) + 452 frame #4: 0x00000001028a3434 libjvm.dylib`CompileQueue::get(CompilerThread*) + 424 frame #5: 0x00000001028a8328 libjvm.dylib`CompileBroker::compiler_thread_loop() + 496 frame #6: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #7: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #8: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #9: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #18, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001032fbd6c libjvm.dylib`NMethodSweeper::sweeper_loop() + 260 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #19, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310ea58 libjvm.dylib`os::PlatformMonitor::wait(long) + 244 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001030c9a1c libjvm.dylib`NotificationThread::notification_thread_entry(JavaThread*, JavaThread*) + 200 frame #5: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #20, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310e9d0 libjvm.dylib`os::PlatformMonitor::wait(long) + 108 frame #3: 0x00000001030a50d4 libjvm.dylib`Monitor::wait_without_safepoint_check(long long) + 252 frame #4: 0x00000001030c91d0 libjvm.dylib`WatcherThread::sleep() const + 232 frame #5: 0x00000001030c9314 libjvm.dylib`WatcherThread::run() + 228 frame #6: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #7: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #8: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #21, stop reason = signal SIGSTOP frame #0: 0x00000001aa12a250 libsystem_kernel.dylib`__psynch_cvwait + 8 frame #1: 0x00000001aa16483c libsystem_pthread.dylib`_pthread_cond_wait + 1236 frame #2: 0x000000010310da88 libjvm.dylib`os::PlatformEvent::park(long) + 528 frame #3: 0x00000001030dd314 libjvm.dylib`ObjectMonitor::wait(long, bool, JavaThread*) + 2384 frame #4: 0x000000010330a5b4 libjvm.dylib`ObjectSynchronizer::wait(Handle, long, JavaThread*) + 124 frame #5: 0x0000000102ce9e20 libjvm.dylib`JVM_MonitorWait + 740 frame #6: 0x0000000113075cd4 frame #7: 0x00000001130722c8 frame #8: 0x0000000113072080 frame #9: 0x0000000113072808 frame #10: 0x00000001130722c8 frame #11: 0x000000011306c1bc frame #12: 0x0000000102bd59b4 libjvm.dylib`JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, JavaThread*) + 1228 frame #13: 0x0000000102bd489c libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Klass*, Symbol*, Symbol*, JavaCallArguments*, JavaThread*) + 504 frame #14: 0x0000000102bd4a60 libjvm.dylib`JavaCalls::call_virtual(JavaValue*, Handle, Klass*, Symbol*, Symbol*, JavaThread*) + 112 frame #15: 0x0000000102d06814 libjvm.dylib`thread_entry(JavaThread*, JavaThread*) + 364 frame #16: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #17: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #18: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #19: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #22, stop reason = signal SIGSTOP frame #0: 0x00000001aa12ee64 libsystem_kernel.dylib`__accept + 8 frame #1: 0x000000010266af14 libjvm.dylib`BsdAttachListener::dequeue() + 96 frame #2: 0x000000010266b2c0 libjvm.dylib`AttachListener::dequeue() + 172 frame #3: 0x0000000102669378 libjvm.dylib`attach_listener_thread_entry(JavaThread*, JavaThread*) + 356 frame #4: 0x0000000103356108 libjvm.dylib`JavaThread::thread_main_inner() + 596 frame #5: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #6: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #7: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #23, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 thread #24, stop reason = signal SIGSTOP frame #0: 0x00000001aa1268cc libsystem_kernel.dylib`semaphore_wait_trap + 8 frame #1: 0x00000001031f8694 libjvm.dylib`OSXSemaphore::wait() + 24 frame #2: 0x000000010344e284 libjvm.dylib`WorkerThread::run() + 84 frame #3: 0x00000001033532f4 libjvm.dylib`Thread::call_run() + 544 frame #4: 0x0000000103104ad8 libjvm.dylib`thread_native_entry(Thread*) + 352 frame #5: 0x00000001aa16426c libsystem_pthread.dylib`_pthread_start + 148 (lldb) quit ---------------------------------------- [2022-05-06 20:30:26] exit code: 0 time: 3087 ms ----------------------------------------
13-05-2022
ILW = HLM = P3
10-05-2022