Bug ID: JDK-8202564 java/lang/management/ThreadMXBean/ThreadCounts.java fails

JDK 11
11 b13Fixed

2 tests failed in my testing of patch with Graal as JIT which is not related to the fix.
09-05-2018
java.lang.management tests passed
08-05-2018
http://cr.openjdk.java.net/~mdoerr/8202564_thread_cnt/webrev.00/
08-05-2018
Problem is that _exiting_daemon_threads_count is handled inconsistently in ThreadService::remove_thread. I'll post a fix.
08-05-2018
Since the problem occurs without Graal as JIT, I believe there's something wrong with the PerfData: static jlong get_live_thread_count() { return _live_threads_count->get_value() - _exiting_threads_count; } static jlong get_daemon_thread_count() { return _daemon_threads_count->get_value() - _exiting_daemon_threads_count; } The test should work with C1 and C2. I'll take a look.
08-05-2018
But as Martin said you can't rely on this property because with Graal as JIT it will not work. Again - add -XX:-UseDynamicNumberOfCompilerThreads to test.
07-05-2018
Native (c1 and c2) compiler threads can't call java - they should be hidden: bool CompilerThread::can_call_java() const { return _compiler != NULL && _compiler->is_jvmci(); }
07-05-2018
Compiler threads are Java threads (they always were and before JVMCI/Graal): class CompilerThread : public JavaThread { I concur Martin's suggestion to add -XX:-UseDynamicNumberOfCompilerThreads to test.
07-05-2018
I don't think JVMCI or Graal was in use in any of the runs that I looked at or ran in Mach5 over the weekend. I suspect that the change in the Compiler Thread lifecycle has changed the way that the M&M code is counting daemon threads. Keep in mind that this failure is intermittent so what ever is wrong with the daemon thread counting does not happen every time.
07-05-2018
JIT Compiler Threads use: bool is_hidden_from_external_view() const { return !can_call_java(); } JVMCI Compiler Threads can call Java (Graal). So I'd say they're no longer hidden if a JVMCI compiler is used.
07-05-2018
Are CompilerThreads no longer "hidden" threads?
07-05-2018
Thanks for reporting. I guess the problem is that JIT Compiler Threads are Java Deamon Threads and the ThreadMXBean includes their count. So the test relies on numbers which include a component which is controlled by the VM itself and not by the Java code. So it looks like the test makes invalid assumtions. A quick workaround could be to use -XX:-UseDynamicNumberOfCompilerThreads for this test. I'm not sure if there's a better way to deal with it.
07-05-2018
[~kvn] - Vladimir, sending this bug your way since it looks like the fix for JDK-8198756 caused this regression. [~mdoerr] - Martin, please coordinate with Vladimir for how you guys want to deal with this issue. These failures are showing up intermittently in both jdk/jdk nightly and in Mach5 adhoc jobs.
07-05-2018
This test failure appears to be caused by JDK-8198756. I've added the 'regression' label and I've bumped the priority from 'P3' -> 'P2'.
07-05-2018
Note: Sorry for the earlier confusing entries that went down a rabbit hole in the Threads::remove_thread() code. I had noticed the use of a PerfVariable on my first walk down the code and should have taken a closer look at this first anomaly instead of chasing into the code that was counting exiting threads... At this point, I think I've deleted all the confusing entries.
04-05-2018
Original messages: ----------System.out:(6/193)---------- Number of daemon threads added = 21 Number of threads added = 11 Daemon threads terminated. Number of threads added = 9 Number of threads terminated = 11 Number of threads terminated = 9 ----------System.err:(18/1336)---------- TEST FAILED: Minimal number of daemon thread count is 21ThreadMXBean.getDaemonThreadCount() returned 20 TEST FAILED: Minimal number of daemon thread count is 21ThreadMXBean.getDaemonThreadCount() returned 20 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 java.lang.RuntimeException: TEST FAILED. at ThreadCounts.main(ThreadCounts.java:173) Messages are printed on stdout and stderr. Here's a merge of the messages: Number of daemon threads added = 21 TEST FAILED: Minimal number of daemon thread count is 21ThreadMXBean.getDaemonThreadCount() returned 20 Number of threads added = 11 TEST FAILED: Minimal number of daemon thread count is 21ThreadMXBean.getDaemonThreadCount() returned 20 Daemon threads terminated. TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 Number of threads added = 9 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 Number of threads terminated = 11 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 Number of threads terminated = 9 TEST FAILED: Minimal number of daemon thread count is 0ThreadMXBean.getDaemonThreadCount() returned -1 There are six calls to test.checkDaemon() and there are six "TEST FAILED" messages from test.checkDaemon() so every call failed. The other check functions did not fail so that indicates that one of the daemon threads did not die unexpectedly. If a thread did die unexpectedly, then I would expect test.checkCount() to also fail since it used to check the current running thread count: :g/test.checkCount/p if ( (!test.checkCount (DAEMON_THREADS)) \|\| if ( (!test.checkCount (DAEMON_THREADS + USER_THREADS_1)) \|\| if ( (!test.checkCount (USER_THREADS_1)) \|\| if ( (!test.checkCount (USER_THREADS_1 + USER_THREADS_2)) \|\| if ( (!test.checkCount (USER_THREADS_2)) \|\| if ( (!test.checkCount (0)) \|\| Here are the test.checkDaemon() calls: :g/test.checkDaemon/p (!test.checkDaemon (DAEMON_THREADS)) (!test.checkDaemon (DAEMON_THREADS)) (!test.checkDaemon (0)) (!test.checkDaemon (0)) (!test.checkDaemon (0)) (!test.checkDaemon (0)) The first and second calls are short by one and the last four calls return a negative number. This looks like the isDaemon() flag for one of the daemon threads is not seen when the thread is first created, but that flag is seen when the daemon thread exits. Here's the daemon thread start sequence: // Start DAEMON_THREADS threads and wait to be sure they all are alive barrier.set(DAEMON_THREADS); for (int i = 0; i < DAEMON_THREADS; i++) { live[i] = true; allThreads[i] = new MyThread(i); allThreads[i].setDaemon(true); allThreads[i].start(); } // wait until all threads have started. barrier.await(); so we very carefully call setDaemon(true) before we start the daemon thread. Here is the checkDaemon() code: private boolean checkDaemon(long min) { long result = mbean.getDaemonThreadCount(); if (result < min) { System.err.println("TEST FAILED: " + "Minimal number of daemon thread count is " + min + "ThreadMXBean.getDaemonThreadCount() returned " + result); return false; } return true; } Time to take a closer look at mbean.getDaemonThreadCount(): $ grep -r getDaemonThreadCount src/java.management src/java.management/share/classes/java/lang/management/ThreadMXBean.java: public int getDaemonThreadCount(); src/java.management/share/classes/sun/management/ThreadImpl.java: public int getDaemonThreadCount() { src/java.management/share/classes/sun/management/ThreadImpl.java: return jvm.getDaemonThreadCount(); src/java.management/share/classes/sun/management/VMManagement.java: public int getDaemonThreadCount(); src/java.management/share/classes/sun/management/VMManagementImpl.java: public native int getDaemonThreadCount(); src/java.management/share/native/libmanagement/VMManagementImpl.c:Java_sun_management_VMManagementImpl_getDaemonThreadCount All that boils down to: src/java.management/share/native/libmanagement/VMManagementImpl.c: JNIEXPORT jint JNICALL Java_sun_management_VMManagementImpl_getDaemonThreadCount (JNIEnv env, jobject dummy) { jlong count = jmm_interface->GetLongAttribute(env, NULL, JMM_THREAD_DAEMON_COUNT); return (jint) count; } src/hotspot/share/services/management.cpp: case JMM_THREAD_DAEMON_COUNT: return ThreadService::get_daemon_thread_count(); src/hotspot/share/services/threadService.hpp: static jlong get_daemon_thread_count() { return _daemon_threads_count->get_value() - _exiting_daemon_threads_count; } Here's all of the _daemon_threads_count refs: :g/_daemon_threads_count/p static PerfVariable _daemon_threads_count; static volatile int _exiting_daemon_threads_count; static jlong get_daemon_thread_count() { return _daemon_threads_count->ge t_value() - _exiting_daemon_threads_count; } static int exiting_daemon_threads_count() { return _exiting_daemon_threads_c ount; } We have two fields that are used to manage daemon thread counts: static PerfVariable* _daemon_threads_count; static volatile int _exiting_daemon_threads_count; _exiting_daemon_threads_count is managed via atomic increments and decrements. _daemon_threads_count; is a PerfVariable which is not managed via atomic operations. Gonna have to take a closer look at that.
04-05-2018
This is what a passing run should print out: ----------System.out:(7/200)---------- Number of daemon threads added = 21 Number of threads added = 11 Daemon threads terminated. Number of threads added = 9 Number of threads terminated = 11 Number of threads terminated = 9 Test passed.
03-05-2018