JDK-8193639 : tests failing intermittently with Error attaching to process: Can't create thread_db agent! on Solaris
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 10,11,12
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • OS: solaris
  • CPU: generic
  • Submitted: 2017-12-15
  • Updated: 2023-10-09
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
JDK-7174275 - nsk.sajdi tests fail on Solaris-sparc64 with exception: Can't create thread_db agent!
Duplicate :  
JDK-8196084 - SA: Attach fails due to 'Can't create thread_db agent!'
Duplicate :  
JDK-8205630 - serviceability/sa/ClhsdbFindPC.java fails with ArrayIndexOutOfBoundsException: Index 1 out of bounds for length 1
Duplicate :  
JDK-8205135 - serviceability/sa/ClhsdbScanOops.java failed with 'java/lang/Object' missing from stdout/stderr
Duplicate :  
JDK-8198446 - Test serviceability/sa/TestType.java failed - Can't create thread_db agent!
Relates :  
JDK-8242235 - Disable SA testing on Solaris. Remove JDK-8193639 entries from ProblemList.txt
Relates :  
JDK-8198446 - Test serviceability/sa/TestType.java failed - Can't create thread_db agent!
Relates :  
JDK-8215042 - Umbrella bug for all SA failures in tier1
Relates :  
JDK-8211953 - ClhsdbFindPC.java fails with java.lang.NullPointerException at jdk.hotspot.agent/sun.jvm.hotspot.tools.StackTrace.run
Sub Tasks
JDK-8203689 :  
Add failure code to threaddb exception messages - Resolved
JDK-8208205 :  
ProblemList tests that fail due to 'Error attaching to process: Can't create thread_db agent!' - Resolved
JDK-8208521 :  
ProblemList more tests that fail due to 'Error attaching to process: Can't create thread_db agent!' - Resolved
JDK-8209029 :  
ProblemList tests that fail due to 'Error attaching to process: Can't create thread_db agent!' in jdk-11+25 testing - Resolved
JDK-8209342 :  
Problemlist SA tests on Solaris due to Error attaching to process: Can't create thread_db agent! - Resolved
Description
The following jhsdb test failed in a recent Solaris X64 'release' config:

sun/tools/jhsdb/BasicLauncherTest.java

Here is a snippet of the .jtr file:

----------System.err:(48/2983)----------
Attaching to process 4368, please wait...
javax.script.ScriptException: TypeError: sapkg.runtime.VM.getVM is not a function in sa.js at line number 54
javax.script.ScriptException: TypeError: sapkg.runtime.VM.getVM is not a function in sa.js at line number 54
finding class loader instances ..done.
computing per loader stat ..done.
please wait.. computing liveness..............................................done.
Error attaching to process: Can't create thread_db agent!
sun.jvm.hotspot.debugger.DebuggerException: Can't create thread_db agent!
        at jdk.hotspot.agent/sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach0(Native Method)
        at jdk.hotspot.agent/sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach(ProcDebuggerLocal.java:146)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerSolaris(HotSpotAgent.java:488)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:334)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:326)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:455)

 stdout: [Attaching to process ID 4484, please wait...
];
 stderr: []
 exitValue = 1

java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'compiler detected' missing from stdout/stderr

        at BasicLauncherTest.launch(BasicLauncherTest.java:187)
        at BasicLauncherTest.launch(BasicLauncherTest.java:197)
        at BasicLauncherTest.main(BasicLauncherTest.java:213)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:564)
        at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:115)
        at java.base/java.lang.Thread.run(Thread.java:844)
Caused by: java.lang.RuntimeException: 'compiler detected' missing from stdout/stderr

        at jdk.testlibrary.OutputAnalyzer.shouldContain(OutputAnalyzer.java:106)
        at BasicLauncherTest.launch(BasicLauncherTest.java:182)
        ... 8 more

JavaTest Message: Test threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'compiler detected' missing from stdout/stderr

JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'compiler detected' missing from stdout/stderr


The test reports that it cannot find the 'compiler detected' pattern, but the
root cause is probably this:

Error attaching to process: Can't create thread_db agent!
sun.jvm.hotspot.debugger.DebuggerException: Can't create thread_db agent!
Comments
Note that we no longer problem list tests due to this CR. Instead SA testing is disabled. See JDK-8242235: --- a/test/lib/jdk/test/lib/Platform.java Tue Apr 07 22:31:18 2020 +0200 +++ b/test/lib/jdk/test/lib/Platform.java Tue Apr 07 13:56:53 2020 -0700 @@ -225,6 +225,8 @@ public static boolean hasSA() { if (isAix()) { return false; // SA not implemented. + } else if (isSolaris()) { + return false; // Testing disabled due to JDK-8193639. } else if (isLinux()) { if (isS390x() || isARM()) { return false; // SA not implemented. You will need to undo this change to get the SA tests to run on Solaris again.
07-04-2020
Setting SA_IGNORE_THREADDB, i could see the VMVersionMismatchException thrown, much earlier than we hit with "Can't create thread_db agent". Error is more meaningful only with SA_IGNORE_THREADDB is set, Unable to open core file ./30536297/core-lib-data/core: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.66-b00. Target VM is 25.231-b11 libproc DEBUG: Prelease: releasing handle 1004cbc80 PS_DEAD of pid 4136 rtld_db: rd_delete(0x1004d7f20) sun.jvm.hotspot.debugger.DebuggerException: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.66-b00. Target VM is 25.231-b11 at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:435) at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305) at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156) at sun.jvm.hotspot.CLHSDB.attachDebugger(CLHSDB.java:202) at sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:63) at sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:40) Caused by: sun.jvm.hotspot.runtime.VMVersionMismatchException: Supported versions are 25.66-b00. Target VM is 25.231-b11 at sun.jvm.hotspot.runtime.VM.checkVMVersion(VM.java:227) at sun.jvm.hotspot.runtime.VM.<init>(VM.java:294) at sun.jvm.hotspot.runtime.VM.initialize(VM.java:370) at sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:431) ... 5 more I am able to open core file with usage of right JDK and SA_IGNORE_THREADDB set. I find there are 2 problems here, 1> VMVersionMismatchException should have been reported even SA_IGNORE_THREADDB is not set. So the execution should have stopped much earlier than we hit with "Can't create thread_db agent" 2> When we are sure from jdk-jdk/open/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp#184 // Is it okay to ignore libthread_db failure? Set env var to ignore // libthread_db failure. You can still debug, but will miss threads // related functionality. This error message should be prompted to user and if he/she wish to continue can still use SA_IGNORE_THREADDB =1 and continue with execution. Below is the problematic code. jdk-jdk/open/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp#189 189 #define HANDLE_THREADDB_FAILURE(msg) \ 190 if (sa_ignore_threaddb) { \ 191 printf("libsaproc WARNING: %s\n", msg); \ 192 return; \ 193 } else { \ 194 THROW_NEW_DEBUGGER_EXCEPTION(msg); \ 195 } for sa_ignore_threaddb is set we provide warning and return, else we just exit without any message. I think we can provide meaningful error and exit, such as "There is libthread_db failure occured. Use env var SA_IGNORE_THREADDB=1 to ignore libthread_db failure. You can still debug, but will miss threads related functionality"
16-12-2019
During generation of new thread debug library handle for process td_err_e te = p_td_ta_new(ph, &p_td_thragent_t); We receive te value as 15 /* TD_DBERR, Internal debug library error. */ Analyzing more to find reason for "Internal debug library error", there is one difference found, core file generated on "SunOS 5.10", and i am using "SunOS 5.11, 11.3" to open the core.
16-12-2019
I am able to reproduce this issue consistently on my solaris box /tmp/fmatte/30536297$ /tmp/fmatte/jdk1.8.0_231/bin/java -cp /tmp/fmatte/jdk1.8.0_231/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB /tmp/fmatte/jdk1.8.0_231/bin $ /tmp/fmatte/jdk1.8.0_231/bin/java -cp /tmp/fmatte/jdk1.8.0_231/lib/sa-jdi.jar sun.jvm.hotspot.CLHSDB /tmp/fmatte/jdk1.8.0_231/bin/java ./core-lib-data/core Opening core file, please wait... Unable to open core file ./core-lib-data/core: Can't create thread_db agent! sun.jvm.hotspot.debugger.DebuggerException: Can't create thread_db agent! at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach0(Native Method) at sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach(ProcDebuggerLocal.java:145) at sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:673) at sun.jvm.hotspot.HotSpotAgent.setupDebuggerSolaris(HotSpotAgent.java:487) at sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:333) at sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:304) at sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:156) at sun.jvm.hotspot.CLHSDB.attachDebugger(CLHSDB.java:202) at sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:63) at sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:40)
10-12-2019
[~lmesnik] - The failure mode that this bug is tracking is specific to Solaris. Any failure linked to this bug must have this string: sun.jvm.hotspot.debugger.DebuggerException: Can't create thread_db agent!
05-09-2019
There are a lot of test failures on all platforms and not only on Solaris. So I removed any solaris specific tags from this issue.
25-02-2019
Here are the logs for my jdk-12+17 Solaris-X64 sightings: sun/tools/jhsdb/heapconfig/JMapHeapConfigTest.java $ unzip -l jdk-12+17_solx64.8193639.zip Archive: jdk-12+17_solx64.8193639.zip Length Date Time Name --------- ---------- ----- ---- 88918 10-26-2018 00:57 jdk-12+17_1/JMapHeapConfigTest.jtr.release --------- ------- 88918 1 file
29-10-2018
Here are the logs for my jdk11+24 Solaris-X64 sightings: $ unzip -l jdk-11+24_solx64.8193639.zip Archive: jdk-11+24_solx64.8193639.zip Length Date Time Name --------- ---------- ----- ---- 15995 07-27-2018 06:22 jdk-11+24_1/ClhsdbVmStructsDump.jtr.fastdebug 18509 07-27-2018 06:23 jdk-11+24_1/TestDefaultMethods.jtr.fastdebug 37132 07-28-2018 10:38 jdk-11+24_2/SASymbolTableTest.jtr.release 17446 07-28-2018 08:54 jdk-11+24_2/TestHeapDumpForInvokeDynamic.jtr.fastdebug 19672 07-29-2018 08:01 jdk-11+24_3/ClhsdbThread.jtr.slowdebug 59699 07-29-2018 11:17 jdk-11+24_3/TestType.jtr.fastdebug --------- ------- 168453 6 files Here's the list of tests that failed in my jdk11+24 Solaris-X64 testing: runtime/SharedArchiveFile/SASymbolTableTest.java serviceability/sa/ClhsdbThread.java serviceability/sa/ClhsdbVmStructsDump.java serviceability/sa/TestDefaultMethods.java serviceability/sa/TestHeapDumpForInvokeDynamic.java serviceability/sa/TestType.java Update: The snapshot for jdk-11+24 did not include the fix for JDK-8208205 which is why I'm still executing these tests in my Thread-SMR stress runs. During my jdk-11+24 Solaris-X64 testing, two additional tests failed due to this failure mode: $ diff 8193639.list.0-1_merge 8193639.list.0-2_merge 14a15 > serviceability/sa/ClhsdbVmStructsDump.java 21a23 > serviceability/sa/TestHeapDumpForInvokeDynamic.java I'll file a new bug to update the ProblemList again.
30-07-2018
Here are the logs for my jdk11+23 Solaris-X64 sightings: $ unzip -l jdk-11+23_solx64.8193639.zip Archive: jdk-11+23_solx64.8193639.zip Length Date Time Name --------- ---------- ----- ---- 20233 07-21-2018 03:16 jdk-11+23_1/ClhsdbJstack.jtr.slowdebug 15693 07-21-2018 03:15 jdk-11+23_1/ClhsdbPrintAll.jtr.slowdebug 115241 07-21-2018 07:24 jdk-11+23_1/ClhsdbScanOops.jtr.fastdebug 113564 07-21-2018 08:52 jdk-11+23_1/ClhsdbScanOops.jtr.release 20989 07-21-2018 08:33 jdk-11+23_1/TestType.jtr.release 19704 07-22-2018 12:54 jdk-11+23_2/ClhsdbFindPC.jtr.fastdebug 24919 07-22-2018 12:55 jdk-11+23_2/ClhsdbInspect.jtr.fastdebug 23080 07-22-2018 14:38 jdk-11+23_2/ClhsdbInspect.jtr.release 19380 07-22-2018 14:38 jdk-11+23_2/ClhsdbJdis.jtr.release 20210 07-22-2018 12:54 jdk-11+23_2/ClhsdbJstack.jtr.fastdebug 20460 07-22-2018 12:55 jdk-11+23_2/ClhsdbPrintAs.jtr.fastdebug 15979 07-22-2018 12:54 jdk-11+23_2/ClhsdbPrintStatics.jtr.fastdebug 15513 07-22-2018 14:38 jdk-11+23_2/ClhsdbPstack.jtr.release 115470 07-22-2018 12:55 jdk-11+23_2/ClhsdbRegionDetailsScanOopsForG1.jtr.fastdebug 115690 07-22-2018 13:31 jdk-11+23_2/ClhsdbScanOops.jtr.fastdebug 16655 07-22-2018 14:56 jdk-11+23_2/ClhsdbScanOops.jtr.release 16057 07-22-2018 12:54 jdk-11+23_2/ClhsdbSymbolTable.jtr.fastdebug 20642 07-22-2018 12:55 jdk-11+23_2/ClhsdbThread.jtr.fastdebug 24909 07-23-2018 14:07 jdk-11+23_3/ClhsdbInspect.jtr.fastdebug 15673 07-23-2018 10:38 jdk-11+23_3/ClhsdbPrintAll.jtr.slowdebug 114309 07-23-2018 14:43 jdk-11+23_3/ClhsdbScanOops.jtr.fastdebug --------- ------- 884370 21 files Update: here's the list of tests that failed in my jdk11+23 Solaris-X64 testing: serviceability/sa/ClhsdbFindPC.java serviceability/sa/ClhsdbInspect.java serviceability/sa/ClhsdbJdis.java serviceability/sa/ClhsdbJstack.java serviceability/sa/ClhsdbPrintAll.java serviceability/sa/ClhsdbPrintAs.java serviceability/sa/ClhsdbPrintStatics.java serviceability/sa/ClhsdbPstack.java serviceability/sa/ClhsdbRegionDetailsScanOopsForG1.java serviceability/sa/ClhsdbScanOops.java serviceability/sa/ClhsdbSymbolTable.java serviceability/sa/ClhsdbThread.java serviceability/sa/TestType.java
25-07-2018
Here are the logs for my jdk11+19 sightings: $ unzip -l jdk11_migrate.8193639.zip Archive: jdk11_migrate.8193639.zip Length Date Time Name --------- ---------- ----- ---- 19436 06-23-2018 14:15 jdk11_migrate_1/ClhsdbFindPC.jtr.fastdebug 23203 06-23-2018 14:15 jdk11_migrate_1/ClhsdbInspect.jtr.fastdebug 22901 06-23-2018 09:25 jdk11_migrate_1/ClhsdbInspect.jtr.slowdebug 19215 06-23-2018 09:25 jdk11_migrate_1/ClhsdbJdis.jtr.slowdebug 15013 06-23-2018 09:25 jdk11_migrate_1/ClhsdbPrintAs.jtr.slowdebug 15657 06-23-2018 09:25 jdk11_migrate_1/ClhsdbPrintStatics.jtr.slowdebug 85050 06-23-2018 14:45 jdk11_migrate_1/ClhsdbScanOops.jtr.fastdebug 15754 06-23-2018 14:15 jdk11_migrate_1/ClhsdbSymbolTable.jtr.fastdebug 31017 06-23-2018 16:53 jdk11_migrate_1/TestUniverse_id0.jtr.release 20102 06-25-2018 06:20 jdk11_migrate_2/ClhsdbPrintAs.jtr.fastdebug 95177 06-25-2018 06:21 jdk11_migrate_2/ClhsdbRegionDetailsScanOopsForG1.jtr.fastdebug 116377 06-25-2018 06:50 jdk11_migrate_2/ClhsdbScanOops.jtr.fastdebug 116427 06-25-2018 09:13 jdk11_migrate_2/ClhsdbScanOops.jtr.release 16264 06-25-2018 06:20 jdk11_migrate_2/ClhsdbSymbolTable.jtr.fastdebug 36639 06-25-2018 08:58 jdk11_migrate_2/SASymbolTableTest.jtr.release 19328 06-27-2018 13:37 jdk11_migrate_3/ClhsdbFindPC.jtr.release 22879 06-27-2018 10:58 jdk11_migrate_3/ClhsdbInspect.jtr.fastdebug 22794 06-27-2018 13:37 jdk11_migrate_3/ClhsdbInspect.jtr.release 23208 06-27-2018 06:04 jdk11_migrate_3/ClhsdbInspect.jtr.slowdebug 19196 06-27-2018 10:58 jdk11_migrate_3/ClhsdbJdis.jtr.fastdebug 19110 06-27-2018 13:37 jdk11_migrate_3/ClhsdbJdis.jtr.release 15147 06-27-2018 06:02 jdk11_migrate_3/ClhsdbPmap.jtr.slowdebug 19060 06-27-2018 10:58 jdk11_migrate_3/ClhsdbPrintAs.jtr.fastdebug 19965 06-27-2018 13:38 jdk11_migrate_3/ClhsdbPrintAs.jtr.release 15587 06-27-2018 13:38 jdk11_migrate_3/ClhsdbPrintStatics.jtr.release 114991 06-27-2018 13:38 jdk11_migrate_3/ClhsdbRegionDetailsScanOopsForG1.jtr.release 84631 06-27-2018 13:52 jdk11_migrate_3/ClhsdbScanOops.jtr.release 16172 06-27-2018 13:38 jdk11_migrate_3/ClhsdbSymbolTable.jtr.release 19842 06-27-2018 13:38 jdk11_migrate_3/ClhsdbThread.jtr.release 15262 06-27-2018 13:38 jdk11_migrate_3/ClhsdbWhere.jtr.release 17123 06-27-2018 13:38 jdk11_migrate_3/DeadlockDetectionTest.jtr.release 16599 06-27-2018 13:38 jdk11_migrate_3/JhsdbThreadInfoTest.jtr.release 15869 06-27-2018 13:38 jdk11_migrate_3/TestClassDump.jtr.release 19090 06-27-2018 13:38 jdk11_migrate_3/TestCpoolForInvokeDynamic.jtr.release 18083 06-27-2018 13:38 jdk11_migrate_3/TestDefaultMethods.jtr.release 17070 06-27-2018 13:38 jdk11_migrate_3/TestG1HeapRegion.jtr.release 25656 06-27-2018 13:39 jdk11_migrate_3/TestUniverse_id0.jtr.release --------- ------- 1224894 37 files Update: here's the list of tests that failed in my jdk11+19 Solaris-X64 testing: runtime/SharedArchiveFile/SASymbolTableTest.java serviceability/sa/ClhsdbFindPC.java serviceability/sa/ClhsdbInspect.java serviceability/sa/ClhsdbJdis.java serviceability/sa/ClhsdbPmap.java serviceability/sa/ClhsdbPrintAs.java serviceability/sa/ClhsdbPrintStatics.java serviceability/sa/ClhsdbRegionDetailsScanOopsForG1.java serviceability/sa/ClhsdbScanOops.java serviceability/sa/ClhsdbSymbolTable.java serviceability/sa/ClhsdbThread.java serviceability/sa/ClhsdbWhere.java serviceability/sa/DeadlockDetectionTest.java serviceability/sa/JhsdbThreadInfoTest.java serviceability/sa/TestClassDump.java serviceability/sa/TestCpoolForInvokeDynamic.java serviceability/sa/TestDefaultMethods.java serviceability/sa/TestG1HeapRegion.java serviceability/sa/TestUniverse.java
25-07-2018
Bug JDK-8205630 closed as a dup of this bug might cause tier1 test failures.
23-07-2018
Thanks [~dcubed] - unfortunately the minimal debug output seems to shed no light.
02-07-2018
[~dholmes]] - These are the logs from your requested debug run: $ grep thread_db 8193639.debug.0/* | sed 's/:.*//' | sort -u 8193639.debug.0/ClhsdbCDSJstackPrintAll.jtr.slowdebug 8193639.debug.0/ClhsdbInspect.jtr.slowdebug 8193639.debug.0/ClhsdbJdis.jtr.slowdebug 8193639.debug.0/ClhsdbScanOops.jtr.fastdebug 8193639.debug.0/ClhsdbScanOops.jtr.release I've attached the logs as 8193639.debug.0.zip There are a few additional messages from libsaproc, but not a lot. Hopefully they will help.
29-06-2018
[~dholmes] - I can do that experiment as soon as my current Thread-SMR stress run is done. I'm on run #3 for the jdk11+19 snapshot and that's the last one. My reproducibility might have something to do with my running with 16 parallel jobs on my Solaris X64 server.
27-06-2018
That's more than what anyone else seems to be able to do. :) Can you try applying the following patch to turn on debugging output and see if the failing tests show any additional info that may help? diff -r afbcf72c389d src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp --- a/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp +++ b/src/jdk.hotspot.agent/solaris/native/libsaproc/saproc.cpp @@ -1281,6 +1281,7 @@ JNIEXPORT void JNICALL Java_sun_jvm_hotspot_debugger_proc_ProcDebuggerLocal_initIDs (JNIEnv *env, jclass clazz) { _libsaproc_debug = getenv("LIBSAPROC_DEBUG") != NULL; + _libsaproc_debug = 1; if (_libsaproc_debug) { // propagate debug mode to libproc.so static const char* var = "LIBPROC_DEBUG=1"; diff -r afbcf72c389d test/hotspot/jtreg/serviceability/sa/JhsdbThreadInfoTest.java --- a/test/hotspot/jtreg/serviceability/sa/JhsdbThreadInfoTest.java +++ b/test/hotspot/jtreg/serviceability/sa/JhsdbThreadInfoTest.java @@ -82,7 +82,7 @@ for (String line : lines) { Matcher m = p.matcher(line); if (!m.matches()) { - throw new RuntimeException("Stderr has output other than VM warnings"); +// throw new RuntimeException("Stderr has output other than VM warnings"); } } } diff -r afbcf72c389d test/hotspot/jtreg/serviceability/sa/TestJhsdbJstackLock.java --- a/test/hotspot/jtreg/serviceability/sa/TestJhsdbJstackLock.java +++ b/test/hotspot/jtreg/serviceability/sa/TestJhsdbJstackLock.java @@ -88,7 +88,7 @@ for (String line : lines) { Matcher m = p.matcher(line); if (!m.matches()) { - throw new RuntimeException("Stderr has output other than VM warnings"); + // throw new RuntimeException("Stderr has output other than VM warnings"); } } } Thanks!
27-06-2018
Don't know about "reliable", but I just had a raft of these failures show up in my first round of Thread-SMR stress testing using jdk11+19 bits on my Solaris-X64 server. I still have to update existing bugs and file some new bugs...
26-06-2018
Thanks [~dcubed] that reinforces the point I was making. Can you still reproduce this in a reliable way?
26-06-2018
[~dholmes] > Note that while this is only reported on sparc ... I filed this bug against Solaris X64 since I originally spotted it on the Solaris-X64 server in my lab so it is not "only reported on sparc".
25-06-2018
I couldn't reproduce locally either so I've made a change to always enable _libsaproc_debug and submitted a mach5 job to run the sa tests 10 time across the 4 solaris platform variants. First lesson: turning on the debug output breaks a couple of the tests: Stderr has output other than VM warnings serviceability/sa/TestJhsdbJstackLock.java serviceability/sa/JhsdbThreadInfoTest.java After fixing that I ran all the serviceability/sa tests, 10 times for each of product/debug and x86/sparc across a range of systems and got ZERO failures. So whatever this is it isn't a simple machine specific issue, it requires the right circumstances to trigger the bug. (Note that while this is only reported on sparc we don't test Solaris x64 normally in mach5 - so it's unlikely this is a sparc-only issue). I then selected one failing test, ClhsdbJdis.java, and ran it 100 times for each of product/debug and x86/sparc across a range of systems and got ZERO failures.
25-06-2018
Unfortunately there are no longer any detailed logs pertaining to JDK-7174275 so we can't see if it was the same underlying issue.
25-06-2018
Given our standardised testing environment it is not surprising that we only see this in Solaris 11.2 and 11.3. Do we even have any other Solaris versions being used for JDK 11 mach 5 testing?
25-06-2018
As per JDK-7174275, this issue was seen with 5u61 too.
24-06-2018
One observation is that all the failures seen so far have been on Solaris 11.3 and 11.2
24-06-2018
I've been unable to reproduce locally on Solaris x86.
24-06-2018
Can we determine if this started failing due to a product change or an OS change? can the test run in an older JDK?
21-06-2018
From the above failure the following exception message is being printed: " Can't create thread_db agent! td_ta_new failed: 15" So, td_ta_new() is failing with 15 (TD_DBERR). The description for TD_DBERR is: TD_DBERR, /* Internal debug library error. */ The failure is due to internal debug library error in Solaris.
31-05-2018
Ran ClhsdbJstack.java 500 times, did not see the problem. Have made code changes to print the error code in case of failure. Will use this bug to checkin the same. When the problem is hit again, we will know reason.
20-05-2018
td_ta_new() fails and from the man page, it could be due to any of these reasons. TD_BADPH A NULL external process handle was passed to td_ta_new(). TD_ERR The ta_pp argument is NULL or an internal error occurred. TD_DBERR A call to one of the imported interface routines failed. TD_MALLOC Memory allocation failure. TD_NOLIBTHREAD The target process does not appear to be multi- threaded. Would need to add perror() to capture the cause for failure.
16-05-2018
Setting the env variable SA_IGNORE_THREADDB might work as a workaround. It might not work for the thread related test cases though.
15-05-2018
The following test: runtime/SharedArchiveFile/SASymbolTableTest.java failed with this snippet in the release config: Error attaching to process: Can't create thread_db agent! sun.jvm.hotspot.debugger.DebuggerException: Can't create thread_db agent! at jdk.hotspot.agent/sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach0(Native Method) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.proc.ProcDebuggerLocal.attach(ProcDebuggerLocal.java:146) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerSolaris(HotSpotAgent.java:488) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:334) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at SASymbolTableTestAgent.main(SASymbolTableTestAgent.java:46) [STDOUT] Attaching to process ID 18586, please wait... I was doing Thread-SMR stress testing on Solaris-X64 using jdk/jdk10 bits at these changesets: $ hg tlog -r tip [/work/shared/bug_hunt/8167108/SMR_base_10]: changeset: 28351:7c216eb69805 tag: tip user: lana date: Tue Feb 20 17:01:15 2018 +0000 summary: Added tag jdk-10+44 for changeset 968ee3e65093 [/work/shared/bug_hunt/8167108/SMR_base_10/open]: changeset: 48546:ed9ff662b1e0 tag: tip user: lana date: Tue Feb 20 17:01:18 2018 +0000 summary: Added tag jdk-10+44 for changeset 663f20fc5109
27-02-2018
Update: Just before the Winter holiday I changed my JTREG test execution execution script to use "make run-test" instead of directly invoking jtreg. I also updated my Thread-SMR repos to the 2017.12.14 JDK10 snapshot to verify Thread-SMR testing for the JDK10/18.3 fork. I ran 9 Thread-SMR stress runs over the Winter holiday and did not see this test fail at all: $ grep sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_2017121[34]* do_bld_and_test_all.log.jdk10_hs_20171214:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo2:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo2:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo2:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo3:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo3:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo3:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo4:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo4:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo4:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo5:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo5:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo5:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo6:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo6:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo6:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo7:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo7:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo7:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo8:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo8:Passed: sun/tools/jhsdb/BasicLauncherTest.java do_bld_and_test_all.log.jdk10_hs_20171214_redo8:Passed: sun/tools/jhsdb/BasicLauncherTest.java The failure was intermittent, but I expected it to show up at least once in this round of Thread-SMR stress testing. I suspect that changing test invocation styles may have "fixed" this, but I have not proven that.
02-01-2018
I have attached the .jtr file from the 'release' bits failure.
15-12-2017