JDK-8248252 : 4 sun/tools/jhsdb tests failed with "RuntimeException: Unable to deduce type of thread"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 16
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: os_x_10.15
  • CPU: x86_64
  • Submitted: 2020-06-24
  • Updated: 2021-07-30
  • Resolved: 2021-07-30
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdResolved
Related Reports
Duplicate :  
JDK-8269830 - SA's vm object vtable matching code sometimes matches on incorrect type
Duplicate :  
JDK-8247940 - SA tests failed with "Error attaching to process: Doesn't appear to be a HotSpot VM"
Duplicate :  
JDK-8259073 - serviceability/sa/ClhsdbFindPC.java#id0 failed with (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process)
Relates :  
JDK-8251852 - 3 SA tests fail with "does not appear to be polymorphic" error
Relates :  
JDK-8253074 - SA tests time out on macOS 10.15.[67]
Description
The following tests failed in the JDK16 CI:

sun/tools/jhsdb/JShellHeapDumpTest.java
sun/tools/jhsdb/HeapDumpTestWithActiveProcess.java
sun/tools/jhsdb/HeapDumpTest.java
sun/tools/jhsdb/BasicLauncherTest.java

Here's a snippet from the JShellHeapDumpTest.java log file:

#section:main
----------messages:(5/259)----------
command: main JShellHeapDumpTest
reason: User specified action: run main/timeout=240 JShellHeapDumpTest 
Mode: othervm [test needs --add-modules]
Additional options from @modules: --add-modules jdk.hotspot.agent,java.management
elapsed time (seconds): 69.832
----------configuration:(3/61)----------
Boot Layer
  add modules: jdk.hotspot.agent java.management

----------System.out:(45/2819)----------
Target JDK is not signed.
Starting Jshell
[JShell]:/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-16+3-50/macosx-x64-debug.jdk/jdk-16/fastdebug/bin/jshell
[JShell] |  Welcome to JShell -- Version 16-ea
[JShell] |  For an introduction type: /help intro
Jshell Started in 3092ms
Starting jmap against 36753
Adding 'sudo -E -n' to the command.
[2020-06-24T05:12:11.327967Z] Gathering output for process 36766
[2020-06-24T05:12:19.069322Z] Waiting for completion for process 36766
[2020-06-24T05:12:19.069488Z] Waiting for completion finished for process 36766
Output and diagnostic info for process 36766 was saved into 'pid-36766-output.log'
jhsdb jmap stdout:
Attaching to process ID 36753, please wait...
Debugger attached successfully.
Server compiler detected.
JVM version is 16-ea+3-50

jhsdb jmap stderr:
java.lang.RuntimeException: Unable to deduce type of thread from address 0x00007fd05b809020 (expected type JavaThread, CompilerThread, ServiceThread, JvmtiAgentThread or CodeCacheSweeperThread)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:187)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Threads.getJavaThreadAt(Threads.java:169)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.dumpStackTraces(HeapHprofBinWriter.java:712)
	at jdk.hotspot.agent/sun.jvm.hotspot.utilities.HeapHprofBinWriter.write(HeapHprofBinWriter.java:439)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.writeHeapHprofBin(JMap.java:182)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.run(JMap.java:97)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.startInternal(Tool.java:262)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:225)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:331)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:483)
Caused by: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x00007fd05b809020
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.InstanceConstructor.newWrongTypeException(InstanceConstructor.java:62)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.VirtualConstructor.instantiateWrapperFor(VirtualConstructor.java:80)
	at jdk.hotspot.agent/sun.jvm.hotspot.runtime.Threads.createJavaThreadWrapper(Threads.java:183)
	... 11 more

###### End of all output:
[2020-06-24T05:12:19.090086Z] Waiting for completion for process 36766
[2020-06-24T05:12:19.090194Z] Waiting for completion finished for process 36766
[2020-06-24T05:12:19.090318Z] Waiting for completion for process 36766
[2020-06-24T05:12:19.090401Z] Waiting for completion finished for process 36766
Destroying jshell
Jshell destroyed
----------System.err:(49/3186)----------

JShellHeapDumpTest.java, HeapDumpTestWithActiveProcess.java and
HeapDumpTest.java fail in a similar way.


BasicLauncherTest.java fails a little differently:

----------System.err:(55/3405)----------
Attaching to process 36601, please wait...
Unable to connect to process ID 36601:

Doesn't appear to be a HotSpot VM (could not find symbol
"gHotSpotVMStructEntryTypeNameOffset" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:431)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:330)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:139)
	at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.attachDebugger(CLHSDB.java:180)
	at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:61)
	at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:40)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runCLHSDB(SALauncher.java:280)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:483)
finding class loader instances ..done.
computing per loader stat ..done.
please wait.. computing liveness..done.
Error attaching to process: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process)
sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupVM(HotSpotAgent.java:431)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:330)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:139)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:187)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JSnap.main(JSnap.java:87)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSNAP(SALauncher.java:352)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:483)

 stdout: [Attaching to process ID 36660, please wait...
];
 stderr: []
 exitValue = 1

java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'java.threads' missing from stdout/stderr 

	at BasicLauncherTest.launch(BasicLauncherTest.java:180)
	at BasicLauncherTest.launch(BasicLauncherTest.java:190)
	at BasicLauncherTest.main(BasicLauncherTest.java:204)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:64)
	at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.base/java.lang.reflect.Method.invoke(Method.java:564)
	at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127)
	at java.base/java.lang.Thread.run(Thread.java:832)
Caused by: java.lang.RuntimeException: 'java.threads' missing from stdout/stderr 

	at jdk.test.lib.process.OutputAnalyzer.shouldContain(OutputAnalyzer.java:189)
	at BasicLauncherTest.launch(BasicLauncherTest.java:175)
	... 8 more

JavaTest Message: Test threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'java.threads' missing from stdout/stderr 

JavaTest Message: shutting down test

STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'java.threads' missing from stdout/stderr
----------rerun:(39/6214)*----------
Comments
I think the original "RuntimeException: Unable to deduce type of thread" issue was fixed by JDK-8269830. The 'could not find symbol "gHotSpotVMTypeEntrySuperclassNameOffset"' issue looks to be a dup of JDK-8247940, so I'm closing this as a dup.
30-07-2021
This symbolicator lookup failure also seems to be the cause of JDK-8251852, which results in tests failing with "does not appear to be polymorphic" because the vtable cannot be symbolically looked up.
31-08-2020
On OSX we rely on the "Symbolicator", which is something for which I've been unable to find any public documentation. This is how we locate the symbolicator: id symbolicator = nil; id jrsSymbolicator = objc_lookUpClass("JRSSymbolicator"); if (jrsSymbolicator != nil) { id (*dynamicCall)(id, SEL, pid_t) = (id (*)(id, SEL, pid_t))&objc_msgSend; symbolicator = dynamicCall(jrsSymbolicator, @selector(symbolicatorForPid:), (pid_t)jpid); } if (symbolicator != nil) { CFRetain(symbolicator); // pin symbolicator while in java heap } And this how we do a symbol to address lookup using the symbolicator: id symbolicator = getSymbolicator(env, this_obj); if (symbolicator != nil) { uint64_t (*dynamicCall)(id, SEL, NSString *) = (uint64_t (*)(id, SEL, NSString *))&objc_msgSend; address = (jlong) dynamicCall(symbolicator, @selector(addressForSymbol:), symbolNameString); } And the only API info I can find is in some 9 year JSRSymbolicator.h file buried in my mac. @interface JRSSymbolicator : NSObject + (JRSSymbolicator *)symbolicatorForPid:(pid_t)pid; - (uint64_t)addressForSymbol:(NSString *)symbolName; @end Update: looks like this file is current and is in more recent versions of XCode. I guess it just hasn't changed in a long time, thus the old copyright. In any case, I suspect this support is broken on 10.15.5, or at least somewhat flaky. The fact that it is failing with different symbols on different runs indicates that it might be mostly working, but has some bugs in it now.
24-06-2020
It was JDK-8247940, although the symbol was different: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process) sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMTypes" in remote process) Was also with a 10.15.5 machine.
24-06-2020
I missed the following in the log: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process) sun.jvm.hotspot.debugger.DebuggerException: Doesn't appear to be a HotSpot VM (could not find symbol "gHotSpotVMStructEntryTypeNameOffset" in remote process) We have another recent bug with this issue. I'll try to find it.
24-06-2020
I missed that NotficationThread mapping was added to SA. I wonder if there is some other thread type we are missing.
24-06-2020
I bet the thread's type is the somewhat new NotificationThread, which SA does not know about. What I don't know is why it just now is turning up.
24-06-2020