JDK-8204994 : SA might fail to attach to process with "Windbg Error: WaitForEvent failed"
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 11,14,15
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows
  • Submitted: 2018-06-13
  • Updated: 2021-02-03
  • Resolved: 2020-06-02
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 15
11.0.9-oracleFixed 13.0.6Fixed 15 b26Fixed
Related Reports
Blocks :  
JDK-8235846 - Improve WindbgDebuggerLocal implementation
Duplicate :  
JDK-8230417 - TestClhsdbJstackLock.java failed due to "Windbg Error: WaitForEvent failed!"
Relates :  
JDK-8235846 - Improve WindbgDebuggerLocal implementation
Relates :  
JDK-8215042 - Umbrella bug for all SA failures in tier1
Relates :  
JDK-8230731 - SA tests fail with "Windbg Error: ReadVirtual failed"
Sub Tasks
JDK-8245660 :  
Try to recover from "Windbg Error: WaitForEvent failed" error - Resolved
Description
The failure is very intermittent. Observed in tests:
serviceability/sa/TestJhsdbJstackLock.java
serviceability/sa/TestInstanceKlassSize.java

The log is:
Error attaching to process: Windbg Error: WaitForEvent failed!
sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: WaitForEvent failed!
	at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach0(Native Method)
	at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach(WindbgDebuggerLocal.java:148)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerWin32(HotSpotAgent.java:568)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:336)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:259)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:450)


 stdout: [Attaching to process ID 16284, please wait...
];
 stderr: [Error attaching to process: Windbg Error: WaitForEvent failed!
sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: WaitForEvent failed!
	at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach0(Native Method)
	at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach(WindbgDebuggerLocal.java:148)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerWin32(HotSpotAgent.java:568)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:336)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
	at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:185)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
	at jdk.hotspot.agent/sun.jvm.hotspot.tools.JStack.runWithArgs(JStack.java:90)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJSTACK(SALauncher.java:259)
	at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:450)

]
 exitValue = 1
Comments
Fix request (13u) Requesting backport to 13u for parity with 11u. The patch applies cleanly. Tested with tier1.
18-01-2021
jdk11 backport request I would like to have the patch in openjdk11 as well, for better parity with 11.0.9-oracle. The patch applies cleanly.
20-07-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/f14ba77ffeba User: amenkov Date: 2020-06-02 20:22:07 +0000
02-06-2020
dbgeng.h contains promissing define DEBUG_ATTACH_NONINVASIVE_ALLOW_PARTIAL: // When doing a non-invasive attach the engine must // recover information for all debuggee elements. The // engine may not have permissions for all elements, // for example it may not be able to open all threads, // and that would ordinarily block the attach. This // flag allows unusable elements to be ignored. #define DEBUG_ATTACH_NONINVASIVE_ALLOW_PARTIAL 0x00000020 Unforrtunately msdn tells nothing about the value and IDebugClient::AttachProcess fails with 0x80070057 (E_INVALIDARG) with this value
19-05-2020
I think we should retry the WaitForEvent() and see if it succeeds. In fact a good way to check if this is helping is to still fail the test, but with a message indicating that the retry succeeded. If we see failures but the log indicates that the retry succeeded, then this is probably the solution. If the retry is failing, then we should detach first: ptrIDebugClient->DetachProcesses() And then retry both the AttachProcess() and the WaitForEvent()
01-05-2020
for COM object - IDebugSymbols::AddSymbolOptions https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/dbgeng/nf-dbgeng-idebugsymbols-addsymboloptions
21-02-2020
SymSetOptions function allows to set "secure mode" (SYMOPT_SECURE option) https://docs.microsoft.com/en-us/windows/win32/api/dbghelp/nf-dbghelp-symsetoptions
21-02-2020
Tried to find any info about secure mode - looks like this is WinDbg feature. I didn't find any mentions about secure mode in COM debug interfaces.
20-02-2020
The docs for WaitForEvent() don't directly mention E_ACCESSDENIED: https://docs.microsoft.com/en-us/windows-hardware/drivers/ddi/dbgeng/nf-dbgeng-idebugcontrol3-waitforevent But does have a link to other possible error codes: https://docs.microsoft.com/en-us/windows-hardware/drivers/debugger/hresult-values On this page you'll find: HRESULT_FROM_WIN32(ERROR_ACCESS_DENIED) The operation was denied because the debugger is in Secure Mode. I assume this is the same as E_ACCESSDENIED. So I wonder if somehow Secure Mode got enabled. However, if that was the case then it seems if one test in the run failed due to this, then we should see a whole bunch fail, yet that does not seem to be the case. We typically at most see just one of SA tests fail in this manner in any given run.
13-02-2020
The latest failure (with improved error reporting) logged 0x80070005 error (this is E_ACCESSDENIED - general "access denied" error)
13-02-2020
Need to improve error reporting to get at least COM error value
11-12-2019
Bumping to P3 to see if this can get some attention.
21-10-2019
Also spotted in this test: serviceability/sa/ClhsdbFlags.java ----------System.out:(12/576)---------- Starting ClhsdbFlags basic test Started LingeredApp with pid 48372 Starting clhsdb against 48372 [2019-08-28T17:27:00.916998100Z] Gathering output for process 43928 [2019-08-28T17:27:02.984699100Z] Waiting for completion for process 43928 [2019-08-28T17:27:02.984699100Z] Waiting for completion finished for process 43928 hsdb> Command not valid until attached to a VM hsdb> Command not valid until attached to a VM hsdb> Command not valid until attached to a VM hsdb> Command not valid until attached to a VM hsdb> Command not valid until attached to a VM hsdb> ----------System.err:(50/3359)*---------- Command line: ['c:\\ade\\mesos\\work_dir\\jib-master\\install\\jdk-14+12-407\\windows-x64-debug.jdk\\jdk-14\\fastdebug/bin/java' '-XX:+UnlockExperimentalVMOptions' '-XX:+UnlockDiagnosticVMOptions' '-XX:-MaxFDLimit' '-XX:MaxRAMPercentage=3' '-cp' 'T:\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_tier1_serviceability\\classes\\5\\serviceability\\sa\\ClhsdbFlags.d;T:\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_tier1_serviceability\\classes\\5\\test\\lib' 'jdk.test.lib.apps.LingeredApp' '5c8c296f-a09d-4ecd-be14-ef9cd6ac7677.lck' ] Attaching to process 48372, please wait... Unable to connect to process ID 48372: Windbg Error: WaitForEvent failed! sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: WaitForEvent failed! at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach0(Native Method) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.windbg.WindbgDebuggerLocal.attach(WindbgDebuggerLocal.java:148) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:672) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerWin32(HotSpotAgent.java:568) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:336) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:141) at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.attachDebugger(CLHSDB.java:180) at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.run(CLHSDB.java:61) at jdk.hotspot.agent/sun.jvm.hotspot.CLHSDB.main(CLHSDB.java:40) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runCLHSDB(SALauncher.java:270) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406) stdout: [ Command not valid until attached to a VM ]; stderr: [ Command not valid until attached to a VM ] exitValue = -1 LingeredApp stdout: []; LingeredApp stderr: [] LingeredApp exitValue = 0 java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'command line' missing from stdout/stderr at ClhsdbFlags.runBasicTest(ClhsdbFlags.java:90) at ClhsdbFlags.main(ClhsdbFlags.java:143) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:127) at java.base/java.lang.Thread.run(Thread.java:830) Caused by: java.lang.RuntimeException: 'command line' missing from stdout/stderr at jdk.test.lib.process.OutputAnalyzer.shouldMatch(OutputAnalyzer.java:306) at ClhsdbLauncher.runCmd(ClhsdbLauncher.java:148) at ClhsdbLauncher.run(ClhsdbLauncher.java:196) at ClhsdbFlags.runBasicTest(ClhsdbFlags.java:86) ... 7 more JavaTest Message: Test threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'command line' missing from stdout/stderr JavaTest Message: shutting down test STATUS:Failed.`main' threw exception: java.lang.RuntimeException: Test ERROR java.lang.RuntimeException: 'command line' missing from stdout/stderr ----------rerun:(35/4860)*----------
28-08-2019