It seems to be somewhat common for SA to fail in various way, but the tests using SA never detect the failure because the info they are looking for is still in the log file. One example is ClhsdbScanOops, which has the following SA failure on every run:

0x00000006c6eb45b8 java/lang/String
0x00000006c6eb45f0 java/lang/String
0x00000006c6eb47c8 java/lang/String
Error: sun.jvm.hotspot.types.WrongTypeException: No suitable match for type of address 0x0000000800000028

However, the test does not check for this failure. All it cares is that it sees java/lang/String and doesn't see java/lang/Thread. So it ends up passing anyway.

Also, on windows we frequently see SA tests failures with the following in the log:

 Error: sun.jvm.hotspot.debugger.DebuggerException: Windbg Error: ReadVirtual failed! 

See JDK-8230731. However, the presence of this error is not why the test fails. It fails because this error resulted in some expected output to be missing. That means for any SA test failure due to missing output you first need to search the log to see if the Windbg error is there. It also likely means that there are cases of tests passing even though this Windbg error is present.

All the clhsdb tests use ClhsdbLauncher. It can be enhanced to do some additional error checking to check for various "Error:" failures in the log, so that error can be called out as the reason the test failed rather than later on the test failing due to missing output.

Also, it would be nice to get a stack trace for the exception being printed with these "Error:" messages. This is easily done by enabling "verbose" mode for clhsdb. It is in fact just for verbose exception tracing, and doesn't make the output verbose in any way unless there is an exception. See jdk.hotspot.agent/share/classes/sun/jvm/hotspot/CommandProcessor.java:

        new Command("verbose", "verbose true | false", true) {
            public void doit(Tokens t) {
                if (t.countTokens() != 1) {
                    usage();
                } else {
                    verboseExceptions = Boolean.valueOf(t.nextToken()).booleanValue();
                }
            }
        },

And the following is a common pattern in CommandProcessor.java:

                            } catch (Exception e) {
                                err.println("Error: " + e);
                                if (verboseExceptions) {
                                    e.printStackTrace(err);
                                }
                            }