Bug ID: JDK-8253074 SA tests time out on macOS after sudo due to runaway coresymbolicationd

JDK-8253074 : SA tests time out on macOS after sudo due to runaway coresymbolicationd

Type: Bug
Component: hotspot
Sub-Component: svc-agent
Affected Version: 11,16,17,18,19,20

Priority: P4
Status: Open
Resolution: Unresolved
OS: os_x_10.15
CPU: x86_64

Submitted: 2020-09-12
Updated: 2024-07-10

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

Other
tbdUnresolved

Related Reports

Duplicate :	JDK-8283354 - serviceability/sa/jmap-hprof/JMapHProfLargeHeapTest.java timed out
Duplicate :	JDK-8282421 - serviceability/sa/ClhsdbDumpclass.java times out
Relates :	JDK-8251852 - 3 SA tests fail with "does not appear to be polymorphic" error
Relates :	JDK-8248252 - 4 sun/tools/jhsdb tests failed with "RuntimeException: Unable to deduce type of thread"
Relates :	JDK-8282421 - serviceability/sa/ClhsdbDumpclass.java times out
Relates :	JDK-8336145 - Three SA tests Passed but then timed out in same test run on macOS
Relates :	JDK-8253811 - SA tests time out on macOS 10.13.6
Relates :	JDK-8283354 - serviceability/sa/jmap-hprof/JMapHProfLargeHeapTest.java timed out
Relates :	JDK-8265350 - SA tests time out on macOS 11.0

Description

The following 16 SA tests all timed out in the JDK16 CI:

serviceability/sa/TestType.java
serviceability/sa/TestSysProps.java
serviceability/sa/TestJmapCoreMetaspace.java
serviceability/sa/ClhsdbPrintAll.java
serviceability/sa/ClhsdbPmap.java#id0
serviceability/sa/ClhsdbLongConstant.java
serviceability/sa/ClhsdbJstackXcompStress.java
serviceability/sa/ClhsdbJstack.java#id1
serviceability/sa/ClhsdbJstack.java#id0
serviceability/sa/ClhsdbJhisto.java
serviceability/sa/ClhsdbJdis.java
serviceability/sa/ClhsdbInspect.java
serviceability/sa/ClhsdbFlags.java
serviceability/sa/ClhsdbFindPC.java#id2
serviceability/sa/ClhsdbFindPC.java#id0

I checked the logs for all 15 failures and I did not see anything unusual.

Here's a snippet from the TestType.java log file:

#section:main
----------messages:(6/199)----------
command: main TestType
reason: User specified action: run main/othervm TestType 
Mode: othervm [/othervm specified]
Timeout information:
--- Timeout information end.
elapsed time (seconds): 1185.214
----------configuration:(0/0)----------
----------System.out:(8/1499)----------
Starting TestType test
Command line: ['/System/Volumes/Data/mesos/work_dir/jib-master/install/jdk-16+16-649/macosx-x64-debug.jdk/jdk-16/fastdebug/bin/java' '-XX:MaxRAMPercentage=4' '-Djava.io.tmpdir=/System/Volumes/Data/mesos/work_dir/slaves/4076d11c-c6ed-4d07-84c1-4ab8d55cd975-S106713/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/cbe87a76-eda9-4d79-abb8-37c29fcdc77d/runs/923b730c-2f53-46b6-afc8-01674b4acc61/testoutput/test-support/jtreg_open_test_hotspot_jtreg_serviceability_sa/tmp' '-Xshare:off' '-showversion' '-cp' '/System/Volumes/Data/mesos/work_dir/slaves/4076d11c-c6ed-4d07-84c1-4ab8d55cd975-S106713/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/cbe87a76-eda9-4d79-abb8-37c29fcdc77d/runs/923b730c-2f53-46b6-afc8-01674b4acc61/testoutput/test-support/jtreg_open_test_hotspot_jtreg_serviceability_sa/classes/5/serviceability/sa/TestType.d:/System/Volumes/Data/mesos/work_dir/slaves/4076d11c-c6ed-4d07-84c1-4ab8d55cd975-S106713/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/cbe87a76-eda9-4d79-abb8-37c29fcdc77d/runs/923b730c-2f53-46b6-afc8-01674b4acc61/testoutput/test-support/jtreg_open_test_hotspot_jtreg_serviceability_sa/classes/5/test/lib' 'jdk.test.lib.apps.LingeredApp' '8536d1c5-fbe1-479c-841d-366c6e2b088c.lck']
Started LingeredApp with pid 52267
Target JDK is not signed.
Starting clhsdb against 52267
Adding 'sudo -E -n' to the command.
[2020-09-11T19:33:52.990270Z] Gathering output for process 52271
Timeout refired 480 times
----------System.err:(3/172)----------
java version "16-ea" 2021-03-16
Java(TM) SE Runtime Environment (fastdebug build 16-ea+16-649)
Java HotSpot(TM) 64-Bit Server VM (fastdebug build 16-ea+16-649, mixed mode)
----------rerun:(36/7292)*----------

Comments

[~cjplummer] - I don't know why top is showing different answers. In some of my entries that are farther back in history, I included both 'ps' and 'top' output and I grumbled about 'top' not telling the same story for the 'coresymbolicationd' line. I did that for a few entries and then I just gave up on 'top' output and now I stick with 'ps'.
29-04-2022
[~dcubed] Can you tell me why ps is showing the above data for the root 320 process, but top only shows the first line? When I just look at top output, it does show heavy CPU % usage: CPU usage: 15.15% user, 80.51% sys, 4.32% idle But all of the listed processes show 0%, including the root 320 process, for which top just shows the first line of output that you see in the ps output.
29-04-2022
Spotted in the jdk-19+6-256-tier5 CI job set: https://mach5.us.oracle.com/mdash/jobs/mach5-one-jdk-19+6-256-tier5-20220118-1514-28245835/tasks/mach5-one-jdk-19+6-256-tier5-20220118-1514-28245835-tier5-svc-svc_tools-macosx-x64-debug-606/results?search=status%3Afailed%20AND%20-state%3Ainvalid macosx-x64-debug: jpg-mac-x64-511.oraclecorp.com This test machine is running Mac_OS_X_10.15.7. 6 more failure sightings: sun/tools/jhsdb/JShellHeapDumpTest.java sun/tools/jhsdb/JStackStressTest.java sun/tools/jhsdb/HeapDumpTestWithActiveProcess.java sun/tools/jhsdb/BasicLauncherTest.java sun/tools/jhsdb/HeapDumpTest.java sun/tools/jhsdb/heapconfig/JMapHeapConfigTest.java
19-01-2022
Note that we are now starting to upgrade these machines to 10.15.7. Hopefully this will resolve this issue, and some symbolicator issues we've seen such as JDK-8251852 and JDK-8248252.
05-10-2020