JDK-8234270 : [REDO] JDK-8204128 NMT might report incorrect numbers for Compiler area
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 8u231,11,14
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2019-11-15
  • Updated: 2024-03-19
  • Resolved: 2019-11-26
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 14 Other
11.0.8Fixed 13.0.4Fixed 14 b25Fixed openjdk8u282Fixed
Related Reports
Relates :  
JDK-8204128 - NMT might report incorrect numbers for Compiler area
Sub Tasks
JDK-8234272 :  
ProblemList runtime/NMT/HugeArenaTracking.java - Resolved
JDK-8234274 :  
[BACKOUT] JDK-8204128 NMT might report incorrect numbers for Compiler area - Resolved
Description
The following test failed in the JDK14 CI:

runtime/NMT/HugeArenaTracking.java

Here's a snippet from the log file:

section:main
----------messages:(5/498)----------
command: main -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:NativeMemoryTracking=detail HugeArenaTracking
reason: User specified action: run main/othervm -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:NativeMemoryTracking=detail HugeArenaTracking 
Mode: othervm [/othervm specified]
Additional options from @modules: --add-modules java.base,java.management --add-exports java.base/jdk.internal.misc=ALL-UNNAMED
elapsed time (seconds): 23.031
----------configuration:(4/115)----------
Boot Layer
  add modules: java.base java.management   
  add exports: java.base/jdk.internal.misc ALL-UNNAMED

----------System.out:(22/1277)*----------
[2019-11-15T21:56:13.932676400Z] Gathering output for process 14812
[2019-11-15T21:56:17.852986500Z] Gathering output for process 60944
# To suppress the following error report, specify this argument
# after -XX: or in .hotspotrc:  SuppressErrorAt=\\services/mallocTracker.hpp:66
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (t:\\workspace\\open\\src\\hotspot\\share\\services/mallocTracker.hpp:66), pid=21124, tid=36528
#  assert(_size >= sz) failed: deallocation > allocated
#
# JRE version: Java(TM) SE Runtime Environment (14.0+24) (fastdebug build 14-ea+24-1076)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-ea+24-1076, mixed mode, sharing, tiered, compressed oops, g1 gc, windows-amd64)
# Core dump will be written. Default location: T:\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_tier2_runtime\\scratch\\1\\hs_err_pid21124.mdmp
#
Unsupported internal testing APIs have been used.

# An error report file with more information is saved as:
# T:\\testoutput\\test-support\\jtreg_open_test_hotspot_jtreg_hotspot_tier2_runtime\\scratch\\1\\hs_err_pid21124.log
#
# If you would like to submit a bug report, please visit:
#   https://bugreport.java.com/bugreport/crash.jsp
#
----------System.err:(0/0)----------
----------rerun:(42/5200)*----------
Comments
Approving for 8u now we are at rampdown for 8u272. This can be pushed once 8u-dev is unfrozen for 8u282.
31-08-2020
This hasn't been tested much in the wild. Lets reconsider for 8u282.
21-07-2020
Fix request (13u): The original change applies cleanly, except method MemoryCounter::resize() (src/hotspot/share/services/mallocTracker.hpp), where the context needs to be updated due to absence of the fix for JDK-8234737 (Harmonize parameter order in Atomic - add) in 13u. Tested with tier1 on linux x86_64. RFR: https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-June/003266.html
09-06-2020
8u Fix Request: I would like to backport this patch to 8u, as the bug is visible to users [1]. Original patch does not apply cleanly. 1) File location mismatches 1) Arena implementation resides in allocation.cpp 2) Test library naming and locations Code review thread: http://mail.openjdk.java.net/pipermail/jdk8u-dev/2020-February/011275.html (reviewed) [1] https://mail.openjdk.java.net/pipermail/hotspot-dev/2019-November/039964.html
16-03-2020
11u Fix Request: I would like to backport this patch to 11u, as the bug is visible to users [1]. Original patch does not apply cleanly, but the conflict is minor, just a line shift in mallocTracker.hpp. Code review thread: https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2020-February/002596.html (Reviewed) [1] https://mail.openjdk.java.net/pipermail/hotspot-dev/2019-November/039964.html
26-02-2020
URL: https://hg.openjdk.java.net/jdk/jdk/rev/ac6f7738a0ee User: zgu Date: 2019-11-26 14:27:46 +0000
26-11-2019
Webrev : http://cr.openjdk.java.net/~zgu/JDK-8234270/webrev.00/ Hopefully, it can be tested by Oracle.
18-11-2019
New assertion may actually catch original problem: assert(sz >= 0 || _size >= size_t(-sz)) failed: Must be suggests that it subtracts a larger number (-sz) from smaller one (_size) and assigns the result (a negative number) to size_t, that shows up a huge number, just as in original bug report.
16-11-2019
The file is uploaded. If you want to pre-test your patch with this test, let me know. I could also test if the original bug is reproduced. (Really, it is the same test. Just check is disabled now.)
16-11-2019
[~lmesnik] Could you upload hs_err file of kitchensink test, Thanks.
16-11-2019
Another similar crash happened after this fix. It happens for other test (not using WB for NMT) and on Linux: # # A fatal error has been detected by the Java Runtime Environment: # # Internal Error (open/src/hotspot/share/services/mallocTracker.hpp:75), pid=2549, tid=2565 # assert(sz >= 0 || _size >= size_t(-sz)) failed: Must be # # JRE version: Java(TM) SE Runtime Environment (14.0) (fastdebug build 14-internal+0-2019-11-15-2209529.leonid.mesnik.ks-8234269) # Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 14-internal+0-2019-11-15-2209529.leonid.mesnik.ks-8234269, mixed mode, sharing, tiered, compressed oops, g1 gc, linux-amd64) # Problematic frame: # V [libjvm.so+0x5e0dc7] Arena::destruct_contents()+0x1c7 # # Core dump will be written. Default location: Core dumps may be processed with "/opt/core.sh %p" (or dumping to /opt/mach5/mesos/work_dir/slaves/6e54f4af-e606-43b0-80ce-0a482a5988b6-S189/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/46761bd8-500d-4b31-991a-467224a7ff35/runs/95a964a2-61db-437d-b1ab-67e6b9f0aae8/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink24H_java/scratch/0/core.2549) # # If you would like to submit a bug report, please visit: # https://bugreport.java.com/bugreport/crash.jsp # --------------- S U M M A R Y ------------ Command Line: -Xbootclasspath/a:. -XX:+UnlockDiagnosticVMOptions -XX:+WhiteBoxAPI -XX:MaxRAMPercentage=6 -XX:MaxRAMPercentage=50 -XX:MaxMetaspaceSize=256m -XX:+HeapDumpOnOutOfMemoryError -XX:+CrashOnOutOfMemoryError -Djava.net.preferIPv6Addresses=false -XX:+DisplayVMOutputToStderr -Xlog:gc*,gc+heap=debug:gc.log:uptime,timemillis,level,tags -XX:+DisableExplicitGC -XX:+StartAttachListener --add-exports=java.base/java.lang=ALL-UNNAMED --add-opens=java.base/java.lang=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.parsers=ALL-UNNAMED --add-exports=java.xml/com.sun.org.apache.xerces.internal.util=ALL-UNNAMED -Djava.io.tmpdir=/opt/mach5/mesos/work_dir/slaves/6e54f4af-e606-43b0-80ce-0a482a5988b6-S189/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/46761bd8-500d-4b31-991a-467224a7ff35/runs/95a964a2-61db-437d-b1ab-67e6b9f0aae8/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink24H_java/scratch/0/java.io.tmpdir -Duser.home=/opt/mach5/mesos/work_dir/slaves/6e54f4af-e606-43b0-80ce-0a482a5988b6-S189/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/46761bd8-500d-4b31-991a-467224a7ff35/runs/95a964a2-61db-437d-b1ab-67e6b9f0aae8/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink24H_java/scratch/0/user.home -agentpath:/opt/mach5/mesos/work_dir/jib-master/install/2019-11-15-2209529.leonid.mesnik.ks-8234269/linux-x64-debug.test/hotspot/jtreg/native/libJvmtiStressModule.so -XX:NativeMemoryTracking=detail applications.kitchensink.process.stress.Main /opt/mach5/mesos/work_dir/slaves/6e54f4af-e606-43b0-80ce-0a482a5988b6-S189/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/46761bd8-500d-4b31-991a-467224a7ff35/runs/95a964a2-61db-437d-b1ab-67e6b9f0aae8/testoutput/test-support/jtreg_closed_test_hotspot_jtreg_applications_kitchensink_Kitchensink24H_java/scratch/0/kitchensink.final.properties Host Intel(R) Xeon(R) Platinum 8167M CPU @ 2.00GHz, 8 cores, 58G, Oracle Linux Server release 7.6 Time: Fri Nov 15 23:31:07 2019 UTC elapsed time: 2976 seconds (0d 0h 49m 36s) --------------- T H R E A D --------------- Current thread (0x00007f826c477000): JavaThread "C1 CompilerThread0" daemon [_thread_in_native, id=2565, stack(0x00007f81fdbfc000,0x00007f81fdcfd000)] Current CompileTask: C1:2976950 22070 % ! 3 spec.jbb.TransactionManager::go @ 230 (694 bytes) Stack: [0x00007f81fdbfc000,0x00007f81fdcfd000], sp=0x00007f81fdcfb810, free space=1022k Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code) V [libjvm.so+0x5e0dc7] Arena::destruct_contents()+0x1c7 V [libjvm.so+0x5e0f09] Arena::~Arena()+0x19 V [libjvm.so+0x884c36] ciEnv::~ciEnv()+0xd6 V [libjvm.so+0x9a81bc] CompileBroker::invoke_compiler_on_method(CompileTask*)+0x39c V [libjvm.so+0x9a9098] CompileBroker::compiler_thread_loop()+0x468 V [libjvm.so+0x1670756] JavaThread::thread_main_inner()+0x226 V [libjvm.so+0x1675e36] Thread::call_run()+0xf6 V [libjvm.so+0x13a7ffe] thread_native_entry(Thread*)+0x10e
15-11-2019
The test has failed in two back-to-back tier2 CI job sets on Win* only. For the first completed tier2, I'm seeing passes on Linux, macOSX, and Solaris SPARC.
15-11-2019
runtime/NMT/HugeArenaTracking.java is a new test that was added by JDK-8204128. Based on the review thread, it looks like this test was only run on Linux: > Test: > hotspot_nmt + new test (fastdebug and release) > on Linux x86_64 I'm not seeing any indication that this test was run on all the platforms in jdk_submit. Since I'm only seeing a failure in Tier2 (so far), a regular jdk_submit run would not have caught this failure. It would have to be a special run where this test is added to tier1 temporarily. Or ask someone with a test farm (Oracle, SAP, etc) to run the test.
15-11-2019
Attached the hs_err_pid21124.log file.
15-11-2019
[~zgu] - This failure happened in the first Tier2 JDK14 CI job set with your fix for JDK-8204128. Can you please take a look?
15-11-2019
This failure occurred in the first Tier2 CI job set that includes the fix for JDK-8204128.
15-11-2019