JDK-8267341 : macos attempt_reserve_memory_at(arg1, arg2, true) failure
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 17,18
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: x86_64
  • Submitted: 2021-05-18
  • Updated: 2022-02-16
  • Resolved: 2022-01-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 18 JDK 19
18.0.1Fixed 19 b07Fixed
Related Reports
Relates :  
JDK-8263464 - NMT: assert in gtest os.release_multi_mappings_vm
Relates :  
JDK-8267339 - Temporarily disable os.release_multi_mappings_vm on macOS x64
Relates :  
JDK-8256844 - Make NMT late-initializable
Relates :  
JDK-8262952 - [macos_aarch64] os::commit_memory failure
Sub Tasks
JDK-8267339 :  
Temporarily disable os.release_multi_mappings_vm on macOS x64 - Resolved
JDK-8271898 :  
disable os.release_multi_mappings_vm on macOS-X64 - Resolved
Description
A fix for JDK-8262952 (macos aarch64) involved using the following pattern:

     q = (address)os::attempt_reserve_memory_at((char*)q, stripe_len, executable);
     EXPECT_NE(q, (address)NULL);
     EXPECT_TRUE(os::commit_memory((char*)q, stripe_len, executable));

where previously it used to be

     q = (address)os::attempt_reserve_memory_at((char*)q, stripe_len);
     EXPECT_NE(q, (address)NULL);
     EXPECT_TRUE(os::commit_memory((char*)q, stripe_len, executable));

where the fix included the 3rd argument of attempt_reserve_memory_at(), which when omitted defaults to "false". This 3 arguments form of the API seems not to always work as expected on macos x64 according to the same test. It works fine locally for me though.

I'm pretty sure that I also ran it previously using Mach and it worked fine, so it looks like it fails sporadically.

We seemed to have uncovered a new bug on a different platform from that where the fix went in.

We should temporarily disable the test on macOS x64 (previously it was disabled on macOS aarch64) and fix the underlaying issue.
Comments
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk18u/pull/10 Date: 2022-01-27 18:11:38 +0000
27-01-2022
Fix Request (18u) Please approve this JDK-18u backport to prevent this test from failing in JDK-18u testing. The fix adds Mac OS specific code to test if the os::reserve_memory() API is available on the system under test. The fix is low risk because only code in the test is changed and the patch applied cleanly. The backport was tested on Mac OS and other platforms using Mach5.
25-01-2022
Fix was pushed while main bug was targeted to JDK18. Reset the main bug to fixed in JDK19 and copied the Robo Duke entry here.
20-01-2022
Changeset: d1efb0cc Author: Gerard Ziemski <gziemski@openjdk.org> Date: 2022-01-19 22:34:54 +0000 URL: https://git.openjdk.java.net/jdk/commit/d1efb0cc56127a40f2b23ccc608b27be2b39a389
20-01-2022
It's not high priority, only P3, and since it went to JDK19, instead of JDK18, then I guess I am good with that. Just have to fix the main/backport issue now...
20-01-2022
[~gziemski] - Your PR was baselined on jdk/jdk so when you integrated this fix it was pushed to jdk/jdk (JDK19) instead of JDK18. Do you plan to backport this fix to JDK18?
19-01-2022
A pull request was submitted for review. URL: https://git.openjdk.java.net/jdk/pull/6960 Date: 2022-01-04 19:18:45 +0000
04-01-2022
The fix for: JDK-8256844 Make NMT late-initializable re-enabled the os.release_multi_mappings_vm subtest in gtest/GTestWrapper.java on macOS-X64 as part of that patch. The os.release_multi_mappings_vm subtest was previously disabled for macOS-X64 by: JDK-8267339 Temporarily disable os.release_multi_mappings_vm on macOS x64 and I've filed the following to disable it again for macOS-X64: 8271898 disable os.release_multi_mappings_vm on macOS-X64
04-08-2021
[~stuefe] - We've just had our third and fourth Tier1 failures. All four failures today are on test machines that are running Mac_OS_X_10.13.6..
04-08-2021
Hi Dan, JDK-8256844 "Make NMT late-initializable" - very unlikely to have anything to do with it. That one was about tracking malloced memory in NMT. JDK-8263464 "NMT: assert in gtest os.release_multi_mappings_vm" - ditto. That one just uncovered a zero-day bug in NMT where it turned out it cannot deal with striped mappings. JDK-8256844 just deactivates the test if NMT is activated until the day JDK-8263464 is fixed. The test does not particularly care for executable permissions; it just alternates the exec flag on the stripes in the mapping to prevent the kernel from folding the mappings. The same could also be done with protection bits (e.g. using os::protect_memory()). It is also not terribly important for MacOS. We use striped allocations with large pages on Linux, and with NUMA on Windows. On MacOS I do not know any striped allocation. Arguably one could deactivate the test for mac os. I would only be just curious what causes the allocation failures. Could the amount of executable memory be limited on MacOS? The stripes are quite large (4M per stripe). ..Thomas
04-08-2021
There have been two Tier1 sightings of the gtest/GTestWrapper.java failure with the os.release_multi_mappings_vm subtest in a row. Both of these Tier1 failures have happened on test machines running Mac_OS_X_10.13.6. This increase in failure frequency might be due to: JDK-8256844 Make NMT late-initializable which is the only changeset integrated in jdk-18+9-408-tier1. There is also this related bug: JDK-8263464 NMT: assert in gtest os.release_multi_mappings_vm If this Tier1 failure reproduces again, we'll have to disable the os.release_multi_mappings_vm subtest in order to reduce the noise in the CI. [~gziemski] and [~stuefe], please investigate this change in behavior for this test. Update: jdk-18+9-410-tier1 is GREEN so it's possible that the failures in jdk-18+9-408-tier1 and jdk-18+9-409-tier1 were just bad luck...
04-08-2021
Here's a snippet from the jdk-18+9-408-tier1 sighting: gtest/GTestWrapper.java [ RUN ] os.release_multi_mappings_vm open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL open/test/hotspot/gtest/runtime/test_os.cpp:377: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL open/test/hotspot/gtest/runtime/test_os.cpp:377: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true A B C [ FAILED ] os.release_multi_mappings_vm (0 ms)
04-08-2021
This is very hard to replicate, and the test seems somewhat optional on macOS, so let's not risk jdk17 build by the fix, as JDK-8262952 broke jdk17 build, and work on it later without the pressure.
27-05-2021
Failing output: [----------] 24 tests from os [ RUN ] os.page_size_for_region_vm [ OK ] os.page_size_for_region_vm (0 ms) [ RUN ] os.page_size_for_region_aligned_vm [ OK ] os.page_size_for_region_aligned_vm (0 ms) [ RUN ] os.page_size_for_region_alignment_vm [ OK ] os.page_size_for_region_alignment_vm (0 ms) [ RUN ] os.page_size_for_region_unaligned_vm [ OK ] os.page_size_for_region_unaligned_vm (0 ms) [ RUN ] os.test_random [ OK ] os.test_random (0 ms) [ RUN ] os.page_size_for_region_with_zero_min_pages_vm_assert [ OK ] os.page_size_for_region_with_zero_min_pages_vm_assert (288 ms) [ RUN ] os.test_print_hex_dump_vm [ OK ] os.test_print_hex_dump_vm (8 ms) [ RUN ] os.vsnprintf_vm [ OK ] os.vsnprintf_vm (0 ms) [ RUN ] os.snprintf_vm [ OK ] os.snprintf_vm (0 ms) [ RUN ] os.jio_vsnprintf_vm [ OK ] os.jio_vsnprintf_vm (0 ms) [ RUN ] os.jio_snprintf_vm [ OK ] os.jio_snprintf_vm (0 ms) [ RUN ] os.release_multi_mappings_vm /opt/mesos/slaves/40f4774a-f408-4e28-8432-1004a7782450-S3307/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/86d3796c-25f3-4e59-9897-a92cdb350ffe/runs/78ff8324-9446-4b98-9413-1611a93f71dd/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:375: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL /opt/mesos/slaves/40f4774a-f408-4e28-8432-1004a7782450-S3307/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/86d3796c-25f3-4e59-9897-a92cdb350ffe/runs/78ff8324-9446-4b98-9413-1611a93f71dd/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true /opt/mesos/slaves/40f4774a-f408-4e28-8432-1004a7782450-S3307/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/86d3796c-25f3-4e59-9897-a92cdb350ffe/runs/78ff8324-9446-4b98-9413-1611a93f71dd/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:375: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL /opt/mesos/slaves/40f4774a-f408-4e28-8432-1004a7782450-S3307/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/86d3796c-25f3-4e59-9897-a92cdb350ffe/runs/78ff8324-9446-4b98-9413-1611a93f71dd/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true A B C [ FAILED ] os.release_multi_mappings_vm (0 ms) [ RUN ] os.release_one_mapping_multi_commits_vm A B C D [ OK ] os.release_one_mapping_multi_commits_vm (0 ms)
26-05-2021
ILW = HLM = P3
25-05-2021
I had bumped this to P2 since it showed up in Tier1, but then I realized that the failure is intermittent so I've lowered it to a P3.
19-05-2021
The build where the change went in and which triggered this issue is jdk-17+23-2040 The next build after that also shows the issue is jdk-17+23-2041 However, the builds after that, i.e. jdk-17+23-2042, jdk-17+23-2043, jdk-17+23-2044, are fine, so the issue seems intermittent (?)
18-05-2021
The test passes locally on my laptop ran either in a suite (like in Mach5) or individually: ------------------------------- The test run on its own: # make run-test TEST=gtest:os.release_multi_mappings_vm Building target 'run-test' in configuration 'macosx-x86_64-server-slowdebug' Test selection 'gtest:os.release_multi_mappings_vm', will run: * gtest:os.release_multi_mappings_vm/server Running test 'gtest:os.release_multi_mappings_vm/server' Note: Google Test filter = os.release_multi_mappings_vm* [==========] Running 1 test from 1 test suite. [----------] Global test environment set-up. [----------] 1 test from os [ RUN ] os.release_multi_mappings_vm A B C [ OK ] os.release_multi_mappings_vm (0 ms) [----------] 1 test from os (661 ms total) [----------] Global test environment tear-down [==========] 1 test from 1 test suite ran. (661 ms total) [ PASSED ] 1 test. ------------------------------------------ The test run as a part of a suite: # make test TEST=test/hotspot/jtreg/:tier1_common Building target 'test' in configuration 'macosx-x86_64-server-slowdebug' Updating hotspot/variant-server/libjvm/gtest/libjvm.dylib due to 1 file(s) Test selection 'test/hotspot/jtreg/:tier1_common', will run: * jtreg:test/hotspot/jtreg:tier1_common Running test 'jtreg:test/hotspot/jtreg:tier1_common' Passed: gtest/MetaspaceGtests.java#balanced-no-ccs Passed: gtest/MetaspaceGtests.java#reclaim-none-debug Passed: gtest/MetaspaceGtests.java#reclaim-aggressive-ndebug Passed: gtest/MetaspaceGtests.java#balanced-with-guards Passed: sanity/BasicVMTest.java Passed: gtest/MetaspaceGtests.java#reclaim-aggressive-debug Passed: gtest/MetaspaceGtests.java#reclaim-none-ndebug Passed: gtest/GTestWrapper.java Test results: passed: 8
18-05-2021
[ RUN ] os.release_multi_mappings_vm /System/Volumes/Data/mesos/work_dir/slaves/40f4774a-f408-4e28-8432-1004a7782450-S72018/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/d121418d-c63f-4d5c-a319-f6ddf0e9a648/runs/7a05ed6b-bbb9-4410-9b58-2d56c12a4884/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:375: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL /System/Volumes/Data/mesos/work_dir/slaves/40f4774a-f408-4e28-8432-1004a7782450-S72018/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/d121418d-c63f-4d5c-a319-f6ddf0e9a648/runs/7a05ed6b-bbb9-4410-9b58-2d56c12a4884/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true /System/Volumes/Data/mesos/work_dir/slaves/40f4774a-f408-4e28-8432-1004a7782450-S72018/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/d121418d-c63f-4d5c-a319-f6ddf0e9a648/runs/7a05ed6b-bbb9-4410-9b58-2d56c12a4884/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:375: Failure Expected: (q) != ((address)__null), actual: NULL vs NULL /System/Volumes/Data/mesos/work_dir/slaves/40f4774a-f408-4e28-8432-1004a7782450-S72018/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/d121418d-c63f-4d5c-a319-f6ddf0e9a648/runs/7a05ed6b-bbb9-4410-9b58-2d56c12a4884/workspace/open/test/hotspot/gtest/runtime/test_os.cpp:376: Failure Value of: os::commit_memory((char*)q, stripe_len, executable) Actual: false Expected: true A B C [ FAILED ] os.release_multi_mappings_vm (0 ms)
18-05-2021