JDK-8300088 : [IMPROVE] OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 21
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: os_x
  • Submitted: 2023-01-12
  • Updated: 2024-01-30
  • Resolved: 2024-01-23
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 23
23 b07Fixed
Related Reports
Cloners :  
JDK-8324577 - [REDO] - [IMPROVE] OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE
Relates :  
JDK-8324578 - [BACKOUT] [IMPROVE] OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE
Relates :  
JDK-8291060 - OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE
Relates :  
JDK-8299258 - Launching ksh (shell) from within Java through ProcessBuilder causes ksh process to crash with EXC_BAD_ACCESS (SIGSEGV)
Relates :  
JDK-8324668 - JDWP process management needs more efficient file descriptor handling
Description
This will track an improvement of JDK-8291060 "OPEN_MAX is no longer the max limit on macOS >= 10.6 for RLIMIT_NOFILE" taking into the account what we learned from JDK-8299258 "Launching ksh (shell) from within Java through ProcessBuilder causes ksh process to crash with EXC_BAD_ACCESS (SIGSEGV)"

The reason I would like to see this properly fixed, is that I would like to see parity of macOS and Linux when it comes to process/file count as it might be important to the os behavior under MACH5 and explain the issues we see on macOS compared to other platforms. This is just one tiny piece in a big puzzle and I would like to know that it's in the right spot.
Comments
There is an issue with this fix and it is being backed out.
24-01-2024
Changeset: 3d823636 Author: Gerard Ziemski <gziemski@openjdk.org> Date: 2024-01-23 16:15:59 +0000 URL: https://git.openjdk.org/jdk/commit/3d8236367d695976946d267ee62f4a26879731ac
23-01-2024
A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/17361 Date: 2024-01-10 21:49:23 +0000
10-01-2024
Proposing a fix https://github.com/openjdk/jdk/pull/17361
10-01-2024
I am starting to work on this.
04-01-2024
Email from Gunnar Wagenknecht <gunnar@wagenknecht.org>: Greetings, I'm reaching out because of an issue with the JVM on MacOS that is hitting us a large scale. It started in MacOS 13 but got really worse in 14.1. We basically now need to use -XX:-MaxFDLimit on MacOS for everything with a classpath of more then 10k jars (monolith). That applies to the Java app itself as well as any Java based IDE (Eclipse, IntelliJ) or build tool. The reason is that the MaxFDLimit implementation on Mac is broken in the JVM. The JVM is applying a lower limit to itself. We discovered the -XX:-MaxFDLimit solution after our old workarounds of increasing the open files on MacOS stopped working. We discovered it in the Bazel repository: https://github.com/bazelbuild/bazel/blob/699208763906fbd4b5e46e445b637436ee2293aa/src/main/cpp/startup_options.cc#L589-L597 // Disable the JVM's own unlimiting of file descriptors. We do this // ourselves in blaze.cc so we want our setting to propagate to the JVM. // // The reason to do this is that the JVM's unlimiting is suboptimal on // macOS. Under that platform, the JVM limits the open file descriptors // to the OPEN_MAX constant... which is much lower than the per-process // kernel allowed limit of kern.maxfilesperproc (which is what we set // ourselves to). In older versions of macOS we used a launch daemon to increase the limit: ❯ cat /Library/LaunchDaemons/limit.maxfiles.plist <?xml version="1.0" encoding="UTF-8"?> <!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd"> <plist version="1.0"> <dict> <key>Label</key> <string>limit.maxfiles</string> <key>ProgramArguments</key> <array> <string>/bin/bash</string> <string>-c</string> <string>launchctl limit maxfiles unlimited unlimited ; launchctl limit maxfiles 1000000 2147483647</string> </array> <key>RunAtLoad</key> <true/> <key>ServiceIPC</key> <false/> </dict> </plist> This stopped working in MacOS 14.1 ❯ sudo launchctl limit maxfiles 1000000 2147483647 Could not set resource limits: 150: Operation not permitted while System Integrity Protection is engaged We got an interesting info from Apple: Defaults Soft - 65535 Hard - unlimited Per Apple - The unlimited keyword is now an alias for 2147483647 (INT32_MAX) where before it was an alias for 10240 (OPEN_MAX). With SIP enabled, the soft limit can be raised, but not lowered once it’s been raised. This is by design and will generate the "Could not set resource limits". So perhaps the code in the JVM should be changed to INT32_MAX? The thing is, we cannot add the flag uniquely. For example, on older Linux installations (eg., Ubuntu 20) we noticed the -XX:-MaxFDLimit is harmful, i.e. we actually do need the JVM to increase its file handles. I found a couple old bugs but nothing actionable. Can this issue be revisited? I am not able to open a JDK bug, though. Thanks a lot! -Gunnar -- Gunnar Wagenknecht gunnar@wagenknecht.org, http://guw.io/
23-12-2023