Most applications run without a security manager. When a security manager is installed, typically it's set via the -Djava.security.manager system property at startup and should not be changed during runtime.
System::setSecurityManager allows a security manager to be set programmatically. That can incur unnecessary runtime overhead, even for applications that do not install a SecurityManager. These applications ideally should not have to incur the cost of supporting a SecurityManager if it is not used.
If the system's security manager can be guaranteed to be immutable, then various startup and runtime performance improvements can be applied. Here are a few of the costs due to a mutable security manager:
1. There are lots of "if (sm != null)" checks in the runtime. An immutable security manager can benefit from JIT constant fold that can impact performance of many security-sensitive methods (see JDK-8130289).
2. FilePermission and various Permission objects are constructed even if a security manager is disabled and no permission check is done at runtime. e.g. FilePermission initialization causes NIO to initialize.
3. AccessControlContexts are captured in various places at runtime but never used.
4. AccessController::doPrivileged may create an additional anonymous class to prepare the privileged action invocation. To avoid such startup overhead, there are many places in java.base with code like this: "if (sm != null) doPrivileged(.....) else action;"
This gives some examples.
Proposal for JDK 12:
Add new tokens ("allow" and "disallow") to the "java.security.manager" system property to allow or disallow the setting of the security manager at runtime, ex: -Djava.security.manager=disallow
Change System::setSecurityManager to throw an UnsupportedOperationException when the "java.security.manager" system property is set to "disallow".
See the CSR for exact details.
Future potential changes beyond JDK 12:
1. If this system property is set and no security manager is enabled, improve the startup performance to create an empty permission object, stop capturing ACC, and possibly have doPrivileged to invoke the action directly as a fast path.
2. Change to disallow setting security manager at runtime by default in a future release (TBD).
Tests are one typical case that sets the security manager at runtime. A test framework like jtreg supports running multiple tests in same VM and so it would have to capture the current security manager, if present, set to the one the test desires (e.g. configure the test security policy) and then restore to the captured one when the test completes. Some existing applications may set security manager programmatically while it isn't clear why it can't be installed during startup.