JDK-8168075 : Custom system class loader + security manager + malformed policy file = recursive initialization
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 8,9
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2016-10-17
  • Updated: 2018-01-30
  • Resolved: 2017-01-24
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10 JDK 9
10Fixed 9 b155Fixed
Related Reports
Relates :  
JDK-8150468 - ClassCircularityError on error in security policy file
Relates :  
JDK-8196215 - sun/security/util/Resources/customSysClassLoader/MessageFormatting.java failed on ar_SA locale.
Relates :  
JDK-8168423 - Test Task: Custom system class loader + security manager + malformed policy file = recursive initialization
Description
If a malformed policy file is encountered when running with a custom system class loader then it leads to a recursive initialization issue.

To demonstrate then see the attached test case and run it with:

java -Djava.system.class.loader=MyClassLoader -Djava.security.manager -Djava.security.policy=java.policy Main

The stack trace with jdk-9+140 is below. It's possible to change SL to not use getSystemClassLoader but that just moves the problem to ResourceBundle. The stack trace with JDK 8 is different but amounts to the same thing.

Error occurred during initialization of VM
java.lang.Error: java.lang.reflect.InvocationTargetException
	at java.lang.ClassLoader.initSystemClassLoader(java.base@9-internal/ClassLoader.java:1748)
	at java.lang.System.initPhase3(java.base@9-internal/System.java:1979)
Caused by: java.lang.reflect.InvocationTargetException
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance0(java.base@9-internal/Native Method)
	at jdk.internal.reflect.NativeConstructorAccessorImpl.newInstance(java.base@9-internal/NativeConstructorAccessorImpl.java:62)
	at jdk.internal.reflect.DelegatingConstructorAccessorImpl.newInstance(java.base@9-internal/DelegatingConstructorAccessorImpl.java:45)
	at java.lang.reflect.Constructor.newInstance(java.base@9-internal/Constructor.java:455)
	at java.lang.ClassLoader.initSystemClassLoader(java.base@9-internal/ClassLoader.java:1746)
	at java.lang.System.initPhase3(java.base@9-internal/System.java:1979)
Caused by: java.lang.InternalError: getSystemClassLoader should only be called after VM booted
	at java.lang.ClassLoader.getSystemClassLoader(java.base@9-internal/ClassLoader.java:1698)
	at java.util.ServiceLoader.loadInstalled(java.base@9-internal/ServiceLoader.java:1053)
	at java.util.ResourceBundle.<clinit>(java.base@9-internal/ResourceBundle.java:462)
	at sun.security.util.ResourcesMgr$1.run(java.base@9-internal/ResourcesMgr.java:47)
	at sun.security.util.ResourcesMgr$1.run(java.base@9-internal/ResourcesMgr.java:44)
	at java.security.AccessController.doPrivileged(java.base@9-internal/Native Method)
	at sun.security.util.ResourcesMgr.getString(java.base@9-internal/ResourcesMgr.java:43)
	at sun.security.provider.PolicyParser.match(java.base@9-internal/PolicyParser.java:746)
Comments
At one point when I looked at this recursive initialization, I suggest not to localize the exception message for malformed policy file that avoids initializing the resource bundle. Or like Alan suggests, the error messages can be localized after VM is booted.
19-10-2016
One suggestion is for the PolicyFile code to not use the resource bundle when VM.isBooted() is false, this will avoid all recursive initialization issues when the policy file is parsed in early startup (the custom system class loader case).
18-10-2016