FULL PRODUCT VERSION :
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux 4.4.0-97-generic #120-Ubuntu SMP Tue Sep 19 17:28:18 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
The ProtectionDomain cache (held at runtime by the PolicyFile instance) changed in the _121 release to maintain only a SoftReference to the PermissionsCollection.  This allows the cached PermissionCollections to be gc'ed at some later point in time.  there seems to be an issue when the PermissionCollection of a custom SecurityManager instance is gc'ed.  The next attempt to check any permission will result in a method call loop which ultimately results in a StackOverflowError.  

REGRESSION.  Last worked in version 8u141

ADDITIONAL REGRESSION INFORMATION: 
This works in any 1.8 release before _121.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
I don't have a minimum code example to reproduce this issue, but i can describe how our product works.

Our product uses a custom subclass of the SecurityManager.  we have a policy file loaded from the local filesystem.  our custom security manager class (ExtendedSecurityManager) is loaded in a separate jar which is granted "all permissions".  the security manager and policy file are configured on the command line.

In the reproducible case in our product, the bug does not manifest for 1 hour.    The app starts up and runs fine for the first hour.  This is presumably due some gc thresholds for SoftReferences.  Right around the hour mark, the next permissions check (and all subsequent checks) will generate StackOverflowErrors.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The permissions checks should proceed normally without generating StackOverflowErrors.
ACTUAL -
Already described above.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
This is a portion of the stack showing the method call loop.  note that the custom securitymanager call in this stack is not ultimately doing anything other than delegating to the relevant method in the SecurityManager base class.

  at java.io.File.isDirectory(File.java:844)
  at sun.net.www.ParseUtil.fileToEncodedURL(ParseUtil.java:269)
  at sun.security.provider.PolicyFile.canonicalizeCodebase(PolicyFile.java:1735)
  at sun.security.provider.PolicyFile.access$700(PolicyFile.java:258)
  at sun.security.provider.PolicyFile$5.run(PolicyFile.java:1188)
  at sun.security.provider.PolicyFile$5.run(PolicyFile.java:1186)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1185)
  at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1132)
  at sun.security.provider.PolicyFile.implies(PolicyFile.java:1086)
  at java.security.ProtectionDomain.implies(ProtectionDomain.java:285)
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
  at java.security.AccessController.checkPermission(AccessController.java:884)
  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
  at com.mycompany.security.ExtendedSecurityManager.checkPermissionImpl(ExtendedSecurityManager.java:151)
  at com.mycompany.security.ExtendedSecurityManager.checkPermission(ExtendedSecurityManager.java:97)
  at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
  at java.io.File.isDirectory(File.java:844)
  at sun.net.www.ParseUtil.fileToEncodedURL(ParseUtil.java:269)
  at sun.security.provider.PolicyFile.canonicalizeCodebase(PolicyFile.java:1735)
  at sun.security.provider.PolicyFile.access$700(PolicyFile.java:258)
  at sun.security.provider.PolicyFile$5.run(PolicyFile.java:1188)
  at sun.security.provider.PolicyFile$5.run(PolicyFile.java:1186)
  at java.security.AccessController.doPrivileged(Native Method)
  at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1185)
  at sun.security.provider.PolicyFile.getPermissions(PolicyFile.java:1132)
  at sun.security.provider.PolicyFile.implies(PolicyFile.java:1086)
  at java.security.ProtectionDomain.implies(ProtectionDomain.java:285)
  at java.security.AccessControlContext.checkPermission(AccessControlContext.java:450)
  at java.security.AccessController.checkPermission(AccessController.java:884)
  at java.lang.SecurityManager.checkPermission(SecurityManager.java:549)
  at com.mycompany.security.ExtendedSecurityManager.checkPermissionImpl(ExtendedSecurityManager.java:151)
  at com.mycompany.security.ExtendedSecurityManager.checkPermission(ExtendedSecurityManager.java:97)
  at java.lang.SecurityManager.checkRead(SecurityManager.java:888)
  at java.io.File.isDirectory(File.java:844)


REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
After much hair pulling, i finally found a workaround.  Once the custom security manager has been installed, i call an "init" method in the startup path of the actual application.  this init method uses reflective code (running in a doPrivileged block within the custom security manager implementation) to reach into the ProtectionDomain cache (hanging off of the PolicyFile instance) and grab the cached PermissionCollection instance for the ProtectionDomain of the custom security manager class (and the instance for the Policy class just to be safe).  the custom security manager maintains hard references to these two PermissionCollection instances, ensuring that they are never gc'ed.  this resolves the problem.