JDK-8222895 : StackOverflowError in custom security manager that relies on ClassSpecializer
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.lang.invoke
  • Affected Version: 12,13
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2019-04-23
  • Updated: 2019-08-08
  • Resolved: 2019-05-02
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 13
13 b20Fixed
Related Reports
Relates :  
JDK-8155090 - String concatenation fails with a custom SecurityManager that uses concatenation
Relates :  
JDK-8181443 - Replace usages of jdk.internal.misc.Unsafe with MethodHandles.Lookup.defineClass
Description
Accidentally found that the following code generates SOE at runtime with JDK13b17, while passing with b16:

        System.setSecurityManager(new SecurityManager() {
            @Override
            public void checkPermission(java.security.Permission p) {
                String e = "a" + p + "b";
            }
        });
        new Object() { Class c = getClass(); };

The stacktrace looks like

Exception in thread "main" java.lang.StackOverflowError
	at java.base/java.util.Arrays$ArrayList.toArray(Arrays.java:4360)
	at java.base/java.lang.invoke.MethodType.listToArray(MethodType.java:233)
	at java.base/java.lang.invoke.MethodType.insertParameterTypes(MethodType.java:449)
	at java.base/java.lang.invoke.MethodHandles.dropArguments0(MethodHandles.java:3623)
	at java.base/java.lang.invoke.MethodHandles.dropArguments(MethodHandles.java:3695)
	at java.base/java.lang.invoke.StringConcatFactory$MethodHandleInlineCopyStrategy.generate(StringConcatFactory.java:1580)
	at java.base/java.lang.invoke.StringConcatFactory.generate(StringConcatFactory.java:780)
	at java.base/java.lang.invoke.StringConcatFactory.doStringConcat(StringConcatFactory.java:689)
	at java.base/java.lang.invoke.StringConcatFactory.makeConcatWithConstants(StringConcatFactory.java:605)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:99)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
...

or

Exception in thread "main" java.lang.StackOverflowError
	at java.base/jdk.internal.org.objectweb.asm.SymbolTable.hash(SymbolTable.java:1268)
	at java.base/jdk.internal.org.objectweb.asm.SymbolTable.addConstantUtf8(SymbolTable.java:795)
	at java.base/jdk.internal.org.objectweb.asm.SymbolTable.addConstantNameAndType(SymbolTable.java:771)
	at java.base/jdk.internal.org.objectweb.asm.SymbolTable.addConstantMemberReference(SymbolTable.java:603)
	at java.base/jdk.internal.org.objectweb.asm.SymbolTable.addConstantFieldref(SymbolTable.java:557)
	at java.base/jdk.internal.org.objectweb.asm.MethodWriter.visitFieldInsn(MethodWriter.java:1036)
	at java.base/java.lang.invoke.ClassSpecializer$Factory.generateConcreteSpeciesCodeFile(ClassSpecializer.java:736)
	at java.base/java.lang.invoke.ClassSpecializer$Factory.generateConcreteSpeciesCode(ClassSpecializer.java:574)
	at java.base/java.lang.invoke.ClassSpecializer$Factory.loadSpecies(ClassSpecializer.java:489)
	at java.base/java.lang.invoke.ClassSpecializer.findSpecies(ClassSpecializer.java:193)
	at java.base/java.lang.invoke.BoundMethodHandle$SpeciesData.extendWith(BoundMethodHandle.java:352)
	at java.base/java.lang.invoke.LambdaFormEditor.newSpeciesData(LambdaFormEditor.java:392)
	at java.base/java.lang.invoke.LambdaFormEditor.makeArgumentCombinationForm(LambdaFormEditor.java:760)
	at java.base/java.lang.invoke.LambdaFormEditor.filterArgumentForm(LambdaFormEditor.java:643)
	at java.base/java.lang.invoke.MethodHandles.filterArgument(MethodHandles.java:3950)
	at java.base/java.lang.invoke.MethodHandles.filterArguments(MethodHandles.java:3922)
	at java.base/java.lang.invoke.StringConcatFactory$MethodHandleInlineCopyStrategy.generate(StringConcatFactory.java:1658)
	at java.base/java.lang.invoke.StringConcatFactory.generate(StringConcatFactory.java:780)
	at java.base/java.lang.invoke.StringConcatFactory.doStringConcat(StringConcatFactory.java:689)
	at java.base/java.lang.invoke.StringConcatFactory.makeConcatWithConstants(StringConcatFactory.java:605)
	at java.base/java.lang.invoke.BootstrapMethodInvoker.invoke(BootstrapMethodInvoker.java:99)
	at java.base/java.lang.invoke.CallSite.makeSite(CallSite.java:307)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSiteImpl(MethodHandleNatives.java:259)
	at java.base/java.lang.invoke.MethodHandleNatives.linkCallSite(MethodHandleNatives.java:249)
...

or could look differently but always with a StackOverflowError
Comments
Verified that below tests are passing with latest JDK 13 build 32 (using JCK13 b12) ��� > api/java_lang/ClassLoader/Ctor.html > api/java_lang/ClassLoader/GetParent.html > api/java_lang/ClassLoader/GetSystemClassLoader.html Corresponding JTR files are attached.
08-08-2019
Root cause is JDK-8181443: any sufficiently complex String concat (or method handle use, generally) that triggers a ClassSpecializer codegen when executing SM.checkPermission will cause a SOE. Updated String concat test case fails on 12 as well.
02-05-2019
This is similar to JDK-8155090, and would have been caught by the regression test added for that issue if not for the fact that we've been a bit too good at pre-generating species classes at build time. A more contorted test case is needed now to ensure we hit the ClassSpecializer codegen paths used in some of the strategies.
24-04-2019
JDK-8222484 inadvertently changed so that a species needed for the specific concatenation shape in the reproducer now has to go via the ClassSpecializer, but this has not been problematic in the past. Root cause instead actually seems to be JDK-8181443 due changing j.l.i.ClassSpecializer to use MethodHandles.Lookup.defineClass rather than Unsafe.defineClass: the former does a call to SM.checkPermission, while before JDK-8181443 we avoided calls to SM.checkPermission in any ClassSpecializer path taken when bootstrapping any String concat expression. I can produce variants of the reproducer that crashes similarly without JDK-8222484 applied, and reverting ClassSpecializer changes of JDK-8181443 makes those tests pass, too.
24-04-2019
Likely caused by JDK-8222484 but a bit surprising since we have a regression test in this area since JDK-8155090 - must somehow have tickled a new corner case not covered by those tests.
23-04-2019