JDK-8180583 : Specifying maxdepth for registryFilter added in Java8u121 seems to have no effect
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.rmi
  • Affected Version: 8u131,9
  • Priority: P3
  • Status: Resolved
  • Resolution: Not an Issue
  • OS: generic
  • CPU: x86_64
  • Submitted: 2017-05-16
  • Updated: 2017-06-06
  • Resolved: 2017-05-25
Related Reports
Relates :  
JDK-8180582 - After updating to Java8u131, the bind to rmiregistry is rejected by registryFilter even though registryFilter is set
Description
FULL PRODUCT VERSION :
java -version'���������
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux xxx 2.6.32-641.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Introduction.

Our system uses RMI, and some RMI stubs include original classes. (* 1)
After updating to Java8u121, this class was rejected to bind to rmiregistry as follows by registryFilter that is added .

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: class <Target Class>, array length: -1, nRefs: 8, depth: 2, bytes: 507, ex: n/a

Therefore, in Java8u121, we could avoid this issue by setting registryFilter in the java.security file as follows.

sun.rmi.registry.registryFilter=<Target Class>

(*1) It uses the original InvocationHandler class which is not the java.rmi.server.RemoteObjectInvocationHandler class.

<Note>
The binding of another RMI stub that does not contain an original InvocationHandler class is not rejected.


A DESCRIPTION OF THE PROBLEM :
<Problem contents>
After updating to Java8u131, binding was rejected by registryFilter as follows when RMI stub containing classes that need to be set in registryFilter was bound to rmiregisty.

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

Based on logs, we supposed that it was rejected because depth exceeded the maximum value of 5.
Therefore, based on JEP 290 (*1), the maxdepth value was added as 7 to the registryFilter.
Below setting image to java.security file.

sun.rmi.registry.registryFilter=<Target Class>;\
maxdepth=7

However, despite specifying maxdepth as 7, it was REJECTed at depth: 6, and it appeared to be ineffective.
There is the our initial investigation result.

<Our primary view>
There is the our initial investigation result.
Based on the source code, in the RegistryImpl class (*2), maxdepth of registryFilter is defined as 5 (*3) as a fixed value, so specifying maxdepth of registryFilter seems to have no effect in the java.security file.

(*1) : <http://openjdk.java.net/jeps/290>
(*2) : src/share/classes/sun/rmi/registry/RegistryImpl.java
 <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/75f31e0bd829/src/share/classes/sun/rmi/registry/RegistryImpl.java>
(*3) : There are two point on RegistryImpl.java
 Line 99:     private static int REGISTRY_MAX_DEPTH = 5;
 Line 397:         if (filterInfo.depth() > REGISTRY_MAX_DEPTH) {

<Related issue>
After updating to Java8u131, the bind to rmiregistry is rejected by registryFilter even though registryFilter is set


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In Java8u131, bind an RMI stub that requires setting registryFilter to rmiregistry.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
If maxdepth is specified for registryFilter, maxdepth is checked with the specified value instead of 5.

ACTUAL -
Although maxdepth is specified for registryFilter, it seems that maxdepth is checked with a fixed value as 5.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
<rmiregistry log (standard output)>
java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

<Part of stack trace>
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:  
 java.io.InvalidClassException: filter status: REJECTED
 at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:450) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:200) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:197) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [rt.jar:1.8.0_131]
 ... 3 more
Caused by: java.io.InvalidClassException: filter status: REJECTED 
 at java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1244) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1664) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.readLocation(MarshalInputStream.java:313) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:189) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 ... 15 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Unfortunately we can not submit it now.
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do not update to Java8u131.
Set registryFilter in Java8u121.
Comments
As noted, the configurable filter can reduce limits not increase them. The related issue 8180582 should increase the limits of the built-in registry filter.
25-05-2017
In the configurable RMI filter (sun.rmi.registry.registryFilter), limits can only be reduced to REJECT additional cases. If the configurable filter does not reject, then the built-in filter is applied. Both the javadoc for ObjectInputFilter.createFilter and JEP 290 make this point as: "If any of the limits are exceeded, the filter returns Status.REJECTED." The built-in depth of 5 was based on expecting simple graphs to be bound. The depth can be raised significantly.
18-05-2017