JDK-8180582 : After updating to Java8u131, the bind to rmiregistry is rejected by registryFilter even though registryFilter is set
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.rmi
  • Affected Version: 8u131,9
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: x86_64
  • Submitted: 2017-05-16
  • Updated: 2017-11-29
  • Resolved: 2017-06-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10 JDK 6 JDK 7 JDK 8 JDK 9 Other
10Fixed 6u161Fixed 7u151Fixed 8u141Fixed 9 b173Fixed openjdk7uFixed
Related Reports
Relates :  
JDK-8180583 - Specifying maxdepth for registryFilter added in Java8u121 seems to have no effect
Description
FULL PRODUCT VERSION :
java version "1.8.0_131"
Java(TM) SE Runtime Environment (build 1.8.0_131-b11)
Java HotSpot(TM) 64-Bit Server VM (build 25.131-b11, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux xxx 2.6.32-641.11.1.el6.x86_64 #1 SMP Wed Oct 26 10:25:23 EDT 2016 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Introduction.

Our system uses RMI, and some RMI stubs include original classes. (* 1)
After updating to Java8u121, this class was rejected to bind to rmiregistry as follows by registryFilter that is added .

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: class <Target Class>, array length: -1, nRefs: 8, depth: 2, bytes: 507, ex: n/a

Therefore, in Java8u121, we could avoid this issue by setting registryFilter in the java.security file as follows.

sun.rmi.registry.registryFilter=<Target Class>

(*1) It uses the original InvocationHandler class which is not the java.rmi.server.RemoteObjectInvocationHandler class.

<Note>
The binding of another RMI stub that does not contain an original InvocationHandler class is not rejected.



A DESCRIPTION OF THE PROBLEM :
<Problem contents>
After updating to Java8u131, binding was rejected by registryFilter as follows when RMI stub containing classes that need to be set in registryFilter was bound to rmiregisty.

java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

Although registryFilter was set as follows in the java.security file, bind was successful in Java8u121.

sun.rmi.registry.registryFilter=<Target Class>

<Our primary view>
There is the our initial investigation result.
Based on logs, we supposed that binding was rejected because the value of depth exceeded 5(*1) on registryFilter.
In our conjecture, the following implementation change of ObjectInputStream.java changed in Java8u131 may be affecting.

<http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/diff/8a2c97926e63/src/share/classes/java/io/ObjectInputStream.java>
8172299: Improve class processing
date: Thu, 09 Feb 2017 16:34:22 +0000 (3 months ago)
method : readNonProxyDesc
-        desc.initProxy(cl, resolveEx, readClassDesc(false));
-
-        // Call filterCheck on the definition
-        filterCheck(desc.forClass(), -1);
+        try {
+            totalObjectRefs++;
+            depth++;
+            desc.initProxy(cl, resolveEx, readClassDesc(false));
+        } finally {
+            depth--;
+        }

Based on the following stack trace, there was a trace of passing through the incrementing logic of depth.

 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]

(*1)The maximum value of depth is based on below.
src/share/classes/sun/rmi/registry/RegistryImpl.java
 <http://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/75f31e0bd829/src/share/classes/sun/rmi/registry/RegistryImpl.java>
 Line 99:     private static int REGISTRY_MAX_DEPTH = 5;

<Related issue>
Specifying maxdepth for rmiregistry added in Java8u121 seems to have no effect


REGRESSION.  Last worked in version 8u121

ADDITIONAL REGRESSION INFORMATION: 
java version "1.8.0_121"
Java(TM) SE Runtime Environment (build 1.8.0_121-b13)
Java HotSpot(TM) 64-Bit Server VM (build 25.121-b13, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
In Java8u131, bind an RMI stub that requires setting registryFilter to rmiregistry.


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
If registryFilter is set, binding to rmiregistry will succeed without being REJECTED.

ACTUAL -
In the case of Java8u131, even if registryFilter is set, binding to rmiregistry is rejected by registryFilter.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
<rmiregistry log (standard output)>
java.io.ObjectInputStream filterCheck
INFO: ObjectInputFilter REJECTED: null, array length: -1, nRefs: 16, depth: 6, bytes: 692, ex: n/a

<Part of stack trace>
Caused by: java.rmi.UnmarshalException: error unmarshalling arguments; nested exception is:  
 java.io.InvalidClassException: filter status: REJECTED
 at sun.rmi.registry.RegistryImpl_Skel.dispatch(Unknown Source) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.oldDispatch(UnicastServerRef.java:450) [rt.jar:1.8.0_131]
 at sun.rmi.server.UnicastServerRef.dispatch(UnicastServerRef.java:294) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:200) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport$1.run(Transport.java:197) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.Transport.serviceCall(Transport.java:196) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport.handleMessages(TCPTransport.java:568) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:826) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:683) [rt.jar:1.8.0_131]
 at java.security.AccessController.doPrivileged(Native Method) [rt.jar:1.8.0_131]
 at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:682) [rt.jar:1.8.0_131]
 ... 3 more
Caused by: java.io.InvalidClassException: filter status: REJECTED 
 at java.io.ObjectInputStream.filterCheck(ObjectInputStream.java:1244) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readHandle(ObjectInputStream.java:1664) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1515) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.readLocation(MarshalInputStream.java:313) [rt.jar:1.8.0_131]
 at sun.rmi.server.MarshalInputStream.resolveClass(MarshalInputStream.java:189) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1826) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1843) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1713) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2000) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:2245) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:2169) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:2027) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1535) [rt.jar:1.8.0_131]
 at java.io.ObjectInputStream.readObject(ObjectInputStream.java:422) [rt.jar:1.8.0_131]
 ... 15 more


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
Unfortunately we can not submit it now.

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Do not update to Java8u131.
Set registryFilter in Java8u121.
Comments
Verified in 9 b173
09-06-2017
This is approved for JDK 9.
29-05-2017
Fix Request The RMI Registry uses the JEP 290 serial filter mechanism to limit the complexity of objects stored in the registry. The initial limit for the depth was too conservative and has caused an existing application to fail without a workaround. The filter configuration property can be used to lower the limits but not raise them so a an application specific workaround is not possible. This fix proposes to increase the allowed depth in an object graph when bound in the RMI registry from 5 to 20. The increase should be more than adequate and can be overridden to a lower value by the filter configuration property if necessary. The risk is minimal and affects only the RMI Registry. A new test is included to verify the change. The change is need to correct an issue in JDK 9 and previous releases. The webrev: is: http://cr.openjdk.java.net/~rriggs/webrev-depth-max-8180582/
25-05-2017
The blocking issue is the same as 8180583, the allowed depth is too small for the application use case. The built-in depth of 5 was based on expecting simple graphs to be bound. The estimate was too conservative and should be raised.
18-05-2017