JDK-8301863 : Mistake in documentation of JEP415 and Javadocs, and possibly mistake in the design that allows cascaded rejectUndecidedClass() filters
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.io
  • Affected Version: 17,19,20
  • Priority: P4
  • Status: New
  • Resolution: Unresolved
  • OS: generic
  • CPU: generic
  • Submitted: 2023-02-05
  • Updated: 2023-02-06
Description
A DESCRIPTION OF THE PROBLEM :
I was trying to get my head around the FilterInThread example in JEP 415 (https://openjdk.org/jeps/415) and the JavaDoc for the ObjectInputFilter (https://docs.oracle.com/en/java/javase/17/docs/api/java.base/java/io/ObjectInputFilter.html)

For example, let's assume we have three filters. The first allow ArrayList, the second allows Integer, the third restricts arrays to not be larger than 1000.

    ObjectInputFilter allowArrayList = ObjectInputFilter.allowFilter(
            Set.of(ArrayList.class, Object.class)::contains, UNDECIDED
    );
    ObjectInputFilter allowInteger = ObjectInputFilter.allowFilter(
            Set.of(Number.class, Integer.class)::contains, UNDECIDED
    );
    ObjectInputFilter restrictLargeArrays =
ObjectInputFilter.Config.createFilter("maxarray=1000");

Let's say that we create a FilterInThread instance and install that as our factory. Furthermore, we set the allowArrayList as the global serial filter. When we call filterInThread.doWithSerialFilter() we pass in the allowInteger filter. Lastly, during the actual deserialization, we call setObjectInputFilter() on the ObjectInputStream with the restrictLargeArrays filter. Ideally, I would want the final filter to look like this:

rejectUndecidedClass(merge(restrictLargeArrays,merge(allowInteger,allowArrayList)))

However, in the FilterInThread example, we add the rejectUndecidedClass() wrapper around each of the steps. Thus we would get something like:

rejectUndecidedClass(merge(restrictLargeArrays,rejectUndecidedClass(merge(allowInteger,rejectUndecidedClass(allowArrayList)))))

Thus we could never allow any classes except for ArrayList.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Mistake in documentation of JEP415 and Javadocs, and possibly mistake in the design that allows cascaded rejectUndecidedClass() filters

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
It should wire the composite filters so that ArrayList is allowed.
ACTUAL -
It rejects ArrayList

---------- BEGIN SOURCE ----------

import java.io.ObjectInputFilter;
import java.util.function.BinaryOperator;

// example from JEP415
public class FilterInThread implements BinaryOperator<ObjectInputFilter> {
    private final ThreadLocal<ObjectInputFilter> filterThreadLocal =
            new ThreadLocal<>();

    // Construct a FilterInThread deserialization filter factory.
    public FilterInThread() {}

    // Returns a composite filter of the static JVM-wide filter, a thread-specific filter,
    // and the stream-specific filter.
    public ObjectInputFilter apply(ObjectInputFilter curr, ObjectInputFilter next) {
        if (curr == null) {
            // Called from the OIS constructor or perhaps OIS.setObjectInputFilter with no current filter
            var filter = filterThreadLocal.get();
            if (filter != null) {
                // Wrap the filter to reject UNDECIDED results
                filter = ObjectInputFilter.rejectUndecidedClass(filter);
            }
            if (next != null) {
                // Merge the next filter with the thread filter, if any
                // Initially this is the static JVM-wide filter passed from the OIS constructor
                // Wrap the filter to reject UNDECIDED results
                filter = ObjectInputFilter.merge(next, filter);
                filter = ObjectInputFilter.rejectUndecidedClass(filter);
            }
            return filter;
        } else {
            // Called from OIS.setObjectInputFilter with a current filter and a stream-specific filter.
            // The curr filter already incorporates the thread filter and static JVM-wide filter
            // and rejection of undecided classes
            // If there is a stream-specific filter wrap it and a filter to recheck for undecided
            if (next != null) {
                next = ObjectInputFilter.merge(next, curr);
                next = ObjectInputFilter.rejectUndecidedClass(next);
                return next;
            }
            return curr;
        }
    }

    // Applies the filter to the thread and invokes the runnable.
    public void doWithSerialFilter(ObjectInputFilter filter, Runnable runnable) {
        var prevFilter = filterThreadLocal.get();
        try {
            filterThreadLocal.set(filter);
            runnable.run();
        } finally {
            filterThreadLocal.set(prevFilter);
        }
    }
}


import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;
import java.io.InvalidClassException;
import java.io.ObjectInputFilter;
import java.io.ObjectInputStream;
import java.io.ObjectOutputStream;
import java.io.UncheckedIOException;
import java.util.ArrayList;
import java.util.Arrays;
import java.util.LinkedList;
import java.util.List;
import java.util.Set;
import java.util.stream.Collectors;
import java.util.stream.IntStream;

import org.opentest4j.AssertionFailedError;

import static org.junit.jupiter.api.Assertions.*;

public class FilterInThreadDemo {
    protected static final ThreadLocal<ObjectInputFilter> ourFilter = new ThreadLocal<>();
    protected static final FilterInThread filterInThread = new FilterInThread();
    private static final String STARS = "*".repeat(20);

    protected static void test(ObjectInputFilter additionalFilter,
                               boolean shouldPass) {
        try {
            filterInThread.doWithSerialFilter(
                    additionalFilter, () -> {
                        try {
                            testAll();
                        } catch (IOException e) {
                            throw new UncheckedIOException(e);
                        }
                    }
            );
            System.out.println(STARS + " SUCCESS " + STARS);
            assertTrue(shouldPass);
        } catch (AssertionFailedError e) {
            System.out.println(STARS + " FAILURE " + STARS);
            System.out.println(e);
            assertFalse(shouldPass);
        }
        System.out.println();
    }

    public static void testAll() throws IOException {
        testGood(); // good
        testGood("hello world"); // good
        testGood("hello", "world"); // good
        testGood(123); // good
        testGood(new ArrayList<>(List.of("Hello", "World"))); // good
        testBad(new LinkedList<>(List.of("Hello", "World"))); // bad
        testBad(List.of("Hello", "World")); // bad
        testBad(Set.of("Hello", "World")); // bad
        testGood(IntStream.range(0, 1000)
                .boxed()
                .collect(Collectors.toList())); // good
        testBad(IntStream.range(0, 1001)
                .boxed()
                .collect(Collectors.toList())); // bad
    }

    private static void testGood(Object... values) {
        assertDoesNotThrow(() -> readAndPrint(makeStream(values)));
        System.out.println();
    }

    private static void testBad(Object... values) {
        assertThrows(InvalidClassException.class, () -> readAndPrint(makeStream(values)));
        System.out.println();
    }

    private static InputStream makeStream(Object... values) throws IOException {
        System.out.println("Making stream from " + Arrays.toString(values)
                + " types " +
                Arrays.stream(values).map(Object::getClass)
                        .map(Class::getTypeName)
                        .collect(Collectors.joining(", ", "[", "]")));
        ByteArrayOutputStream bout = new ByteArrayOutputStream();
        try (ObjectOutputStream out = new ObjectOutputStream(bout)) {
            for (Object value : values) {
                out.writeObject(value);
            }
            out.writeObject(null);
        }
        return new ByteArrayInputStream(bout.toByteArray());
    }

    public static void readAndPrint(InputStream is)
            throws IOException, ClassNotFoundException {
        System.out.println("Testing ...");
        try (ObjectInputStream in = new ObjectInputStream(is)) {
            ObjectInputFilter filter = ourFilter.get();
            if (filter != null) in.setObjectInputFilter(filter);
            Object obj;
            while ((obj = in.readObject()) != null) {
                System.out.println("Read: " + obj);
            }
        }
    }
}



import static org.junit.jupiter.api.Assertions.assertDoesNotThrow;
import static org.junit.jupiter.api.Assertions.assertThrows;

public class FilterInThreadDemo1WithNoFactory extends FilterInThreadDemo {
    public static void main(String... args) {
        // expect to fail
        test(null, false);

        ourFilter.set(Filters.compositeFilter);
        // expect to pass
        test(null, true);
        ourFilter.remove();
    }
}



import java.io.ObjectInputFilter;

public class FilterInThreadDemo2WithFactory extends FilterInThreadDemo {
    public static void main(String... args) {
        ObjectInputFilter.Config.setSerialFilterFactory(filterInThread);

        // expect to fail
        test(null, false);

        ourFilter.set(Filters.compositeFilter);
        // expect to pass
        test(null, true);
        ourFilter.remove();
    }
}



import java.io.ObjectInputFilter;

public class FilterInThreadDemo3WithFactoryAndGlobalFilter extends FilterInThreadDemo {
    public static void main(String... args) {
        ObjectInputFilter.Config.setSerialFilterFactory(filterInThread);
        ObjectInputFilter.Config.setSerialFilter(Filters.allowArrayList);

        // expect to fail
        test(null, false);

        ourFilter.set(Filters.restrictLargeArrays);
        // expect to pass, but fails
        test(Filters.allowInteger, true);
        ourFilter.remove();

        ourFilter.set(Filters.compositeFilter);
        // expect to pass, but fails
        test(null, true);
        ourFilter.remove();

        ourFilter.set(Filters.limitedCompositeFilter);
        // expect to pass, but fails
        test(null, true);
        ourFilter.remove();

        // expect to pass, but fails
        test(Filters.limitedCompositeFilter, true);

        // expect to pass
        test(Filters.compositeFilter, true);
    }
}


import java.io.ObjectInputFilter;
import java.util.ArrayList;
import java.util.LinkedList;
import java.util.Set;

import static java.io.ObjectInputFilter.Status.UNDECIDED;
import static java.io.ObjectInputFilter.allowFilter;
import static java.io.ObjectInputFilter.rejectFilter;
import static java.util.Set.of;

public class Filters {
    public static final ObjectInputFilter allowArrayList =
            allowFilter(of(ArrayList.class, Object.class)::contains, UNDECIDED);
    public static final ObjectInputFilter rejectLinkedList =
            rejectFilter(LinkedList.class::equals, UNDECIDED);
    public static final ObjectInputFilter allowInteger = allowFilter(
            of(Number.class, Integer.class)::contains, UNDECIDED
    );
    public static final ObjectInputFilter restrictLargeArrays =
            ObjectInputFilter.Config.createFilter("maxarray=1000");
    public static final ObjectInputFilter limitedCompositeFilter =
            ObjectInputFilter.merge(allowInteger, restrictLargeArrays);
    public static final ObjectInputFilter compositeFilter =
            ObjectInputFilter.rejectUndecidedClass(
                    ObjectInputFilter.merge(allowArrayList, limitedCompositeFilter));
}


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Not sure how to work around this, except that perhaps merge() could remove downstream "rejectUndecidedClass" filters. However, that behaviour would need to be carefully documented.

FREQUENCY : always



Comments
The observations on Windows 10: JDK 17: Failed, test failed observed. JDK 19: Failed. JDK 20ea+23: Failed.
06-02-2023

The reproducer contains an unknown class, Filters. Requested more details from the submitter.
06-02-2023