In the valhalla nestmates branch, lambda metafactory is updated and replace the use of 
`Unsafe::defineAnonymousClass` with `Lookup::defineHiddenClass`.
The lambda proxy class is a hidden nestmate of the target class and it follows the proper access control checks.

In javac, LambdaToMethod::isProtectedInSuperClassOfEnclosingClassInOtherPackage determines
when to desugar a protected method reference to a static method.  In the following example, 
p.T.Sub and p.T.Sub1 are a subclass of q.I whereas p.T is not a subclass of q.I.

In Sub::test, super::readFile will get desugared to a static method in p.T.Sub to invoke q.I::readFile.
However, this::readFile does not get desugared and such a protected method reference passed to 
p.T::m, p.T has no access to q.I::readFile.   When the lambda proxy class of p.T is defined as a hidden
nestmate class, IAE is thrown as it fails to access q.I::readFile.

In Sub1::test, such a protected method reference passed to Sub1::test, Sub1 has access to q.I::readFile but the lambda proxy class does not since it's not a subclass of q.I.   In this case, it should be desugared as well.

------------------- p/T.java
package p;
import q.I;
import java.nio.file.Path;
import java.nio.file.Paths;
import java.util.function.Function;

public class T {
    public static void main(String... args) throws Throwable {
        Path p = Paths.get("test");
        Sub sub = new Sub();
        sub.test(p);
        Sub1 sub1 = new Sub1();
        sub1.test(p);
    }

    private static void m(Function<Path,String> fileReader, Path path) {
        System.out.println(fileReader.apply(path));
    }

    public static class Sub extends I {
        public void test(Path outputDir) {
            // this method reference is desugared to a static method:
            //   REF_invokeSpecial p/T$Sub.lambda$test$0:(Ljava/nio/file/Path;)Ljava/lang/String;
            // as Sub is in package p, a different package from its its superclass
            m(super::readFile, outputDir);

            // this compiles to indy LMF with:
            //   REF_invokeVirtual q/I.readFile:(Ljava/nio/file/Path;)Ljava/lang/String;
            //
            // p.T is not a subclass of q.I and so it has no access to q.I::readFile 
            m(this::readFile, outputDir);
        }
    }

    public static class Sub1 extends I {
        public void test(Path outputDir) {
            test(this::readFile, outputDir);
        }
        private void test(Function<Path,String> fileReader, Path path) {
            System.out.println(fileReader.apply(path));
        }
    }
}

----- q/I.java 

package q;
import java.nio.file.Path;
public class I {
    protected String readFile(Path file) {
        return file.toString();
    }
}

-----------------------

Attached test.zip reproduces the error when running with the binary built from valhalla nestmates branch.
$ javac p/*.java q/*.java
$ java p.Main

Exception in thread "main" java.lang.IllegalAccessError: class p.Main$$Lambda$1$$/0x0000000800b76a70 tried to access protected method 'java.lang.String q.I.readFile(java.nio.file.Path)' (p.Main$$Lambda$1$$/0x0000000800b76a70 and q.I are in unnamed module of loader 'app')
	at q.J.checkFile(J.java:14)
	at q.J.check(J.java:18)
	at p.Main.test(Main.java:31)
	at p.Main.main(Main.java:13)

test/langtools/jdk/javadoc/doclet/testSingletonLists/TestSingletonLists.java also fails because of this issue.