JDK-8169000 : Define reference reachability more precisely in java.lang.ref package
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.lang
  • Affected Version: 6,7,8,9,10
  • Priority: P5
  • Status: Resolved
  • Resolution: Not an Issue
  • Submitted: 2016-11-01
  • Updated: 2017-11-29
  • Resolved: 2017-11-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbd_majorResolved
Related Reports
Duplicate :  
Description
Phantom referenced objects may not appear in the ReferenceQueue if VM run with -Xcomp.

The minimal test demonstrating the bug looks like:
--- 8< ------- Phantom.java --------------------
import java.lang.ref.PhantomReference;
import java.lang.ref.Reference;
import java.lang.ref.ReferenceQueue;

public class Phantom {
    static ReferenceQueue rq = new ReferenceQueue();
    public static void main(final String[] args) throws Exception {
        PhantomReference ref = new PhantomReference(new Object(), rq);
        System.out.println("ref: " + ref);
        System.gc();

        Reference rmRef = rq.remove(1000);
        if (rmRef == null) {
            for (int i = 0; i < 10; i++) {
                System.gc();
            }          
            rmRef = rq.remove(1000);
            if (rmRef == null) {
                throw new Error("Test failed");
            }
        }
        System.out.println("Test passed");
    }
}
--- 8< ---------------------------------------------------------

# java Phantom
ref: java.lang.ref.PhantomReference@614c5515
Test passed
# java -Xcomp Phantom
ref: java.lang.ref.PhantomReference@614c5515
Exception in thread "main" java.lang.Error: Test failed
        at Phantom.main(Phantom.java:19)

Please note: Declaring 'PhantomReference ref' as a static member of the Phantom class, not as a local variable, will make the test pass.
Comments
Reopen JDK-8168682 instead of this issue (which is about a spec issue).
29-11-2017

reopening, the test still fails in hotspot testing. if it's not a product bug then it's a test bug which should be fixed.
28-11-2017

[~mchung] the ClassForNameLeak test is still failing in HotSpot PIT testing: ----------System.err:(13/842)---------- java.lang.RuntimeException: ClassLoader was never enqueued! at ClassForNameLeak.main(ClassForNameLeak.java:114) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:564) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:115) at java.base/java.lang.Thread.run(Thread.java:844) JavaTest Message: Test threw exception: java.lang.RuntimeException: ClassLoader was never enqueued! JavaTest Message: shutting down test ClassForNameLeak is referenced by JDK-8168682, which is closed as a duplicate of this bug. If this bug is not an issue, should ClassForNameLeak be modified or removed?
14-11-2017

JLS 12.6.2 specifies the condition when an object is definitely reachable. When a reference object is marked as unreachable at a reachability point, the reference object will not be enqueued. java.lang.ref package summary is correct. I think the reachability section is clear. It is unspecified when an object becomes unreachable.
18-09-2017

I'm inclined to close this as not a bug. Reachability is specified clearly in java.lang.ref package summary when an object is unreachable. When a registered Reference object is unreachable, it will not get enqueued. The specification does not specify neither how the garbage collector detects when an object becomes unreachable nor the runtime state maps exactly to the source level. The concerning lines of source are: PhantomReference ref = new PhantomReference(new Object(), rq); // --> The specification does not specify that object pointed by "ref" is unreachable even if it's not referenced in this method body System.gc(); Setting ref = null will assist the runtime to ensure that the object is not referenced from that point. Then both -Xint and -Xcomp will have the same result and the reference is not enqueued. If the test wants to ensure the reference object is reachable, it should make sure `ref` is "definitely reachable" as specified in JLS 12.6.2.
14-09-2017

Updated ILW: - not breaking specification (instead specification can be made more precise); - not a defect, but issue might lead to unreasonable expectations as to how references work; - no workaround = LLH=P5
11-11-2016

Thank you, Maurizio, Alex, and David for the feedback. @Maurizio: Yes, you are right, I did not suggest that javac treats PhantomReference variables in any special way. That would be -- as you pointed out -- unfeasible (also for the JIT, for similar reasons). I was more curious of how javac could better detect and indicate that a variable has become unreachable (i.e., by "flushing" it). I did not know that javac keeps a local variable alive for the entire scope containing the variable's declaration. Changing that seems like a non-trivial task. @Alex: The problem with the PhantomSimplified program is that interpreted and C1-compiled code behave differently. With the interpreter, the phantom reference is kept alive in local variable 1 (which is not obvious from the source code). I did not say that the JIT can't detect that a phantom reference is dead: The C1 compiler detects that the phantom reference is dead before the System.gc() call and thus "discards" it. I was wondering how it would be possible to make interpreted and compiled code behave the same way. Making the JIT keep phantom references alive longer is difficult to do (that's what I meant by "special compiler treatment for phantom references"). Changing the interpreter to discard dead variables early (or flush them) is unfeasible as well. That is why I asked what javac can do. @David: I agree with your suggestion to update the documentation. That way expectations are more clearly set.
11-11-2016

'ref' is not phantom-reachable. As long as ref is reachable then the newly constructed object is phantom-reachable. Note the final "reachability" definition: "Finally, an object is unreachable, and therefore eligible for reclamation, when it is not reachable in any of the above ways." This does not mandate reclamation, it only defines eligibility for reclamation. So again, detecting when something is unreachable, is not definitively specified - nor should it be.
10-11-2016

Exactly - I was thinking more about this - and read the 'reachability' definition here: http://docs.oracle.com/javase/8/docs/api/java/lang/ref/package-summary.html In this definition, whether a variable is used or not seems to be irrelevant. In the simplified example, as long as 'ref' stays inside the local variable slot, it is phantom reachable. Ultimately, I think it's at least dubious to expect that the example above should behave as described.
10-11-2016

The problem is this statement in the java.lang.ref package doc: " If a registered reference becomes unreachable itself, then it will never be enqueued." This is being taken as "must never be enqueued" but I think that is too strong. The point at which 'ref' is _detected_ as unreachable is not specified so while we can look at the source and say "it is unreachable after its last use", the VM is not obliged to detect that it is unreachable at exactly that point. I would revise the statement to: "If a registered reference is detected as unreachable itself, then it will never be enqueued." that leaves things loose as to when detection may, and must occur.
10-11-2016

The JLS has never specified how a Java compiler associates local variables in source with local variables in bytecode. JLS8 12.6.1 gives nothing more than a non-normative suggestion to reduce the number of reachable objects by "flushing" (good word) variables. Even if javac is reengineered to special case the storage of PhantomReference variables (and WeakReference? and SoftReference?), it would be most unusual to force every Java compiler to do precisely the same. (I was sure the complaint about bytecode storing into locals "unnecessarily" had come up before, but could only find JDK-1184986. Perhaps I'm thinking of the liveness of locals in the LocalVariableTable attribute.) Also, I don't understand which component of Oracle's JVM implementation is hurt by the overly broad storage of 'ref'. On the one hand, Zoltan seemed to say that HotSpot C1 doesn't keep 'ref' reachable: "With the PhantomSimplified program, why is 'ref' (incorrectly) enqueued into 'rq' if program executes with the interpreter? And why isn't 'ref' enqueued if the method main() is (C1)-compiled?" But on the other hand, he says "the JIT" can't detect that 'ref' has become unreachable: "There is not much the JIT can do to address this problem, either. Special compiler treatment for phantom references (keeping phantom references alive as defined by the bytecode semantics) would in theory solve the problem. But the JIT can't determine (at least not in all cases) that a reference is of type phantom." That said, it's best for a JVM implementation to address reachability, not have great expectations of source compilers.
10-11-2016

I think I get what you mean now - you are not speaking about language DA/DU - but merely about the range in which the local slot should be alive in the bytecode. Now, I think javac is correct in storing into the local slot - after all a subsequent instruction could, in principle, need that value again: PhantomReference ref = new PhantomReference(new Object(), rq); String refStr = ref.toString(); I don't see a way to do with w/o storing the result of the 'new' inside a local. What javac could do would be to basically overwrite the local slot after it detects that the variable is unreachable. In other words, the following sequence of opcodes: aconst_null astore 1 Could be generated, which, I believe, should 'flush' the contents of the slot #1 - and that slot won't be part of the GC roots anymore. Unfortunately, this logic will need to be special cased inside the javac's code generator, as the only notion of slot ranges that javac has have to do with variable scoping - meaning that, normally, a variable like 'ref' would be declared alive until the end of the block containing that var declaration. Also, for completeness, Eclipse compiler has same behavior as javac. So, I don't think we can treat this as an implementation change only - the JLS/JVMS would have to be involved too, to make sure all implementations agree on how to translate such idioms.
10-11-2016

To make sure I understand the problem - are you proposing that javac marks PhantomReference vars unreachable after a call to System.gc() ? If this is the proposed approach, I have several concerns: * what happens if the call to System.gc() happens in another method (or even another thread)? How is javac supposed to understand that? * what if GC is triggered implicitly (as part of normal operations, as opposed to an explicit System.gc) ? * how is javac supposed to understand that rq.remove will remove 'ref'? What if code has been executed in between? * are we sure that other VM implementations will be happy with this treatment? * adding special treatment to DA/DU rules based on the fact that the variable type is sensitive to the GC seems a questionable approach
10-11-2016

There interpreter can't do much to address this problem, as the semantics enforced by the interpreter closely matches that of the bytecodes. There is not much the JIT can do to address this problem, either. Special compiler treatment for phantom references (keeping phantom references alive as defined by the bytecode semantics) would in theory solve the problem. But the JIT can't determine (at least not in all cases) that a reference is of type phantom. Thus, the JIT can't keep only phantom references alive. That is, the JIT would keep other references alive as well, which would result in a performance degradation. Can somebody on the javac team please investigate if it is possible to update 'javac' to generate bytecodes that better match the semantics of Java programs using references? In the particular case of the PhantomSimplified program that would mean making 'ref' and the Object truly unreachable at the program point indicated by "->"? Below is a modified version of the PhantomSimple program. The bytecodes for main avoid storing 'ref' in local variable 1 longer than necessary. As a result, the remove() operation on the reference queue works as expected. super public class CorrectedFantom version 50:0 { static Field rq:"Ljava/lang/ref/ReferenceQueue;"; static Field errorMsg:"Ljava/lang/String;"; static Field successMsg:"Ljava/lang/String;"; public Method "<init>":"()V" stack 1 locals 1 { aload_0; invokespecial Method java/lang/Object."<init>":"()V"; return; } public static Method main:"([Ljava/lang/String;)V" throws java/lang/Exception stack 3 locals 3 { new class java/lang/ref/PhantomReference; new class Indicator; dup; invokespecial Method Indicator."<init>":"()V"; getstatic Field rq:"Ljava/lang/ref/ReferenceQueue;"; invokespecial Method java/lang/ref/PhantomReference."<init>":"(Ljava/lang/Object;Ljava/lang/ref/ReferenceQueue;)V"; invokestatic Method java/lang/System.gc:"()V"; getstatic Field rq:"Ljava/lang/ref/ReferenceQueue;"; ldc2_w long 1000l; invokevirtual Method java/lang/ref/ReferenceQueue.remove:"(J)Ljava/lang/ref/Reference;"; astore_2; aload_2; ifnonnull L43; new class java/lang/Error; dup; getstatic Field errorMsg:"Ljava/lang/String;"; invokespecial Method java/lang/Error."<init>":"(Ljava/lang/String;)V"; athrow; L43: stack_frame_type append; locals_map class java/lang/ref/Reference; getstatic Field java/lang/System.out:"Ljava/io/PrintStream;"; getstatic Field successMsg:"Ljava/lang/String;"; invokevirtual Method java/io/PrintStream.println:"(Ljava/lang/String;)V"; return; } static Method "<clinit>":"()V" stack 2 locals 0 { new class java/lang/ref/ReferenceQueue; dup; invokespecial Method java/lang/ref/ReferenceQueue."<init>":"()V"; putstatic Field rq:"Ljava/lang/ref/ReferenceQueue;"; ldc String "Test failed"; putstatic Field errorMsg:"Ljava/lang/String;"; ldc String "Test passed"; putstatic Field successMsg:"Ljava/lang/String;"; return; } } // end Class CorrectedFantom
10-11-2016

Updated ILW: - breaking specification; - failure appears due to interpretation and thus can happen with the default -Xmixed flag, however, the impact is less serious than originally thought, as it the problem causes only false positives (some unreachable phantom references are enqueued instead of being dropper) but not false negatives (reachable phantom references are not enqueue). That is, the VM does not miss and phantom references the programmer is interested in (and that is the more frequent use case); - no workaround = HLH = P2
10-11-2016

Thank you, Dmitry, for filing this bug. It is a relevant issue, even though the Phantom.java test you posted is not correct. (But many thanks for filing the test -- it helped a lot to understand the problem.) Please see more details below about the test and about the underlying problem. First, here is a simplified version of Dmitry's test: import java.lang.ref.PhantomReference; import java.lang.ref.Reference; import java.lang.ref.ReferenceQueue; public class PhantomSimplified { static ReferenceQueue rq = new ReferenceQueue(); static String errorMsg = "Test failed"; static String successMsg = "Test passed"; public static void main(final String[] args) throws Exception { PhantomReference ref = new PhantomReference(new Object(), rq); System.gc(); Reference rmRef = rq.remove(1000); if (rmRef == null) { throw new Error(errorMsg); } System.out.println(successMsg); } } In this program, neither the phantom reference, nor the Object registered with it are used after the point indicated by '->' below. public static void main(final String[] args) throws Exception { PhantomReference ref = new PhantomReference(new Object(), rq); System.out.println("ref: " + ref); // -> ref and also the newly created object become unreachable at this point. System.gc(); Both 'ref' and the Object are unreachable at '->' because - neither of them are used afterwards - "The relationship between a registered reference object and its queue is one-sided. That is, a queue does not keep track of the references that are registered with it." (from the Java SE documentation for the java.lang.ref package [1]). Therefore, both 'ref' and the Object can be garbage collected by the System.gc() call following '->'. Expecting ref to be enqueued afterwards Reference rmRef = rq.remove(1000); if (rmRef == null) { is wrong, because "If a registered reference becomes unreachable itself, then it will never be enqueued. It is the responsibility of the program using reference objects to ensure that the objects remain reachable for as long as the program is interested in their referents. It is the responsibility of the program using reference objects to ensure that the objects remain reachable for as long as the program is interested in their referents." (also from [1]) As stated before, neither 'ref' nor the Object are reachable at the System.gc(), so 'ref' must not be enqueued. So the test should throw an exception if it obtains 'ref' from the queue (and not the other way around). The UnexpectedPhantomNotification.java test below just does that: import java.lang.ref.PhantomReference; import java.lang.ref.Reference; import java.lang.ref.ReferenceQueue; public class UnexpectedPhantomNotification { static ReferenceQueue rq = new ReferenceQueue(); static String errorMsg = "Test failed"; static String successMsg = "Test passed"; public static void main(final String[] args) throws Exception { PhantomReference ref = new PhantomReference(new Object(), rq); String refStr = ref.toString(); System.out.println("The PhantomReference to be obtained: " + refStr); // The programmer expects the Object instance created in this // method and the phantom reference 'ref' to be dead at this // point. Therefore, the programmer also expects to *not* get // any notification when polling the reference queue. (As both // the reference and its referent are dead then gc() is // called, the remove() operation should not return 'ref', no // matter if GC was effectively run or not). System.gc(); Reference removedRef = rq.remove(1000); if (removedRef != null) { if (removedRef.toString().equals(refStr)) { System.out.println("The test obtained the reference " + removedRef + "from the queue although it should not have"); throw new Error(errorMsg); } } System.out.println(successMsg); } } The question(s) that needs to be answered is: With the PhantomSimplified program, why is 'ref' (incorrectly) enqueued into 'rq' if program executes with the interpreter? And why isn't 'ref' enqueued if the method main() is (C1)-compiled? The answer is that the bytecodes generated by javac keep 'ref' reachable for longer than what is intended by the programmer. That is, the bytecodes keep 'ref' reachable also at the point when System.gc() is called. Therefore, the garbage collector sees 'ref' as reachable (but not the Object) and thus enqueues 'ref' into 'rq' (which is not what the programmer intended). Please find the bytecodes below to illustrate that: public static void main(java.lang.String[]) throws java.lang.Exception; Code: 0: new #2 // class java/lang/ref/PhantomReference 3: dup # 'ref' is in the first two stack slots 4: new #3 // class java/lang/Object 7: dup 8: invokespecial #1 // Method java/lang/Object."<init>":()V 11: getstatic #4 // Field rq:Ljava/lang/ref/ReferenceQueue; 14: invokespecial #5 // Method java/lang/ref/PhantomReference."<init>":(Ljava/lang/Object;Ljava/lang/ref/ReferenceQueue;)V # 'ref' is consumed stack slot 1, 'ref' remains in stack slot 0 17: astore_1 # 'ref' is stored into local variable 1 and is removed from stack slot 0. # As a result, 'ref' is reachable for GC, even though it is not used anymore in the program. 18: invokestatic #6 // Method java/lang/System.gc:()V 21: getstatic #4 // Field rq:Ljava/lang/ref/ReferenceQueue; 24: ldc2_w #7 // long 1000l 27: invokevirtual #9 // Method java/lang/ref/ReferenceQueue.remove:(J)Ljava/lang/ref/Reference; 30: astore_2 31: aload_2 32: ifnonnull 46 35: new #10 // class java/lang/Error 38: dup 39: getstatic #11 // Field errorMsg:Ljava/lang/String; 42: invokespecial #12 // Method java/lang/Error."<init>":(Ljava/lang/String;)V 45: athrow 46: getstatic #13 // Field java/lang/System.out:Ljava/io/PrintStream; 49: getstatic #14 // Field successMsg:Ljava/lang/String; 52: invokevirtual #15 // Method java/io/PrintStream.println:(Ljava/lang/String;)V 55: return When compiling main, the C1 compiler determines that 'ref' is not used after '->' and thus it does not keep the reference around anymore. As a result, when GC is called, neither 'ref' nor the Object is found, which conforms with the expectations of the programmer. [1] http://docs.oracle.com/javase/8/docs/api/java/lang/ref/package-summary.html
10-11-2016

ILW=most likely breaking specification,failure appears with Xcomp (but could appear with Xmixed as well, only later during the test's lifetime,no workaround=MMH=P3
02-11-2016

The problem was originally discovered by JDK-8168682: jdk/test/java/lang/ClassLoader/forNameLeak/ClassForNameLeak.java fails with -Xcomp: ClassLoader was never enqueued!
01-11-2016