ADDITIONAL SYSTEM INFORMATION :
Reproduced on:
RHEL6, java-1.8.0-openjdk-1.8.0.232.b09-1.el6_10.x86_64
RHEL7, java-1.8.0-openjdk-1.8.0.242.b08-0.el7_7.x86_64
Ubuntu 18.04, OpenJDK 11.0.6+10-1ubuntu1~18.04.1
Ubuntu 18.04, OpenJDK from master branch of https://github.com/openjdk/jdk
RHEL7, IBM java 1.8.0_221
A DESCRIPTION OF THE PROBLEM :
When using HttpURLConnection or HttpsURLConnection, the underlying HttpClient object is normally added to the KeepAliveCache for reuse when the response body has been fully read. However, if the response body is never fully read (for example, if a 201 response code is returned by the server and the code that is using HttpURLConnection reads the "Location" header but doesn't bother reading the response body), then the underlying HttpClient object is added to the KeepAliveCache by the MeteredStream/KeepAliveStream finalizer when the HttpURLConnection/HttpClient/InputStream/etc is garbage collected.
Unfortunately, "Resurrecting" the HttpClient object from the MeteredStream/KeepAliveStream finalizer like this is unreliable and leads to intermittent failures in subsequent HttpURLConnection calls. For example, GC may identify that both the MeteredStream and Socket underlying an HttpURLConnection are ready to be finalized, and may finalize the MeteredStream before the Socket causing the HttpClient object to be "resurrected" and added to the KeepAliveCache. If a new HttpURLConnection is created (in another thread that is running in parallel with GC) after this happens and before GC gets around to finalizing the Socket, the new HttpURLConnection may reuse this HttpClient. If GC then gets around to finalizing the Socket while the new HttpURLConnection is actively using it, the Socket finalizer will close the underlying connection while it is in use and cause an error. If sun.net.http.retryPost is set to "false" then this error may propagate up to user-level code. Alternatively, if the above condition is combined with conditions that trigger the bug described in https://bugs.openjdk.java.net/browse/JDK-8209178 then such errors can lead to hung connections after the HttpClient automatically retries the interrupted connection.
I've posted a proposed fix for this at https://github.com/PaulSD/jdk/commit/9a3fb5d8f6ec31bfcdf8a25588383d46014811fa
This fix works by adding each HttpClient object to a new data structure in KeepAliveCache immediately after it is created, then using a WeakReference/ReferenceQueue on the HttpURLConnection/HttpsURLConnection to detect when the associated HttpURLConnection/HttpsURLConnection is garbage collected so the KeepAliveCache can either drop the HttpClient reference or move it to the "idle and available for reuse" data structure as appropriate. This eliminates the current "object resurrection" behavior and prevents GC from finalizing the HttpClient or any underlying objects while they are still in use.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Download reproduction script/code from: https://github.com/PaulSD/jdk/tree/test
./test
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Garbage collector did not collect in expected order within 3 tries, giving up
ACTUAL -
Erroneous behavior has been successfully reproduced: Underlying Socket is being closed after request and before response in main thread
---------- BEGIN SOURCE ----------
https://github.com/PaulSD/jdk/blob/test/Test.java
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
Ensure that all code that uses HttpURLConnection or HttpsURLConnection always calls `myhttpurlconnection.getInputStream().close();` before abandoning the HttpURLConnection/HttpsURLConnection object.
FREQUENCY : occasionally