JDK-8239054 : JarFile created by URLJarFile is not closed properly
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 8u241,11,13.0.2,15
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • OS: windows_10
  • CPU: x86_64
  • Submitted: 2020-02-12
  • Updated: 2024-08-16
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Relates :  
Relates :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Issue was also reproduced with Java 1.8.0_221-x64 and Java 11.0.2

A DESCRIPTION OF THE PROBLEM :
Given a jar file entry URL
When closing the InputStream created by a call to URL.openStream API
Then the close method inherited by the java.util.jar.JarFile created by the JDK is not called
And an attempt to delete the jar file using Java libraries or Windows Explorer fails with "The process cannot access the file because it is being used by another process."

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
For the below test code, create a the folder D:\temp\jdkbug and copy into it jrt-fs.jar (from the jdk).
- One can of course change the directories and adapt the code
- There's a huge chance any jar and any entry of that jar will reproduce the issue


---------- BEGIN SOURCE ----------
import java.io.IOException;
import java.io.InputStream;
import java.net.URL;
import java.nio.file.Files;
import java.nio.file.Paths;

public class UrlStreamClose {
    public static void main(String[] args) throws Exception {
        try (InputStream is = new URL("jar:file:D:/temp/jdkbug/jrt-fs.jar!/META-INF/MANIFEST.MF").openStream()) {
            System.out.println("hello");
        }
        Files.delete(Paths.get("D:\\temp\\jdkbug\\jrt-fs.jar"));
    }
}
---------- END SOURCE ----------

FREQUENCY : always



Comments
Two possible workaround: 1. disable caching on the `JarURLConnection` before opening the stream. 2. disable caching globally for the `jar:` protocol This is related to JDK-8132359 and JDK-8232854 and need to be examined in conjunction with these other issues.
08-07-2020

test call flows for scenario using explicit JarFile, ZipFile, and ZipEntry i.e. the internal workings of the first scenario. Explicitly close the JarFile after the try with resources block. Delete succeeds private static void testWithJarFile() throws IOException, MalformedURLException { JarFile testJarFile = new JarFile(JAR_FILE_PATH); try (InputStream is = testJarFile.getInputStream(testJarFile.getEntry(JAR_FILE_ENTRY))) { System.out.println("jar file opened"); System.out.println("is is instance of == " + is.getClass().getName()); } testJarFile.close(); System.out.println("Delete jar file"); Files.delete(Paths.get(JAR_FILE_PATH)); System.out.println("Delete completed"); } in attachment testWithJarFile-call-flows
18-04-2020

trace call flow output for test scenario using JarURLConnection and setting useCaches to false -- delete succeeds private static void testWithJarUrlConnection() throws IOException, MalformedURLException { URL jarFileUrl = new URL(JAR_FILE_URL); JarURLConnection jarUrlConnection = (JarURLConnection) jarFileUrl.openConnection(); // URL jarFileUrl2 = jarUrlConnection.getJarFileURL(); System.out.println("jar:file url protocol is " + jarFileUrl.getProtocol()); System.out.println("jarFileUrl == " + jarFileUrl); System.out.println("defaultUseCaches == " + jarUrlConnection.getDefaultUseCaches()); System.out.println("useCaches == " + jarUrlConnection.getUseCaches()); jarUrlConnection.setUseCaches(false); // System.out.println("jarFileUrl2 protocol is " + jarFileUrl2.getProtocol()); // System.out.println("jarFileUrl2 == " + jarFileUrl2); try (InputStream is = jarUrlConnection.getInputStream()) { System.out.println("jar file opened"); System.out.println("is is instance of == " + is.getClass().getName()); } Files.delete(Paths.get(JAR_FILE_PATH)); } in attachment testWithJarUrlConnection-call-flows.rtf
18-04-2020

trace call flow output for the original test scenario using URL private static void testWithJarFileUrl() throws IOException, MalformedURLException { try (InputStream is = new URL(JAR_FILE_URL).openStream()) { System.out.println("jar file opened"); System.out.println("is is instance of == " + is.getClass().getName()); } Files.delete(Paths.get(JAR_FILE_PATH)); } in attachment jdk-8239054-call-flows-original-test.rtf
18-04-2020

a further consideration in these scenarios (JDK-8132359, JDK-8232854) is that on macOS (probably linux) the delete of a resource i.e the jar file proceeds even though that resource is cached, which in turn invalidates the caching. So the question arises: Is it correct that a delete of a cached file succeeds, as in the case of macOS, while the resource is in use OR is the fact that Windows prohibits the deletion a more reasonable strategy ?
31-03-2020

There is also the question of whether caching, where provided, should be enabled OOTB, or whether it should be left OFF to be enabled at the discretion of the application.
30-03-2020

The above property setting would be global across all possible protocols It is possible to set it at a per protocol basis - although it is not concurrency friendly as one thread could impact the setting of another requiring application level co-ordination for the setting As such a simple alternative workaround for these issues on Windows is to use the URLConnection.setDefaultUseCaches(String protocol, boolean defValue) method to set jar caching to false. Thus, allowing an application to delete a recently accessed jar file, if required. URLConnection.setDefaultUseCaches("jar", false); will allow the jar file to be deleted after the main block in each test is executed.
30-03-2020

adding a system property java.net.URLConnection.defaultUseCaches={true|false} will allow an application overcome the perceived jar file delete problem (on Windows) adding the following to URLConnection, to set the member variable defaultUseCaches from a property setting, and facilitate the disabling of the JarFile cache mechanism. This will allow the useCaches boolean flag to be set to false, and allow the deletion of the jarfile. {code} diff -r 8e9261c404fc src/java.base/share/classes/java/net/URLConnection.java --- a/src/java.base/share/classes/java/net/URLConnection.java Fri Mar 13 10:29:03 2020 -0400 +++ b/src/java.base/share/classes/java/net/URLConnection.java Wed Mar 25 22:35:15 2020 +0000 @@ -223,7 +223,16 @@ */ protected boolean allowUserInteraction = defaultAllowUserInteraction; - private static volatile boolean defaultUseCaches = true; + //private static volatile boolean defaultUseCaches = true; + + private static volatile boolean defaultUseCaches = + getDefaultUseCachesProperty(); + + private static boolean getDefaultUseCachesProperty() { + return Boolean.parseBoolean( + GetPropertyAction. + privilegedGetProperty("java.net.URLConnection.defaultUseCaches", "true")); + } {code} java -Djava.net.URLConnection.defaultUseCaches=false JarFileCloseTest will see the jarfile deleted
25-03-2020

Looking at this issue. The jar URL is a bit of an onion, as it has multiple layers. It is jar and then file, creating multiple layer URLs. Similarly the creation of an Input stream for a particular entry specified in the URL can be seen to construct multiple layers which when "peeled" away results a JarFile/ZipFile being created and a JarFileInputStream FliterInputStream encapsulating a ZipFileInflaterInputStream. The call flows in the various comments below show various invocations in the construction of the relevant objects URL URLJarFile (JarFile ZipFile) JarURLConnection JarURLInputStream ZipFileInflaterInputStream and so on. During the autoclose call flow, it can be seen that a close is invoked on JarURLInputStream, FilterInputStream and ZipFileInflaterInputStream. Then the associated Cleaner invocations are made. The essence of all this manufacturing can be reduced to a basic equivalent scenario of creating a JarFile opening an InputStream with an explicit references to a ZipEntry for the manifest. This will result in the same FileSystemException being thrown for the delete private static void testWithJarFile() throws IOException, MalformedURLException { JarFile testJarFile = new JarFile(JAR_FILE_PATH); try (InputStream is = testJarFile.getInputStream(testJarFile.getEntry(JAR_FILE_ENTRY))) { System.out.println("input stream is of type == " + is.getClass().getName()); System.out.println("close the jar file automatically"); } System.out.println("Delete jar file"); Files.delete(Paths.get(JAR_FILE_PATH)); System.out.println("Delete completed"); } Exploring the calls flows a bit, all seems to be in order. Streams are being created, and closed with associated Cleaners being invoked. This is not a problem on macOS, the delete executes without problem BUT on Windows an FileSystemException is thrown. Looking at the Windows FS API docs there is a hint in the following https://docs.microsoft.com/en-us/windows/win32/api/fileapi/nf-fileapi-deletefilew The suggestion is that if a file is open for reading, then it can't be deleted. In both test scenarios the jar file is open for reading with the creation of the ZipEntry. The suggestion from the exception is that "something" is hold an open handle to the file. So although the calls flows are indicating that the autoclosing of the stream and associated resources has occurred, on Windows there is a "stray reference" holding onto the file to be closed. If the basic scenario using the JarFile and then an InputStream from the explcitly ceated ZipEntry, is modified to close the JarFile explicitly prior to invoking the delete, then the delete will succeed. private static void testWithJarFile() throws IOException, MalformedURLException { JarFile testJarFile = new JarFile(JAR_FILE_PATH); try (InputStream is = testJarFile.getInputStream(testJarFile.getEntry(JAR_FILE_ENTRY))) { System.out.println("input stream is of type == " + is.getClass().getName()); System.out.println("close the jar file automatically"); // is.close(); } testJarFile.close(); System.out.println("Delete jar file"); Files.delete(Paths.get(JAR_FILE_PATH)); System.out.println("Delete completed"); } So it would seem that the explicit close of the JarFile associated with the InputStream, in the URL scenario is required for a delete to succeed. But, there is no such access in the URL scenario. So in the URL example, examine the call flows a bit more deeply and it is seen that JarURLInputStream.close is called and this will explicitly close the jarFile BUT only if useCaches is false. class JarURLInputStream extends java.io.FilterInputStream { JarURLInputStream (InputStream src) { super (src); } public void close () throws IOException { try { super.close(); } finally { if (!getUseCaches()) { jarFile.close(); } } } } Thus the jarFile is never closed and in the Windows case the OS File object handle is not released. Hence the delete can't take place. This doesn't seem to be the case on macOS, or at least doesn't interfere with the delete operation. In the URL example, it is not possible to obtain a reference to the JarFile without doing a few extra explicit steps in instantiating the InputStream. So creating a thrid test example, which creates the URL, then a JarURLConnection, read useCaches, which will show that it is true and then set it to (setUseCaches) false, and then executes the getInputStream followed by the autoclose and then execute the delete and it succeeds. private static void testWithJarUrlConnection() throws IOException, MalformedURLException { URL jarFileUrl = new URL(JAR_FILE_URL); JarURLConnection jarUrlConnection = (JarURLConnection) jarFileUrl.openConnection(); System.out.println("jar:file url protocol is " + jarFileUrl.getProtocol()); System.out.println("jarFileUrl == " + jarFileUrl); System.out.println("defaultUseCaches == " + jarUrlConnection.getDefaultUseCaches()); System.out.println("useCaches == " + jarUrlConnection.getUseCaches()); jarUrlConnection.setUseCaches(false); try (InputStream is = jarUrlConnection.getInputStream()) { System.out.println("input stream is of type == " + is.getClass().getName()); System.out.println("close the jar file automatically"); } Files.delete(Paths.get(JAR_FILE_PATH)); } So on Windows, the underlying OS file object handle, created for the reading of the ZipEntry, is not fully released and so the delete cannot occur as an open file handle is retained on the jar file. In the URL case the non access to the useCaches property prohibits the altering of the default behaviour to allow the jar file deletion. I suppose at a minimum an API note or an implNote indicating the variation in behaviour for Windows would be useful. Alternatively, don't enable caching by default and that will give consistency across OS platforms - are there side effects or backward compatibility issues ? This latter strategy could be provided by adding a network property java.net.URLConnection.defaultUseCaches {yes|no} - allowing the use of cache to be turned off SUMMARY: For the test example given for URL, the useCaches internal property prohibits the release of the OS File object handle, on Windows, and so prohibits the deletion of the jar file. Similarly when using more explicit access to JarURLConnection object, the same Exception behaviour occurs. BUT it is possible by setting the useCaches property equal to false to allow a Files delete operation to succeed. There is exists a similar scenario when explicitly manipulating JarFile and ZipFile objects. BUT by explicitly closing the created JarFile, then the delete operation can be completed successfully.
25-03-2020

Sort of! I've put a temporary solution in the updates. Alan wishes to pursue an alternative for JDK15. (once its in I'm hoping I can backport it to the updates) My fix went in under https://bugs.openjdk.java.net/browse/JDK-8232854
20-02-2020

Is this related to JDK-8132359?
14-02-2020

I assume this is the jar protocol handler.
14-02-2020

JarFile created by URLJarFile fails to closed properly with exception "The process cannot access the file because it is being used by another process." While closing the InputStream created by a call to URL.openStream API, the close method inherited by the java.util.jar.JarFile created by the JDK is not called. This issue is reproducible as stated with JDK 8u as well as 15 ea build. Result: ========== 8u241: Fail 11: Fail 13.0.2: Fail 14 ea b36: Fail 15 ea b10: Fail To verify, run the following steps: - Create a the folder D:\temp\jdkbug and copy into it jrt-fs.jar (from the jdk). Note: Likely any jar and any entry of that jar will reproduce the issue - Run the attached test case Output with JDK 13.0.2: ====================== $ java UrlStreamClose hello Exception in thread "main" java.nio.file.FileSystemException: D:\temp\jdkbug\jrt-fs.jar: The process cannot access the file because it is being used by another process at java.base/sun.nio.fs.WindowsException.translateToIOException(WindowsException.java:92) at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:103) at java.base/sun.nio.fs.WindowsException.rethrowAsIOException(WindowsException.java:108) at java.base/sun.nio.fs.WindowsFileSystemProvider.implDelete(WindowsFileSystemProvider.java:274) at java.base/sun.nio.fs.AbstractFileSystemProvider.delete(AbstractFileSystemProvider.java:105) at java.base/java.nio.file.Files.delete(Files.java:1146) at UrlStreamClose.main(UrlStreamClose.java:12)
14-02-2020