JDK-8246714 : URLClassLoader/JAR protocol handler poisons the global JarFile cache under concurrent load
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 8u45,14
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2020-06-06
  • Updated: 2021-03-29
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Relates :  
Relates :  
Relates :  
Description
Jar files opened via URL, URLConnection, or URLClassLoader utilize an on-by-default cache of open JarFile instances. This map is maintained by JarFileFactory and maps jar URLs to already-opened JarFiles, presumably to reduce the number of open files when the same jar is used by multiple consumers.

https://github.com/openjdk/jdk/blob/master/src/java.base/unix/classes/sun/net/www/protocol/jar/JarFileFactory.java#L80-L97

In the cases of URL and URLConnection, the "connection" to the jar resource is made via this cached JarFile, which is left in the cache after accessing the resource. The closing of an InputStream from URL.openStream or closing the connection acquired from URL.openConnection does not damage the cached JarFile.

URLClassLoader, on the other hand, stores in its list of "closeables" any jar files used to acquire resources.

https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/net/URLClassLoader.java#L294-L300

Unless the cache use is disabled via URLConnection.setUseCaches(false), closing the URLClassLoader will also close these JarFile instances.

https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/net/URLClassLoader.java#L349-L359

Closing the JarFile instance is supposed to also remove it from the global cache.

https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/sun/net/www/protocol/jar/URLJarFile.java#L167-L169

Unfortunately, the removal and close is not performed as an atomic operation, which means there's a window of time where another thread could acquire a reference to a doomed JarFile. Basically, if two isolated URLClassLoader happen to access the same jar file concurrently, and one of them is closed, the other may error out.

https://github.com/openjdk/jdk/blob/master/src/java.base/unix/classes/sun/net/www/protocol/jar/JarFileFactory.java#L112-L118

The result is that other threads may attempt to use JarFile instances that are about to be closed.

These errors have been reported many times in many forms but usually without a clear reproduction or analysis. The following example fails quickly:

https://gist.github.com/headius/32416a79faf14f63d660c40d83021bcf

Example errors:

{noformat}
java.lang.IllegalStateException: zip file closed
	at java.base/java.util.zip.ZipFile.ensureOpen(ZipFile.java:915)
	at java.base/java.util.zip.ZipFile.getInputStream(ZipFile.java:378)
	at java.base/java.util.jar.JarFile.getInputStream(JarFile.java:835)
	at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:167)
	at java.base/java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:328)
	at BrokenURLClassLoader.lambda$main$0(BrokenURLClassLoader.java:27)

java.lang.NullPointerException
	at java.base/sun.net.www.protocol.jar.JarURLConnection.connect(JarURLConnection.java:133)
	at java.base/sun.net.www.protocol.jar.JarURLConnection.getInputStream(JarURLConnection.java:155)
	at java.base/java.net.URLClassLoader.getResourceAsStream(URLClassLoader.java:328)
	at BrokenURLClassLoader.lambda$main$0(BrokenURLClassLoader.java:27)
{noformat}

We have also seen occasional "inflater closed" errors within resource/classloading logic while working on JRuby. They may be related but this case does not appear to reproduce them.

I know of two possible workarounds for user code:

* Disable the use of these caches by always acquiring the intermediate URLConnection before proceeding to access resources. It is not possible to set an entire URLClassLoader to not use the cache, however, so this does not help classloading requests from elsewhere.
* Do not close URLClassLoader. This will leave open JarFile instances in the cache, which may be benign (static jar files from the filesystem) or malignant (temporary jar files created or unpacked by library or app code).

To summarize: URLClassLoader's usage of the global JarFile cache leads to improperly closed JarFile instances and subsequent errors under concurrent load. All uses of URLClassLoader with jar files will potentially fail under concurrent load. This affects OpenJDK versions at least back to Java 7, when URLClassLoader.close was added. 
Comments
I should add that our workaround does not protect us from anyone else closing a URLClassLoader that cached the same loose jar files. If a non-JRuby URLClassLoader happens to access resources from a jar file that JRuby uses, we may see these errors again. This case should represent a very small part of JRuby uses, since we typically only add new jar URLs to support JRuby libraries that ship jar files, or to allow dynamically loading jar files that were not put on the system classpath (and therefore unlikely to be also used by other code in the same JVM).
09-06-2020

For now, we have modified the close operation in JRuby's classloader to only close the jar files we know are temporary and specific to a given JRuby instance. Jar files loose on the system will be left open, unless JRuby is configured to also close those jars (via URLClassLoader.close). https://github.com/jruby/jruby/pull/6273
09-06-2020

Disabling the caching globally for "jar" is definitely an option for projects that don't have to support Java 8 and don't want (or don't need) the benefits of the cache.
08-06-2020

An other possible temporary workaround would be to disable caching by default for the `jar` protocol - as the first thing in the main() method, or before creating any custom URLClassLoader: https://docs.oracle.com/en/java/javase/14/docs/api/java.base/java/net/URLConnection.html#setDefaultUseCaches(java.lang.String,boolean) The per-protocol default is available since Java 9.
08-06-2020

Not closing the URLClassLoader was the first workaround I considered, but it doesn't work in our case because we unpack temporary jars on a per-JRuby-instance basis. We have teardown logic that cleans up the temporary files as part of closing the classloader. It's not too bad for jar files that exist on the filesystem, since they'll be cached and reused across classloaders. We are experimenting with a multi-level classloader system where the temp jars go into a child classloader that's closed, while the static already-on-filesystem jars go into the parent classloader which we do not close.
07-06-2020

This is.a nasty bug. A short solution would be to keep JAR files opened by URLClassLoader out of the JAR file cache (to at least help the cases where there are many URLClassLoader with URLs to the same JAR file). More medium term will need deeper surgery (maybe a ewrite) on the JAR protocol handler as it is a ver problematic area.
07-06-2020

Credit to a JRuby user for coming up with a neat little reproduction that allowed us to finally reproduce and investigate this issue. https://github.com/jruby/jruby/issues/6218
06-06-2020