JDK-8156014 : (sl) File handle leaked for jar of class loaded with java.util.ServiceLoader
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.util
  • Affected Version: 8u40
  • Priority: P4
  • Status: Closed
  • Resolution: Won't Fix
  • OS: linux_ubuntu
  • CPU: x86_64
  • Submitted: 2015-09-25
  • Updated: 2018-07-17
  • Resolved: 2018-07-17
Related Reports
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.8.0_40"
Java(TM) SE Runtime Environment (build 1.8.0_40-b26)
Java HotSpot(TM) 64-Bit Server VM (build 25.40-b25, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux smidge 3.13.0-63-generic #103-Ubuntu SMP Fri Aug 14 21:42:59 UTC 2015 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
If a service class is loaded from jar file via the java.util.ServiceLoader, if that jar file is subsequently deleted, a native file handle is leaked. This occurs when the jar file is NOT on the system classpath, but on a child URLClassLoader. 

The most easily seen use case is the redeployment of  a webapp context: the webapp context is deployed as a war file, unpacked by the web server to a temporary location, and a URLClassLoader created for the jars in the unpacked WEB-INF/lib location. If one of these jars contains a javax.servlet.ServletContainerInitializer that is loaded via the java.util.ServiceLoader, a subsequent redeploy of the webapp (involving the deletion of the previous temporary unpack location) will leave a file handle open, marked as deleted like so:

java    18249 janb  DEL    REG                8,5          17705538 /tmp/foo-1443160317112/sci-1.0-SNAPSHOT.jar
java    18249 janb   72r   REG                8,5     2142 17705538 /tmp/foo-1443160317112/sci-1.0-SNAPSHOT.jar (deleted)

Note that the URLClassLoader was properly disposed of, with a call to close() first, and that the ServiceLoader's reload() method was also called to flush it's internal cache. 

This is NOT specific to the ServletContainerInitializer, it happens with any class that is loaded via the ServiceLoader.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Create a simple interface eg MyInterface
2. Create a simple implementation of that interface eg MyImplementation and put it in a jar eg impl.jar
3. Create an application that only has the interface on the runtime classpath, and that copies the impl.jar to a tmp location and then creates a URLClassloader with this location.  The app should then use the ServiceLoader to load MyImplementation using MyInterface.class.  Then, the app should clear out the ServiceLoader's cache, close the URLClassLoader and delete the jar file in the temporary location. 
4. Using lsof -p <pid> you will see the file handles remaining to the impl.jar.

I have a small example maven project that I could attach to this issue if necessary.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Expected result is that the impl.jar should disappear from the list of open files.
ACTUAL -
File handles remain to the deleted impl.jar until the app's process is stopped.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
MyInterface.java
-----------------------

package leaker;

public interface MyInterface
{
   public long getLong();
}


MyImpl.java
-----------------
package leaker;

public class MyImpl implements MyInterface
{
   public long getLong()
   {
      return 1L;
   }
}

META-INF/services/leaker.MyInterface
----------------------------------------------------
leaker.MyImpl


App.java
-------------
package leaker;


import java.io.File;
import java.io.IOException;
import java.net.URL;
import java.net.URLClassLoader;
import java.nio.file.*;
import java.nio.file.attribute.*;
import java.util.ServiceLoader;


public class App
{
   ClassLoader old;
   ServiceLoader<MyInterface> services;
   URLClassLoader ucl;
   File tmpdir;

   public App()
   {
   }


   public void copy (File from, File to)
   throws IOException
   {
       Files.copy(from.toPath(), to.toPath());
   }




    public void setupLoader ()
    throws Exception
    {
         File jars = new File("../sci/target");
         File f = new File(jars, "sci-1.0-SNAPSHOT.jar");

         tmpdir = new File("/tmp/foo-"+System.currentTimeMillis());
         tmpdir.mkdirs();
         File copy = new File (tmpdir, "sci-1.0-SNAPSHOT.jar");

         copy(f, copy);
         System.err.println("Copied jar containing service to: "+copy.getAbsolutePath());

         URL u = copy.toURL();
         old = Thread.currentThread().getContextClassLoader();
         ucl = new URLClassLoader(new URL[]{u}, old);
         Thread.currentThread().setContextClassLoader(ucl);

         services = ServiceLoader.load(MyInterface.class);

         for (MyInterface m:services)
         {
             System.err.println("Loaded service: "+m.getClass().getName());
         }
    }




   public void teardownLoader()
   throws Exception
   {
      //replace the classloader
      Thread.currentThread().setContextClassLoader(old);

      //clear services cache
      services.reload();

      //close all urls in classloader
      ucl.close();
      

      ucl = null;
      services = null;
      System.gc();


      //delete the tmp directory
      Files.walkFileTree(tmpdir.toPath(), new SimpleFileVisitor<Path>() 
      {
         @Override
         public FileVisitResult visitFile(Path file, BasicFileAttributes attrs)
             throws IOException
         {
             Files.delete(file);
             return FileVisitResult.CONTINUE;
         }
         @Override
         public FileVisitResult postVisitDirectory(Path dir, IOException e)
             throws IOException
         {
             if (e == null) 
             {
                 Files.delete(dir);
                 return FileVisitResult.CONTINUE;
             } 
             else 
             {
                 // directory iteration failed
                 throw e;
             }
         }
     });
   }




   public static final void main (String[] args)
   {
        App app = new App();

        try
        {
              System.err.println();
              System.err.println();
              System.err.println("Do \"ps -ef | grep maven\" to find the pid for this process using another window.");
              System.err.println();
              System.err.println("Setting up service loader ...");
              app.setupLoader();
              System.err.println("Services loaded. Do \"lsof -p <this pid> | grep sci\" to see the file handle to the jar containing the service class.");
              System.err.println("Sleeping 30sec ...");
              Thread.currentThread().sleep(30000);
              System.err.println("Clearing loader ...");
              app.teardownLoader();
              System.err.println("Cleared loader. Do \"lsof -p <this pid> | grep sci\" to see the leaked deleted file handle to the jar containing the service class.");
              System.err.println("Sleeping 20sec ...");
              Thread.currentThread().sleep(20000);
              System.err.println("Done");

        } 
        catch (Exception e)
        {
          e.printStackTrace();
        }

   }

}


---------- END SOURCE ----------


Comments
This issue is JDK 8 and older. There are no current plans to backport the changes to disable the JAR file caching in the older releases so closing this bug.
17-07-2018

From submitter: I have just tested with jdk9+149 early access and no longer see the problem. However, the problem is still present in jdk8u112.
25-01-2017

Much of ServiceLoader has been replaced in JDK 9 by way of adding support for modules. So it doesn't have a specific fix that I can point at. Note that JDK-8013099 seems to be the same issue.
04-05-2016

I assume the issue here is caching in the jar protocol handler. ServiceLoader does not use this in JDK 9 and would be interesting to see if the submitter could repeat the tests with JDK 9.
04-05-2016

Could not reproduce with JDK 8u66/JDK9. Once the loader is cleared, don't see any handles to the jar. Marking as Cannot reproduce. Here is the output with JDK 8u66: ----------------------------------------------------- Window 1: p@p-VirtualBox:/shared$ /shared/jdk1.8.0_66/bin/javac leaker/App.java Note: leaker/App.java uses or overrides a deprecated API. Note: Recompile with -Xlint:deprecation for details. p@p-VirtualBox:/shared$ /shared/jdk1.8.0_66/bin/java leaker.App Do "ps -ef | grep maven" to find the pid for this process using another window. Setting up service loader ... Copied jar containing service to: /tmp/foo-1462347224475/sci-1.0-SNAPSHOT.jar Loaded service: leaker.MyImpl Services loaded. Do "lsof -p <this pid> | grep sci" to see the file handle to the jar containing the service class. Sleeping 30sec ... Clearing loader ... Cleared loader. Do "lsof -p <this pid> | grep sci" to see the leaked deleted file handle to the jar containing the service class. Sleeping 20sec ... Done Window 2: p@p-VirtualBox:/shared$ ps -ef |grep java p 2918 2357 9 13:03 pts/0 00:00:01 /shared/jdk1.8.0_66/bin/java leaker.App p 2932 2433 0 13:03 pts/3 00:00:00 grep --color=auto java p@p-VirtualBox:/shared$ lsof -p 2918 |grep sci java 2918 p mem REG 8,1 4247 608522 /tmp/foo-1462347224475/sci-1.0-SNAPSHOT.jar java 2918 p 4r REG 8,1 4247 608522 /tmp/foo-1462347224475/sci-1.0-SNAPSHOT.jar p@p-VirtualBox:/shared$ lsof -p 2918 |grep sci p@p-VirtualBox:/shared$ lsof -p 2918 |grep sci
04-05-2016