FULL PRODUCT VERSION :
java version "9.0.4"
Java(TM) SE Runtime Environment (build 9.0.4+11)
Java HotSpot(TM) 64-Bit Server VM (build 9.0.4+11, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601]

A DESCRIPTION OF THE PROBLEM :
We're using Nashorn in OSGi environment, where each bundle has own class-loader and on update the bundle gets a new class-loader. We noticed that memory leak occurred when updating bundles which generate interface implementations from script objects, using javax.scripting.Invocable.getInterface methods.

A test application for reproducing the leak is provided. It consist of 3 files:
  - TestService.java - a simple interface with a single method - "callMe". This interface is implemented by the test script
  - CustomClassLoader.java - custom class loader, which loads the TestService class
  - Test.java - the main test class. It start a thread that repeatedly evaluates a test script - object with single "callMe" function - and generates interface implementation from the resulting object. On each execution the TestService class is loaded anew and new ScriptEngine is used. Periodically the used heap memory is printed.

On java version "9.0.4" we observe constant increase in used heap memory. 

The leak is even more severe on java version "1.8.0_121", where constant increase in loaded classes and heap memory is observed.

On java 1.7, with Rhino engine, the leaks is not observed.

REGRESSION.  Last worked in version 7u79

ADDITIONAL REGRESSION INFORMATION: 
java version "1.7.0_79"
Java(TM) SE Runtime Environment (build 1.7.0_79-b15)
Java HotSpot(TM) 64-Bit Server VM (build 24.79-b02, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Compile the provided java files - TestService.java, CustomClassLoader.java and Test.java - and run Test class.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Used heap remains in some boundaries.
ACTUAL -
Used heap increases constantly.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
---------- BEGIN TestService.java ----------

public interface TestService {

  public int callMe(int num);
}

---------- END TestService.java ----------

---------- BEGIN CustomClassLoader.java ----------

import java.io.ByteArrayOutputStream;
import java.io.IOException;
import java.io.InputStream;

public class CustomClassLoader extends ClassLoader {

  public static final String THE_CLASS = "the_class";

  public CustomClassLoader(ClassLoader parent) {
    super(parent);
  }

  @Override
  public Class<?> loadClass(String name) throws ClassNotFoundException {
    if (THE_CLASS.equals(name)) {
      try {
        byte[] data = loadClassData("TestService.class");

        return defineClass("TestService", data, 0, data.length);
      } catch (IOException ex) {
        ex.printStackTrace();
        return null;
      }
    }

    return super.loadClass(name);
  }

  private byte[] loadClassData(String name) throws IOException {
    InputStream in = getClass().getResourceAsStream(name);

    try {
      ByteArrayOutputStream out = new ByteArrayOutputStream();
      byte[] buff = new byte[1024];

      int read = 0;
      while ((read = in.read(buff)) > 0) {
        out.write(buff, 0, read);
      }

      return out.toByteArray();
    } finally {
      in.close();
    }

  }

}

---------- END CustomClassLoader.java ----------

---------- BEGIN Test .java ----------

import java.lang.reflect.Method;

import javax.script.Invocable;
import javax.script.ScriptEngine;
import javax.script.ScriptEngineManager;

public class Test {

  private static final String SCRIPT      = "var impl = { \r\n" +
      "  callMe : function(number) {\r\n" +
      "    return number + 1;\r\n" +
      "  }\r\n" +
      "}\r\n" +
      "  \r\n" +
      "impl";

  private static final long   DUMP_PERIOD = 1000 * 2;
  private static final int    MB          = 1024 * 1024;

  public static void main(String[] args) throws Exception {
    System.out.format("Java Version: %s %n", System.getProperty("java.version"));

    long mark = 0;
    int counter = 0;

    while (true) {
      test(counter);

      if (System.currentTimeMillis() - mark > DUMP_PERIOD) {
        System.gc();
        long used = Runtime.getRuntime().totalMemory() - Runtime.getRuntime().freeMemory();

        System.out.format("%.2f MB%n", (float) used / MB);

        mark = System.currentTimeMillis();
      }

      Thread.sleep(50);

      counter++;
    }
  }

  public static void test(int index) throws Exception {
    ClassLoader cl = new CustomClassLoader(Test.class.getClassLoader());

    Class<?> clazz = cl.loadClass(CustomClassLoader.THE_CLASS);
    Method method = clazz.getMethod("callMe", int.class);

    ScriptEngineManager manager = new ScriptEngineManager();
    ScriptEngine engine = manager.getEngineByExtension("js");
    Invocable invocable = (Invocable) engine;

    Object obj = engine.eval(SCRIPT);

    Object service = invocable.getInterface(obj, clazz);

    int result = (Integer) method.invoke(service, index);

    if (result != index + 1) {
      System.err.format("Wrong result: expected %d, but was %d %n", index + 1, result);
    }
  }
}


---------- END Test .java ----------

---------- END SOURCE ----------