FULL PRODUCT VERSION :
java version "1.7.0_02"
Java(TM) SE Runtime Environment (build 1.7.0_02-b13)
Java HotSpot(TM) 64-Bit Server VM (build 22.0-b10, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux etc003520a.etc.test.gs.com 2.6.32-131.14.1.el6.x86_64 #1 SMP Fri Sep 9 07:13:46 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux

Red Hat Enterprise Linux Server release 6.1 (Santiago)


EXTRA RELEVANT SYSTEM CONFIGURATION :
8 cores machine

A DESCRIPTION OF THE PROBLEM :
If you use a java.lang.instrument agent trying to instrument a java application with a daemon thread, you could trigger an assert inside the JPLIS code:

*** java.lang.instrument ASSERTION FAILED ***: "error == JVMTI_ERROR_NONE" at ../../../src/share/instrument/Reentrancy.c line: 161


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
First, I have a test program here that creates a background *daemon* thread and load lots of classes:


import java.io.File;
import java.lang.reflect.InvocationHandler;
import java.lang.reflect.Method;
import java.lang.reflect.Proxy;
import java.net.URL;
import java.net.URLClassLoader;
import java.util.jar.JarFile;

class LoadClasses implements Runnable {
       URLClassLoader loader = null;
       
       public LoadClasses(File f) throws Exception {
              loader = new URLClassLoader(new URL[] {f.toURI().toURL()});
       }
       
       public void run() {
              for(int i=1; i<=10000; i++) {
                     try {
                           Class c = loader.loadClass("com.xx.ClassinLargeJar"+i);
                           System.out.println("Loaded: c.getName()");
                     } catch (Exception e) {
                           System.err.println("Can't load class");
                           e.printStackTrace();
                     }
              }
       }
}

public class TestDaemonThread {
       
       public static void main(String[] args) throws Exception {
              Thread t = new Thread(new LoadClasses(new File(args[0])));
              t.setDaemon(true);
              t.start();
              Thread.sleep(1000);
       }
}

Also create a java.lang.instrument dummy agent that does nothing:

import java.lang.instrument.ClassFileTransformer;
import java.lang.instrument.IllegalClassFormatException;
import java.lang.instrument.Instrumentation;
import java.security.ProtectionDomain;


public class DummyAgent implements ClassFileTransformer {

       @Override
       public byte[] transform(ClassLoader loader, String className,
                     Class<?> classBeingRedefined, ProtectionDomain protectionDomain,
                     byte[] classfileBuffer) throws IllegalClassFormatException {
              
              System.out.println("seen "+className);
              return classfileBuffer;
       }
       
       public static void premain(String agentArgs, Instrumentation inst) {
              inst.addTransformer(new DummyAgent(), false);
       }

}

If you run the agent in the command line multiple times, it would trigger the assert around 6 out of 10 times:



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
No assert should occur.
ACTUAL -
$java -javaagent:$HOME/DummyAgent.jar -jar $HOME/DummyAgent.jar $HOME/TestLargeJar.jar
seen sun/launcher/LauncherHelper
seen java/lang/Enum
seen java/util/jar/Manifest
seen java/io/ByteArrayInputStream
seen java/util/jar/Attributes
seen java/util/jar/Manifest$FastInputStream
seen sun/nio/cs/UTF_8$Decoder
seen java/util/jar/Attributes$Name
���������������.. removed for brevity ������������������������������.
seen com/xx/ClassinLargeJar793
Loaded: c.getName()
seen com/xx/ClassinLargeJar794
Loaded: c.getName()
seen com/xx/ClassinLargeJar795
Loaded: c.getName()
seen com/xx/ClassinLargeJar796
seen java/lang/Shutdown
Loaded: c.getName()
seen java/lang/Shutdown$Lock
seen com/xx/ClassinLargeJar797
*** java.lang.instrument ASSERTION FAILED ***: "error == JVMTI_ERROR_NONE" at ../../../src/share/instrument/Reentrancy.c line: 161


REPRODUCIBILITY :
This bug can be reproduced always.
Note - if the customer who report this has a support contract then please file this through support for proper prioritization. Thanks Tuva Palm, Sr Eng Mgr Java SE Serviceability