JDK-8058536 : java/lang/instrument/NativeMethodPrefixAgent.java fails due to VirtualMachineError: out of space in CodeCache for method handle intrinsic
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 8u40,9
  • Priority: P1
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2014-09-16
  • Updated: 2017-08-10
  • Resolved: 2014-09-27
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8 JDK 9
8u40Fixed 9 b35Fixed
Related Reports
Relates :  
JDK-8058733 - [TESTBUG] java/lang/invoke/LFCaching/LFSingleThreadCachingTest.java and LFMultiThreadCachingTest.java failed on some platforms due to java.lang.VirtualMachineError
Relates :  
JDK-8068234 - java/lang/instrument/NativeMethodPrefixAgent.java is still in exclude list
Relates :  
JDK-8058975 - NoSuchMethodException: j.l.i.MethodHandle.linkToStatic during JVM startup cause fatal error: ExceptionMark destructor expects no pending exceptions
Relates :  
JDK-8058092 - Test vm/mlvm/meth/stress/compiler/deoptimize. Assert in src/share/vm/classfile/systemDictionary.cpp: MH intrinsic invariant
Relates :  
JDK-8157956 - OverflowCodeCacheTest.java fails with Out of space in CodeCache for method handle intrinsic
Sub Tasks
JDK-8058899 :  
Put test 'java/lang/instrument/NativeMethodPrefixAgent.java' on ProblemList - Resolved
Description
----------messages:(3/215)----------
command: main -javaagent:NativeMethodPrefixAgent.jar NativeMethodPrefixApp
reason: User specified action: run main/othervm -javaagent:NativeMethodPrefixAgent.jar NativeMethodPrefixApp 
elapsed time (seconds): 0.421
----------System.out:(6/224)----------
Premain
tr1: Retransforming java/lang/Runtime
tr0: Retransforming java/lang/Runtime
Error occurred during initialization of VM
java.lang.NoClassDefFoundError: Could not initialize class java.lang.invoke.MethodHandleNatives

----------System.err:(59/4582)----------
ERROR: Injection failure: java.lang.InternalError: java.lang.NoSuchMethodException: no such method: java.lang.invoke.MethodHandle.linkToStatic(MemberName)Object/invokeStatic
java.lang.InternalError: java.lang.NoSuchMethodException: no such method: java.lang.invoke.MethodHandle.linkToStatic(MemberName)Object/invokeStatic
	at java.lang.invoke.MethodHandleStatics.newInternalError(MethodHandleStatics.java:120)
	at java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(DirectMethodHandle.java:214)
	at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:188)
	at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(DirectMethodHandle.java:177)
	at java.lang.invoke.DirectMethodHandle.make(DirectMethodHandle.java:84)
	at java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(MethodHandles.java:1655)
	at java.lang.invoke.MethodHandles$Lookup.getDirectMethod(MethodHandles.java:1600)
	at java.lang.invoke.MethodHandles$Lookup.findStatic(MethodHandles.java:777)
	at java.lang.invoke.MethodHandleImpl.findCollector(MethodHandleImpl.java:1223)
	at java.lang.invoke.MethodHandleImpl.makeArrays(MethodHandleImpl.java:1263)
	at java.lang.invoke.MethodHandleImpl.<clinit>(MethodHandleImpl.java:1271)
	at java.lang.invoke.MethodHandleNatives.<clinit>(MethodHandleNatives.java:80)
	at NativeMethodPrefixAgent$Tr.transform(NativeMethodPrefixAgent.java:66)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
	at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
	at NativeMethodPrefixAgent.premain(NativeMethodPrefixAgent.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Caused by: java.lang.NoSuchMethodException: no such method: java.lang.invoke.MethodHandle.linkToStatic(MemberName)Object/invokeStatic
	at java.lang.invoke.MemberName.makeAccessException(MemberName.java:873)
	at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:990)
	at java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(DirectMethodHandle.java:212)
	... 22 more
Caused by: java.lang.NoSuchMethodError: java.lang.invoke.MethodHandle.linkToStatic(Ljava/lang/invoke/MemberName;)Ljava/lang/Object;
	at java.lang.invoke.MethodHandleNatives.resolve(Native Method)
	at java.lang.invoke.MemberName$Factory.resolve(MemberName.java:962)
	at java.lang.invoke.MemberName$Factory.resolveOrFail(MemberName.java:987)
	... 23 more
Caused by: java.lang.VirtualMachineError: out of space in CodeCache for method handle intrinsic
	... 26 more
Returning bad class file, to cause test failure
ERROR: Injection failure: java.lang.ArrayIndexOutOfBoundsException: 6
java.lang.ArrayIndexOutOfBoundsException: 6
	at jdk.internal.org.objectweb.asm.ClassReader.readShort(ClassReader.java:2355)
	at jdk.internal.org.objectweb.asm.ClassReader.<init>(ClassReader.java:198)
	at jdk.internal.org.objectweb.asm.ClassReader.<init>(ClassReader.java:182)
	at asmlib.Instrumentor.<init>(Instrumentor.java:99)
	at asmlib.Instrumentor.instrFor(Instrumentor.java:89)
	at NativeMethodPrefixAgent$Tr.transform(NativeMethodPrefixAgent.java:66)
	at sun.instrument.TransformerManager.transform(TransformerManager.java:188)
	at sun.instrument.InstrumentationImpl.transform(InstrumentationImpl.java:428)
	at sun.instrument.InstrumentationImpl.retransformClasses0(Native Method)
	at sun.instrument.InstrumentationImpl.retransformClasses(InstrumentationImpl.java:144)
	at NativeMethodPrefixAgent.premain(NativeMethodPrefixAgent.java:130)
	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
	at java.lang.reflect.Method.invoke(Method.java:498)
	at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(InstrumentationImpl.java:386)
	at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(InstrumentationImpl.java:401)
Returning bad class file, to cause test failure
Comments
So, the LF caching tests are a different kind of failure - they truly exhaust the code cache. I don't think we can do anything about that given the current design (at least not short term). The segfault with LFMultiThreadCachingTest is a bit unsettling, there is no hs_err file, and it doesn't seem to reproduce for me.
26-09-2014
We have to fix this one ASAP.
26-09-2014
My last comment in the issue is: When manually trying to run the plugin process, I get the following: @@@@@@ in FXUIDriver classPath=C:\Program Files (x86)\Java\jre1.8.0_40\classes;C:/Users/ddehaven/Jemmy/unittestdriver.jar @@@@@@ going to start fx platform java.lang.reflect.InvocationTargetException at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source) at java.lang.reflect.Method.invoke(Unknown Source) at sun.instrument.InstrumentationImpl.loadClassAndStartAgent(Unknown Source) at sun.instrument.InstrumentationImpl.loadClassAndCallPremain(Unknown Source) Caused by: java.lang.InternalError: java.lang.NoSuchMethodException: no such method: java.lang.invoke.MethodHandle.linkToStatic(MemberName)Object/invokeStatic at java.lang.invoke.MethodHandleStatics.newInternalError(Unknown Source) at java.lang.invoke.DirectMethodHandle.makePreparedLambdaForm(Unknown Source) at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Unknown Source) at java.lang.invoke.DirectMethodHandle.preparedLambdaForm(Unknown Source) at java.lang.invoke.DirectMethodHandle.make(Unknown Source) at java.lang.invoke.MethodHandles$Lookup.getDirectMethodCommon(Unknown Source) at java.lang.invoke.MethodHandles$Lookup.getDirectMethod(Unknown Source) at java.lang.invoke.MethodHandles$Lookup.findStatic(Unknown Source) at java.lang.invoke.MethodHandleImpl.findCollector(Unknown Source) at java.lang.invoke.MethodHandleImpl.makeArrays(Unknown Source) at java.lang.invoke.MethodHandleImpl.<clinit>(Unknown Source) at java.lang.invoke.MethodHandleNatives.<clinit>(Unknown Source) at com.sun.javafx.application.PlatformImpl.startup(Unknown Source) at deploy.sqe.drivers.FXUIDriver.premain(FXUIDriver.java:65) ... 6 more Caused by: java.lang.NoSuchMethodException: no such method: java.lang.invoke.MethodHandle.linkToStatic(MemberName)Object/invokeStatic at java.lang.invoke.MemberName.makeAccessException(Unknown Source) at java.lang.invoke.MemberName$Factory.resolveOrFail(Unknown Source) ... 19 more Caused by: java.lang.NoSuchMethodError: java.lang.invoke.MethodHandle.linkToStatic(Ljava/lang/invoke/MemberName;)Ljava/lang/Object; at java.lang.invoke.MethodHandleNatives.resolve(Native Method) at java.lang.invoke.MemberName$Factory.resolve(Unknown Source) ... 20 more Caused by: java.lang.VirtualMachineError: out of space in CodeCache for method handle intrinsic ... 22 more FATAL ERROR in native method: processing of -javaagent failed Exception in thread "main" This happens with 8u40 b07.
26-09-2014
That link just gives me a "Permission Violation".
26-09-2014
This is FX issue - https://javafx-jira.kenai.com/browse/RT-38794
26-09-2014
What and where is RT-38794 ?
26-09-2014
Based on the Dmitry's comment above I raised the priority back to P1
26-09-2014
this issue might be a reason of RT-38794, RT-38794 is a blocker for deployment SQE.
26-09-2014
Sounds like a plan. Another approach to fixing it might be to allowing the intrinsics to be without compiled forms during the startup of the VM, since there's not compiled code. Once we are able to generate compiled code we would generate compiled forms as we go. The invariant that we should never try to call the interpreted for of the intrinsic via a c2i adapter should hold.
22-09-2014
Not a real regression - just a manifestation of an older problem which was masked by a different error previously. Lowering priority to P2
22-09-2014
Yes, that sounds good. Leave this bug open and lower it to a P2.
22-09-2014
Ok, so putting this test on ProblemList.txt is in order now, right?
22-09-2014
The reasons the failures are visible are my fixes JDK-8058092 and JDK-8056154. Before, I suppose, using lambdas in an agent worked just by accident (we just skipped generating a compiled form of the MH intrinsic), later on if we call it from a compiled method we would crash. I think the real fix would be to be able to generate compiled MH intrinsics before the agent is run.
22-09-2014
We've already encountered a few problems with trying to use lambdas "too early" in the VM initialization process. This may just be another case. But the lambdas were introduced in 8037082 and didn't cause immediate failure so the problem has been introduced elsewhere - perhaps a change to how the lambdas are processed?
22-09-2014
Is there any reason why we can't use a lambda expression at that place? It would be good to understand the real reason and fix that problem. However, if that will take time we may have to at least exclude this test until the real fix.
22-09-2014
Actually, the failure is located in the agent code itself and not in the generated bytecode. It seems to be related to the usage of lambdas in the ClassFileTransformer::transform(byte[]) method. A quick fix might be using an anonymous inner class instead of the lambda expression.
22-09-2014
JDK-8037082 was fixed for b29 but the failures are observed first in b31. However, I will investigate the generated bytecode.
22-09-2014
A probable cause is the fix for JDK-8037082 which changed how the test does byte code instrumentation.
22-09-2014
There are always hidden dependencies. The initialization sequence is very fragile. Has something changed in this area to cause the failures - new test? Larger test?
22-09-2014
Perhaps this is happening because agents are run before compilers are initialized. Can the compiler initialization be moved up? Any hidden dependencies?
22-09-2014
ILW=VM Error, medium, none=HMH=P1
17-09-2014
How can you tell the size of the code cache? The test itself does not set anything as far as I can tell.
17-09-2014
It's running with 4M of code cache, this is not going to work.
17-09-2014
Test is failing in jdk9/dev CI system.
17-09-2014
java.lang.VirtualMachineError: out of space in CodeCache for method handle intrinsic ... 26 more Looks like some limitation in the compiler?
17-09-2014