JDK-8157807 : JNI CreateJavaVM call exits without proper return code when it fails to allocate max heap size
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 8u91,9
  • Priority: P3
  • Status: Closed
  • Resolution: Won't Fix
  • OS: windows_7
  • CPU: x86
  • Submitted: 2016-05-16
  • Updated: 2020-12-02
  • Resolved: 2020-12-02
Related Reports
Duplicate :  
JDK-6859079 - JNI_CreateJavaVM() terminates the process on error condition instead of returning error code
Relates :  
JDK-8166250 - fails on running java or even java -version
Relates :  
JDK-4898900 - proper handling of HeapCreate failure through invocation interface
Description
FULL PRODUCT VERSION :
Java version "1.8.0_25" (x86)  onwards. Also reprodeuced with latest release of "1.8.0_91" (x86).


FULL OS VERSION :
Windows 7 x64 and x86

A DESCRIPTION OF THE PROBLEM :
If a create a JVM through the invocation interface using
JVM.DLL -> JNI_CreateJavaVM and the -Xmx parameter
specifies a heap larger than the VM can allocate, the VM
prints a message to stderr and *completely exits the
process*.

I am using the invocation interface to create a VM in-
process of another application. The application is not
JAVA.EXE but a GUI application. The message to stderr is
never seen, and the app disappears w/ no error or acces
violation messages of any kind.

As per JNI (invocation interface) documentation the CreateJavaVM call should return an error code of -4. Which is the behavior wit Java version 1.7.0_705 (x86).

From your bug database I found a similar issue reported against Java 1.4 and was fixed for Java 1.5. So it seems to have been re-introduced in Java 1.8. Here is the link to that Bug report : https://bugs.openjdk.java.net/browse/JDK-4898900

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Did not try

REGRESSION.  Last worked in version 7u76

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Create a GUI app that creates its own VM via the
invocation interface and passes a very large value as an -
Xmx parameter (like 1800M).
2. Watch the app disappear at runtime w/ no error message.
Running from the command line also results in no error output since the app is not a console app. The only way to capture the error message is to redirect the app's output
to a file (either from the command line or programmatically) and view the output file after the app disappears.

In Java 1.7 createJavaVM() would return with an error code of -4 which would allow us to reset the Xmx value to 256 and get the application started. 


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED VERSUS ACTUAL BEHAVIOR :
I would expect the JNI_CreateJavaVM to return a failure code of -4 as documented in JNI error codes  and allow my application to continue, handling the error gracefully if possible (for instance, if the VM is only needed for one feature that can be disabled at runtime
if the VM
REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Not found any.
Comments
Similar issues linked here were closed as WNF, doing the same for this issue for the same reasons.
02-12-2020
GC team can handle however they see fit of course but note this is just the most recent reporting of a very long standing, non-trivial, problem - see JDK-6859079. Arguably this could be closed as a dup of JDK-6859079. Though in that case JDK-6859079 should be moved to GC subcomponent.
01-08-2016
It's unclear if this bug was deferred to 10 on purpose. Setting it to 9 until properly deferred.
01-08-2016
Reset to "new" so GC team can triage again.
28-07-2016
There are multiple levels of problem here as I suspected - some design issues with regards to failure reporting, others outright bugs. First failure I encountered was running G1: V [libjvm.so+0xa94258] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd8 V [libjvm.so+0x135fa71] os::pd_commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0x111 V [libjvm.so+0x13568fd] os::commit_memory_or_exit(char*, unsigned long, unsigned long, bool, char const*)+0x1d V [libjvm.so+0xc87f58] G1PageBasedVirtualSpace::commit_preferred_pages(unsigned long, unsigned long)+0x78 as you can see a deliberate call to pd_commit_memory_or_exit. So tried with SerialGC - which showed: V [libjvm.so+0xa94258] report_vm_out_of_memory(char const*, int, unsigned long, VMErrorType, char const*, ...)+0xd8 V [libjvm.so+0x135f942] os::pd_commit_memory(char*, unsigned long, unsigned long, bool)+0xf2 V [libjvm.so+0x13565df] os::commit_memory(char*, unsigned long, unsigned long, bool)+0x1f V [libjvm.so+0x16adb45] commit_expanded(char*, unsigned long, unsigned long, bool, bool)+0x25 V [libjvm.so+0x16aeac2] VirtualSpace::expand_by(unsigned long, bool)+0x272 V [libjvm.so+0xd299be] Generation::Generation(ReservedSpace, unsigned long)+0x8e This time it called pd_commit_memory - but that still exits! That seems like a bug to me. So added a bypass for that and we still get: Error occurred during initialization of VM Could not reserve enough space for object heap because here: Generation::Generation(ReservedSpace rs, size_t initial_size) : _ref_processor(NULL) { if (!_virtual_space.initialize(rs, initial_size)) { vm_exit_during_initialization("Could not reserve enough space for " "object heap"); } if we fail to initialize then we exit! Worse, this is a constructor so there is no way to return an error code. Fixing this requires major redesign of the API's involved. I will leave this open but target to 10 (which still leaves the option of doing something in a 9-update release).
28-07-2016
I will reopen this and see if the failure point is restricted to one piece of code. If it is then we may be able to fix in 9 and potentially backport to 8. The problem is that a slightly smaller requested heap size might succeed but consume so much memory that some other internal allocation fails which will then abort in the same way. This might lead to a situation where, as an example: -Xmx1986M gives an error code -Xmx1985M silently exits even though it seems to require less memory
23-06-2016
Same basic issue already covered by JDK-6859079
09-06-2016
Unfortunately there are numerous initialization paths in the VM that call vm_exit_during_initialization, or a variant thereof, instead of propagating an error back up to the CreateJavaVM call allowing it to report an error. 4898900 may have been fixed in 5 but I suspect it was broken again shortly after. I counted 140 such paths in existing code. There is no simple fix here and we have long standing open issues to better support the invocation API. Moving to GC team for comment as this is specifically about heap sizing options.
25-05-2016
Forwarding for dev teams evaluation
25-05-2016