JNI_CreateJavaVM() terminates the process after having encountered an 
error condition (e.g.: not enough memory), instead of returning an
appropriate error code (e.g.: -4)

from jni.h:
/*
 * possible return values for JNI functions.
 */

#define JNI_OK           0                 /* success */
#define JNI_ERR          (-1)              /* unknown error */
#define JNI_EDETACHED    (-2)              /* thread detached from the VM */
#define JNI_EVERSION     (-3)              /* JNI version error */
#define JNI_ENOMEM       (-4)              /* not enough memory */
#define JNI_EEXIST       (-5)              /* VM already created */
#define JNI_EINVAL       (-6)              /* invalid arguments */


The behavior is strictly reproducible and can be observed with
JDK 1.2.2, 1.3.1, 1.4.2, 1.5 and 1.6.

Here are 3 examples of the behavior:

1. Successful execution
-----------------------
JNI_CreateJavaVM() was successfully created: JNI_OK is returned:
Java program "HelloWorld" is executed.
Everthing is fine

% JNITest
JNI_CreateJavaVM returned 0
Hello World
%

2. Error condition encountered
------------------------------
JNI_CreateJavaVM() could not be created because of invalid memory
requirement: JNI_EINVAL is returned, and can be handled by the 
calling C program.
Everything is fine.

% JNITest
Invalid initial heap size: -Xms4096m
The specified size exceeds the maximum representable size.
JNI_CreateJavaVM returned -6
%

3. Error condition on insufficient memory
-----------------------------------------
JNI_CreateJavaVM() could not be created because of insufficient 
memory available: no error code is returned; the process is 
immediately terminated.
This is what the complaint is about.

% ./JNITest
Error occurred during initialization of VM
Could not reserve enough space for object heap
%


Here is what the docs say
-------------------------
 According to the docs, JNI_CreateJavaVM() "returns a negative number on failure."

 http://java.sun.com/j2se/1.4.2/docs/guide/jni/spec/invocation.html#wp16334 
 --------------------------------------------------------------------
 JNI_CreateJavaVM

 jint JNI_CreateJavaVM(JavaVM **p_vm, JNIEnv **p_env, void *vm_args);

 Loads and initializes a Java VM. The current thread becomes the main
 thread. Sets the env argument to the JNI interface pointer of the main
 thread.

 JDK 1.1 does not support creating more than one VM in a single process.
 The version field in vm_args must2 be set to 0x00010001.

 PARAMETERS:
[ ... ]

 RETURNS:
 Returns "0" on success; returns a negative number on failure.
 --------------------------------------------------------------------