JDK-6269555 : creating JVM via C program "steals" space from main thread stack rendering JNI useless
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 5.0u2
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: solaris_8
  • CPU: generic
  • Submitted: 2005-05-12
  • Updated: 2016-12-02
  • Resolved: 2005-07-13
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
1.4.2_10Fixed 6 b43Fixed
Related Reports
Relates :  
Relates :  
Problem:  Initializing a Java VM "steals" stack space from other
threads.  It basically renders JNI unusable in any non-trivial way.  The
problem occurs on Solaris 8/10 on either sparc/x86.  A test program is
included to demonstrate the problem.

Java Version:

>> java -version java version "1.5.0_02" Java(TM) 2 Runtime Environment,

Standard Edition (build 1.5.0_02-b09) Java HotSpot(TM) Server VM (build
1.5.0_02-b09, mixed mode)

This program demonstrates that initializing a Java VM "steals"
  * stack space from other threads.
  * For instance, in the sample output below, before initializing the
  * JVM, the application has used up 1.8M of stack space.  But after
  * initializing the JVM, the main thread's allocated stack space has
  * shrunk to just the single 488K block at 0xFFB76000!  That means
  * that the main thread can never again exceed 488K of stack, or else
  * the program will crash.

  Stack starts near 0xffbeef10

  Initial stack:
  FF3A0000    176K read/exec         /usr/lib/ld.so.1
  FF3DC000      8K read/write/exec   /usr/lib/ld.so.1
  FF3DE000      8K read/write/exec   /usr/lib/ld.so.1
  FFBEC000     16K read/write/exec     [ stack ]
   total     7352K

  recurse() stack near 0xffa1a2ec

  Stack after recurse(20000):
  FF3A0000    176K read/exec         /usr/lib/ld.so.1
  FF3DC000      8K read/write/exec   /usr/lib/ld.so.1
  FF3DE000      8K read/write/exec   /usr/lib/ld.so.1
  FFA18000   1888K read/write/exec     [ stack ]
   total     9224K

   Stack after jvm:
   FF3A0000    176K read/exec         /usr/lib/ld.so.1
   FF3DC000      8K read/write/exec   /usr/lib/ld.so.1
   FF3DE000      8K read/write/exec   /usr/lib/ld.so.1
   FFA18000   1376K read/write/exec     [ stack ]
   FFB70000     24K -                   [ stack ]
   FFB76000    488K read/write/exec     [ stack ]
    total    82304K

To compile:

solaris sparc:
   cc -g -I/usr/local/java/jdk/include
-L/usr/local/java/jdk/jre/lib/sparc -R/usr/local/java/jdk/jre/lib/sparc
-ljvm -lpthread stack.c

solaris x86:
   cc -g -I/usr/local/java/jdk/include
-I/usr/local/java/jdk/include/solaris -L/usr/local/java/jdk/jre/lib/i386
-R/usr/local/java/jdk/jre/lib/i386 -ljvm -lpthread stack.c

   gcc -g -I/usr/local/java/jdk/include
-Wl,-rpath,/usr/local/java/jdk/jre/lib/i386/server -ljvm -lpthread stack.c

#include <stdio.h>
#include <stdlib.h>
#include <unistd.h>
#include <jni.h>

void recurse(int n)
     if (n > 0) {
     } else {
  printf("recurse() stack near 0x%x\n", &n);

void pmap()
     /* check if we have gnu grep installed */
     int ret = system("echo hi | gnu grep -1 hi > /dev/null 2>&1");
     char buf[BUFSIZ];
     if (ret == 0) {
  snprintf(buf, sizeof(buf), "pmap %d | gnu grep -3 stack", getpid());
     } else {
  snprintf(buf, sizeof(buf), "pmap %d | grep stack", getpid());

void jvm()
     JavaVMInitArgs vm_args;
     vm_args.version = JNI_VERSION_1_4;
     vm_args.nOptions = 0;
     JavaVM *jvm;
     JNIEnv *env;
     long result = JNI_CreateJavaVM(&jvm, (void **)&env, &vm_args);
     if (result == JNI_ERR) {
  fprintf(stderr, "ERROR initializing the JVM\n");

int main(int argc, char **argv)
     int x;
     printf("Stack starts near 0x%x\n", &x);
     printf("\nInitial stack:\n");

     int n = 20000;
#ifdef __linux
     /* linux appears to give you a larger initial stack */
     n = 100000;
     if (argc >= 2) {
  n = atoi(argv[1]);

     printf("\nStack after recurse(%d):\n", n);


     printf("\nStack after jvm:\n");

     printf("\nPrepare to crash:\n");

     printf("\nWOOHOO!  No crash!\n");
###@###.### 2005-05-12 19:57:37 GMT

Here's the bug eval: ------------------------------ From hui@xxx: When VM is started we limit primordial thread's stack size to be the same as other Java threads by setting up a guard page. I recall there used to be a bug long time ago about recursive functions behaving differently in main/primordial thread and in other thread (runs ok in primordial thread, but stack overflow if called from other threads), I suspect it might be the reason to limit primoridal thread's stack size. BTW, I believe the right behavior is to treat standard launcher and native application differently. That is, if JVM is created by one of our standard launcher, then limit stack size because we know there are few frames on stack and the thread will run java code; if JVM is created by native application, then don't limit stack size, treat main thread the same way as we would for other native threads, because there might be many native frames already on stack and it's likely the thread will run java code only occasionally. Another note, to get around the problem, you could port Linux behavior. On Linux we limit stack size only when -Xss is specified. I'm going to add a private property -Dsun.java.launcher=java to our launcher code so VM would know if it's created by standard launcher or not. After that it would be possible to implement the logic as I mentioned in the previous email. ------------------------------ Will wait for the launcher fix to provide a fix for this bug as suggested above. stephen@xxx 2005-06-01 13:56:32 GMT The bug will be addressed as follows: 1. Solaris - If CreateJavaVM is called on the primordial thread by a user application which embeds the VM (and not by the Java launcher), instead of using ThreadStackSize we will set the stack size to the user's ulimit stacksize up to a maximum of 8MB. We cannot accommodate an unlimited stack size due to the OS reporting a stack size of 2GB in this case, which prevents us from setting up guard pages. 8MB is a reasonably large value that should be sufficient. Note that even if ThreadStackSize is explicitly set via -Xss we will ignore this setting and use the ulimit setting for the primordial thread IFF the VM was not created by a Java launcher. The reasoning being, for an application that embeds the Java VM, the primordial thread is more a native thread than a Java thread, and we should treat it as such. A new property, sun.java.launcher, will be used to detect whether the VM is running under the launcher or is embedded by a native user application. 2. Linux - no change. The Linux VM already behaves like (1), except that the maximum primordial thread stack size value is 2MB (4MB for 64-bit VM). Given the limitations of accurately getting primordial thread stack information from the OS: In the future we should consider modifying the Java launcher to not create the VM in the primordial thread as described in the evaluation of 5099186. Even better would be to encourage developers of native applications embedding the VM to do the same.

WORK AROUND Use -Xss<stacksize> set to the same value as "limit" reports. e.g. $ limit cputime unlimited filesize unlimited datasize unlimited stacksize 8192 kbytes coredumpsize unlimited vmemoryuse unlimited descriptors 1024 Pass "-Xss8192k" as a member of JavaVMOption array to JNI_CreateJavaVM. If stacksize is unlimited, just set -Xss to an appropriately large value. NOTE that this setting will be used for stacksize for ALL Java threads so it may not be appropriate depending on the application. But it should avoid the crash in the testcase on Solaris. Linux only: Initial thread has a hard limit of 2m (4m for 64-bit VM) due to limitations of older glibc/kernel combination. Due to this, the above workaround will be less effective but may still give some relief. ###@###.### 2005-06-02 19:21:32 GMT This workaround isn't working on my linux-suse-9.2. -Xss set stack size for all threads except main process thread. Maximum size of stack for main thread is 8mb on linux. Running jvm reduce this value to 2mb. ###@###.### 2005-06-03 07:53:22 GMT Another possible work around is to avoid calling JNI_CreateJavaVM from the primordial thread. It is easier for the VM to detect the stack size of non-primordial thread and should avoid creating guard pages in the middle of the stack. ###@###.### 2005-06-06 19:32:48 GMT

EVALUATION See comments. ###@###.### 2005-06-01 13:58:03 GMT