JDK-5099186 : Linux: JVM does not work well with exec-shield
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 5.0,6
  • Priority: P3
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: generic,x86
  • Submitted: 2004-09-08
  • Updated: 2005-08-29
  • Resolved: 2005-08-29
Related Reports
Duplicate :  
JDK-6316197 - Java launcher should create JVM from non-primordial thread
Duplicate :  
JDK-6299862 - Apparent intermitent VM Crash during startup.
Relates :  
JDK-6267724 - Crash occurs with debug options, jdwp needs to improve memory management way
Description
A recently introduced security feature (exec-shield) may cause JVM to throw
StackOverflowException or even die on a crash. The problem is that the
exec-shield patch randomizes the starting stack pointer so much that the
main thread is started below (or near) its stack limit.

Please see Redhat bugzilla:
   https://bugzilla.redhat.com/bugzilla/show_bug.cgi?id=126297

This should not affect pure Java application, as the default Java launcher
does not have PT_GNU_STACK, so exec-shield should be disabled by default.
However, native application using JNI_CreateJavaVM to invoke JVM might
run into troubles on RHEL 3 or Fedora Core 2.

Anyway, if you are seeing this problem, you could get around the issue 
by disabling exec-shield-randomize, e.g. (run as root):
  sysctl -w kernel.exec-shield-randomize=0
or,
  echo "0" > /proc/sys/kernel/exec-shield-randomize
Comments
EVALUATION VM needs to know the precise location of thread stacks in order to correctly handle StackOverflowError. With stack randomization there is no way to reliably detect initial thread's stack location and stack size. All we can do is to detect OS version and guess the stack bottom. That probably can work around the current problem, but it likely will backfire in the future. Rather than further complicating the VM code that deals with initial thread, I'm fixing this in mustang under bugid 6316197. The idea is to let Java launcher create a new thread and call JNI_CreateJavaVM from the newly created thread. VM will never see primordial thread as long as it's invoked by standard launcher. If you invoke JVM with your own launcher, it's recommended to create JVM from non-primordial thread too due to this problem and several other issues we've seen with primordial thread. I'm closing this bug as a duplicate of 6316197, as the change for 6316197 will fix this problem automatically.
29-08-2005
EVALUATION See comments. Post-tiger work. One possible fix is to detect VM is created by default Java launcher and skip initial thread compiletely. ###@###.### 2004-09-20
20-09-2004