When a customer tries to invoke their application server with debug options
in RHEL3 Update4 with Exec-Shield "on", core dump appears or invocation
terminates with the following message.
 "FATAL ERROR in native method: JDWP "threadControl.c" (Apr  3 2004),
  line 1551: Unable to create thread table entry"

CONFIGURATION:
 OS : RHEL3 Update4 for x86
 JDK : 1.4.2
 JDK options : -Xrunjdwp:transport=dt_socket,server=y,address=8788,suspend=n

Unfortunately, they does not send any test case, but they reported the problem
in memory management in jvm with debug options.


INVESTIGATION :
 There seems 2 problems in the way of memory management in jdwp.

 (1) jdwp should improve memory management.

     When Exec-Shield is "on", there are some cases that the maximum
     memory(heap) size of which sbrk can get is very small.
    
     When debug options are specified, which is, jdwp is working, jdwp gets
     heap area by using sbrk.
     The reason why the sbrk area becomes small is as follows.

     When Exec-Shield is "on", OS(linux) keeps the executable area
     within some reasonable size.
     (Please see the attached file(Memorylayout-en.pdf) showing memory layout.)
     JVM gets Java heap by mmap() with  executable attribute.
     Then Java heap is allocated just under text area.
     That reduces the space where sbrk can use.
    
     The problem is not that Java heap is allocated with executable attribute,
     but jdwp uses only sbrk to get memory area, which is, problem seems
     the way of memory management in jdwp.

     malloc in glibc seems to use mmap() also in case sbrk fails to get memory area.
     Jdwp should improve memory management, like malloc in glibc does.
  
     If no improvement, every time new RHEL version is released, similar problems
     will occur.


 (2) jdwp forces to execute program even on sbrk failure

     core dump sometimes occurs with the following trace information.

(gdb) bt
>>#0 0x00365731 in pthread_mutex_init () from /lib/tls/libpthread.so.0
>>#1 0x00c6bb7c in ObjectMonitor::ObjectMonitor () from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
>>#2 0x00c1c894 in CreateRawMonitor () from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
>>#3 0x002fd872 in JVM_OnLoad () from /opt/app_servers/jdk/jre/lib/i386/libjdwp.so
>>#4 0x00c10601 in JvmdiInternal::post_event () from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
>>#5 0x00c1feea in jvmdi::post_vm_initialized_event () from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
>>#6 0x00cc8aa8 in Threads::create_vm ()from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
>>#7 0x00baca38 in JNI_CreateJavaVM () from /opt/app_servers/jdk/jre/lib/i386/client/libjvm.so
         
     The scenario which the above message shows is,
        - sbrk fails to get memory
        - jdwp forces to execute the program
        - null is passed to pthread_mutex_init
        - core dump occurs

NOTE:
  The customer also inform us that RedHat does not recommend to use sbrk() 
  because of legacy interface.
  They request the jdk versions which support RHEL 3 (and subsequent) should not
  use that interface any longer.
  

###@###.### 2005-05-10 02:20:36 GMT