have simulated most of this behavior (minus the socket listening)
inside the attached file "invoke3.cc". The objective is to have the
Server talk to a Java application. The way we plan of doing this is to
use the JNI Invocation API and create a JVM from inside the C++ code.
he exact place where this JVM is created (i.e., either before the
memory-mapping step -2-, or after it and before the fork step -4-, or
inside the child process and before the thread creation step -5-, or
after the threads are created) is up to us. We look for the most
efficient and flexible solution.
 
The problem we face is that when we try to create the JVM in the parent
process after the memory-mapping and before the fork, we get signal 11> errors because apparently there is not enough memory for the JVM to
start. If I reduce the size of the memory-mapped file to less than 1 GB
everything works fine and the JVM is created successfully. I have tried
to use the "-Xmx" flag inside the C++ code to reduce the size of the JVM
heap but it did not work. My current environment has about 48 GB of memory.

Attached is said test case:
the Makefile is currently set to my machine with the latest java_g build of 1.5.0 (java_g is used by using jvm_g LIB arg in the makefile)
This also cores with 1.4.2 and 1.4.1
You'll most likely want to change this and then run make again.

Looks initially similar to bugs 4467771 and 4475282, but with this particular problem it appears the VM doesn't appear to start or at least has problems and these bugs are showing fixed in merlin....

You can change VM args in the actual invoke3.cc code.

run make

to run the test case
invoke3 50000
There is a cutoff for the number for the argument in the startup.
48609 for the regular non debug JDK. below that will allow the JVM to start
and the test case to finish.

It will core almost immediately otherwise.  depending how close you get to 48609 it will be earlier in the running or later.

I originally thought that this was a JNI issue as running with the non debug jvm and the flag -XX:ShowMessageBoxOnError it simply cored and the flag did not trap any jvm failure, but with the debug jvm (java_g) the flag caught the failure and the subsequent stack and error message was given:

(this was with 1.5.0 beta 2)

(dbx) where
current thread: t@1
  [1] _read(0x0, 0xffffffff7fffdd80, 0x10, 0x2f, 0x0, 0xff00), at 0xffffffff7cea67e0 
  [2] _ti_read(0x0, 0xffffffff7fffdd80, 0x10, 0xffffffff7fffde78, 0x1a8, 0xf51b), at 0xffffffff7f3108a4 
  [3] os::message_box(0xffffffff7ebf1c07, 0xffffffff7eea5780, 0xffffffff7ebf1b4a, 0xf51b, 0x1, 0xff00), at 0xffffffff7e5c9f0c 
  [4] VMError::show_message_box(0xffffffff7fffe140, 0xffffffff7eea5780, 0x7d0, 0x0, 0x0, 0x0), at 0xffffffff7e91e230 
  [5] VMError::report_and_die(0xffffffff7fffe140, 0x100150d58, 0xffffffff7e9ac6f1, 0xffffffff7e9ac6a7, 0x1a8, 0x0), at 0xffffffff7e91bda8 
  [6] report_assertion_failure(0xffffffff7e9ac6a7, 0x1a8, 0xffffffff7e9ac6f1, 0x0, 0x0, 0x0), at 0xffffffff7df40de4 
  [7] MacroAssembler::call(0x100151438, 0xffffffff7dc8ff08, 0x6, 0x0, 0xffffffff7fffe300, 0xffffffff7fffe45c), at 0xffffffff7dca064c 
  [8] MacroAssembler::get_thread(0x100151438, 0xffffffff7fffe45c, 0x0, 0x1, 0xffffffff7fffe45c, 0x0), at 0xffffffff7dc90120 
  [9] StubGenerator::generate_forward_exception(0xffffffff7fffe7c0, 0x100151338, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e7bff28 
  [10] StubGenerator::generate_initial(0xffffffff7fffe7c0, 0x100151338, 0x4e70, 0x0, 0x0, 0x0), at 0xffffffff7e7bac28 
  [11] StubGenerator::StubGenerator(0xffffffff7fffe7c0, 0x100151338, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e7b9ee8 
  [12] StubGenerator_generate(0x100151338, 0x0, 0x4e20, 0x1, 0x0, 0x0), at 0xffffffff7e7b9b3c 
  [13] StubRoutines::initialize1(0x0, 0x0, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e7cb840 
  [14] stubRoutines_init1(0xffffffff7fffecf8, 0x100150d58, 0x0, 0x0, 0x0, 0x0), at 0xffffffff7e7cbad4 
  [15] init_globals(0xffffffff7fffecf8, 0x100152008, 0x0, 0x1, 0x0, 0x0), at 0xffffffff7e0c936c 
  [16] Threads::create_vm(0xffffffff7fffefd8, 0xffffffff7ee4ca7c, 0x0, 0x100001edc, 0x0, 0x0), at 0xffffffff7e84b860 
  [17] JNI_CreateJavaVM(0xffffffff7fffeff8, 0xffffffff7ffff000, 0xffffffff7fffefd8, 0x80000001, 0x3, 0x0), at 0xffffffff7e1ddb74 
=>[18] debugJVM(), line 302 in "invoke3.cc"
  [19] main(argc = 2, argv = 0xffffffff7ffff1b8), line 90 in "invoke3.cc"


current thread: t@1
current frame:  [18]
g0-g1    0x0000000000000000 0x0000000000000003
g2-g3    0x0000000000000001 0x0000000000000000
g4-g5    0x0000000000000001 0x0000000083159800
g6-g7    0x0000000000000000 0xffffffff7c100000
o0-o1    0xffffffff7fffeff8 0xffffffff7ffff000
o2-o3    0xffffffff7fffefd8 0x0000000080000001
o4-o5    0x0000000000000003 0x0000000000000000
o6-o7    0xffffffff7fffe6b1 0x0000000100001edc
l0-l1    0xffffffff7fffefd8 0xffffffff7fffeff8
l2-l3    0xffffffff7ffff000 0x0000000000000001
l4-l5    0x0000000000008104 0x0000000000008104
l6-l7    0x000000000000000a 0x000000000000000a
i0-i1    0x0000000000000000 0xfffffffffffd0260
i2-i3    0x0000000000000000 0xffffffff7ffff40d
i4-i5    0xffffffff7ffff408 0xffffffff7cfbd051
i6-i7    0xffffffff7fffe811 0x00000001000013cc
y        0x0000000000000000
ccr      0x0000000000000044
pc       0x0000000100001edc:debugJVM+0xd4       call    JNI_CreateJavaVM [PLT]
npc      0xffffffff7cea67e4:_read+0xc   bcc,pt %icc,_read+0x28


sr1-unsh01-04{msusko}65: more hs_err_pid62747.log
#
# An unexpected error has been detected by HotSpot Virtual Machine:
#
#  Internal Error (/BUILD_AREA2/jdk1.5.0/hotspot/src/cpu/sparc/vm/assembler_sparc.inline.hpp, 424 [ Unknown ]), pid=62747
, tid=1
#
# Java VM: Java HotSpot(TM) 64-Bit Server VM (1.5.0-beta2-b41-debug mixed mode)
#
# Error: assert(labs(intptr_t(d) - intptr_t(pc())) <= (intptr_t)((1 << 31 ) - 1),"target address not reachable with call 
instruction")

---------------  T H R E A D  ---------------

Current thread (0x0000000100150d58):  JavaThread  [_thread_in_vm, id=1]

Stack: [0xffffffff7ff00000,0xffffffff80000000)
[error occurred during error reporting, step 110, id 0xa]


[error occurred during error reporting, step 120, id 0xa]


---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )

Other Threads:

VM state:not at safepoint (not fully initilizated)

VM Mutex/Monitor currently owned by a thread: None

Heap


interesting with just java the stack is different but the regs shows a bad address

t@1 (l@1) terminated by signal SEGV (no mapping at the fault address)
0xffffffff7eb3aa9c: JVM_handle_solaris_signal+0x0554:   ld      [%l3], %g1
Current function is debugJVM
  302           jint res = JNI_CreateJavaVM(&jvm, (void**)&env, &vm_args);
(dbx) where
current thread: t@1
  [1] JVM_handle_solaris_signal(0xb, 0xc438, 0xffffffff7fffda40, 0x10012bfd0, 0xffffffff7fffdd20, 0xffffffff7efcf780), at 0xffffffff7eb3aa9c 
  [2] __sighndlr(0xb, 0xffffffff7fffdd20, 0xffffffff7fffda40, 0xffffffff7eb3a410, 0x0, 0x0), at 0xffffffff7f318ae4 
  [3] call_user_handler(0xffffffff7cf00000, 0xffffffff7fffdd20, 0xffffffff7fffda40, 0x0, 0x0, 0x0), at 0xffffffff7f312718 
  [4] sigacthandler(0xb, 0xffffffff7fffdd20, 0xffffffff7fffda40, 0xffffffff7dc9ef50, 0xffffffff7f0f4000, 0x0), at 0xffffffff7f3128dc 
  ---- called from signal handler with signal 11 (SIGSEGV) ------
  [5] 0xfffffffe7ea3b1c0(0x10012bfd0, 0xb8, 0xffffffff7fffdfb0, 0xffffffff7dc9ef50, 0xffffffff7f0f4000, 0x0), at 0xfffffffe7ea3b1bf 
  [6] 0xfffffffefd417024(0x0, 0xffffffff7f1853b8, 0x0, 0x2000, 0xe6f8, 0xffffffff7fffd701), at 0xfffffffefd417023 
  [7] 0xfffffffefd40013c(0xffffffff7fffe0d8, 0xffffffff7fffe2f0, 0xa, 0xfffffffef6407e58, 0xfffffffefd40b400, 0xffffffff7fffe300), at 0xfffffffefd40013b 
  [8] JavaCalls::call_helper(0x0, 0x10012bfd0, 0xa, 0x1002138b0, 0xffffffff7fffe0f8, 0xffffffff7f0f4000), at 0xffffffff7ea6ba1c 
  [9] JavaCalls::call(0xffffffff7fffe2e8, 0x1002138b0, 0xffffffff7fffe2f8, 0x10012bfd0, 0x676140, 0xffffffff7ea15ae4), at 0xffffffff7ea7dee8 
  [10] instanceKlass::call_class_initializer_impl(0xffffffff7fffe300, 0x10012bfd0, 0xfffffffef6407e58, 0x63264c, 0x1002138b0, 0xdc00), at 0xffffffff7eac1aa8 
  [11] instanceKlass::initialize_impl(0x7, 0x10012bfd0, 0xffffffff7cd004c8, 0x100104a70, 0x0, 0xfffffffef6407ec0), at 0xffffffff7ea511d4 
  [12] instanceKlass::initialize(0x1002138a8, 0xffffffff7ea2f39c, 0x3, 0xffffffff7f1783c8, 0xffffffff7f1783c8, 0x0), at 0xffffffff7ea39048 
  [13] instanceKlass::initialize_impl(0xfffffffef6407ed0, 0x10012bfd0, 0x1001aa110, 0xfffffffef6410610, 0x0, 0xffffffff7fffe9b8), at 0xffffffff7ea50efc 
  [14] instanceKlass::initialize(0x1001aa118, 0xffffffff7ea2f39c, 0x10012bfd0, 0x1, 0xffffffff7f1783c8, 0x5b0df8), at 0xffffffff7ea39048 
  [15] Threads::create_vm(0xffffffff7f185520, 0x10012bfd0, 0x10012bfd0, 0x0, 0x11c00, 0xffffffff7f19d980), at 0xffffffff7ec547d4 
  [16] JNI_CreateJavaVM(0xffffffff7f17a484, 0x10400, 0xffffffff7fffefd8, 0x4995d8, 0xffffffff7ffff000, 0xffffffff7fffeff8), at 0xffffffff7ec5aadc 
=>[17] debugJVM(), line 302 in "invoke3.cc"
  [18] main(argc = 2, argv = 0xffffffff7ffff1b8), line 90 in "invoke3.cc"
(dbx) regs
current thread: t@1
current frame:  [17]
g0-g1    0x0000000000000000 0xffffffffffffff20
g2-g3    0x000000010012bfd0 0xfffffffef64988a8
g4-g5    0xffffffff7fffde20 0xfffffffef6407e58
g6-g7    0x0000000000000000 0xffffffff7cf00000
o0-o1    0xffffffff7f17a484 0x0000000000010400
o2-o3    0xffffffff7fffefd8 0x00000000004995d8
o4-o5    0xffffffff7ffff000 0xffffffff7fffeff8
o6-o7    0xffffffff7fffe6b1 0x0000000100001edc
l0-l1    0xffffffff7fffefd8 0xffffffff7fffeff8
l2-l3    0xffffffff7ffff000 0x0000000000000001
l4-l5    0x0000000000008104 0x0000000000008104
l6-l7    0x000000000000000a 0x000000000000000a
i0-i1    0x0000000000000000 0xfffffffffffd0260
i2-i3    0x0000000000000000 0xffffffff7ffff40d
i4-i5    0xffffffff7ffff408 0xffffffff7ddbd051
i6-i7    0xffffffff7fffe811 0x00000001000013cc
y        0x0000000000000000
ccr      0x0000000000000099
pc       0x0000000100001edc:debugJVM+0xd4       call    JNI_CreateJavaVM [PLT]
npc      0xfffffffe7ea3b1c4:0xfffffffe7ea3b1c4  <bad address 0xfffffffe7ea3b1c4>
(dbx) 

the npc shows a bad address. 

Not sure the interaction of the 64 bit nondebug vs debug JVM has on this...