VM crash bug reported by one of CAP members. Their tests are not 
very portable, it would take a lot of work to set up and run. 
Error message(hs_err_pid26796.log) and core files were attached instead.
They are willing to work with someone in HotSpot team to narrow it down
and produce a case outside their product code if possible.

----------------------------------------------------------------------------------
J2SE Version (please include all output from java -version flag):

java version "1.4.0-beta"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-beta-b65)
Java HotSpot(TM) Client VM (build 1.4.0-beta-b65, mixed mode)

Does this problem occur on J2SE 1.3?  Yes / No (pick one)

No

Operating System Configuration Information (be specific):

Solaris 7 with the latest patches.

Hardware Configuration Information (be specific):

This was run on a Sun 420R quad processor with 1 GB RAM

Bug Description:
When using more than one thread in a section of code it is possible 
(it seems) for HotSpot to zero out the instructions which one of the 
two (or more) threads is currently executing (or will execute in the 
near future).  This appears to happen after executing the code several 
hundred times, as though HotSpot is coming back and reoptimizing the 
code segment by zeroing the old instructions out and then letting another 
thread kill the JVM because the SPARC processor cannot execute the 
instruction 0x0.


Detail problem description from customer:
+++++++++++++++++++++++++++++++++++++++++

JVM was configured with:
Tests 1 - 7: -server -Xms128m -Xmx512m
Tests 8:     -server -Xms512m -Xmx512m
Tests 9:     -server -Xms512m -Xmx512m

Tests:
Test 1:
Running any test against MapXtreme Java causes the JVM to crash.  
Every crash is caused specifically by Solaris delivering a SIGILL 
to one of the threads running inside of the JVM.  The JVM traps 
the SIGILL (signal 4) and prints an error report to the console, 
and then calls abort() to create a core file.  Note that abort() 
kills its calling process by sending a SIGKILL (signal 9) to itself.

This "test" was actually run many times with differing user loads.  
The number of virtual users ranged from 50 down to 1, always running 
in stress mode (no think time).  In all tests involving more than 10 
users, 10 users were started at test start, and the number of users 
was ramped up at a rate of 10 users per minute.

Crashes produced during this test always had the same core image 
file "appearance".  Many threads were active at once (say 50 threads), 
but only one had received the SIGILL which caused the process to shutdown.
Additionally, the stack frame for that thread had 24 HotSpot created 
functions on the stack frame.  (HotSpot created functions do not 
have symbol entries in the symbol tables and as a result appear 
"??" in GDB.) 

Oddly enough, all memory around the instruction which caused the 
SIGILL (and the instruction itself) are zerod out in the core file.  
Is this a "feature" of the core dump facility on Solaris, a bug 
in GDB or really what happened to the JVM?  (i.e. did HotSpot or 
the GC zero out memory it wasn't supposed to zero out?)  I don't 
think this is a GDB bug, as other core files created due to SIGSEGV 
seem to have legal SPARC instructions at the reported PC.

Test 2:
What effect does single threading the server have by reconfiguring 
Silk to only issue one request at a time per user?  If this the 
crash is thread related we should not see a crash in a single 
threaded case.

One virtual user was configured to issue only one request at a time, 
but with no think time between requests.

Result: 3,559 requests in 28 minutes.  No errors.

Test 3:
If test #2 holds true, that single threading works, then running only 
two threads may be able to reproduce the crash.

One virtual user was configured to issue two concurrent requests 
at a time, with no think time between requests.

Result: crash around 5,000 requests after 10 minutes of load.  Crash 
appears to be the same as observed in test #1.  (SIGILL delivered 
with 24 stack frames of HotSpot created functions.)

Test 4:
According to the help for the "java" command, -Xbatch should prevent 
HotSpot from replacing the code of a method at runtime.  Since the 
crash occurs only after processing several hundred requests successfully, 
the crash must be caused by either replacement code generated by 
HotSpot in the middle of the test or a periodic background cleanup 
task triggering off.  Since the single user case held, I'm leaning 
towards the former case.  I'm also thinking that perhaps HotSpot is 
changing the code for a method while another thread is attempting to 
execute it, and the crash is because the executing thread is seeing 
the machine code in the middle of the change (an unstable state).

Result: Using -Xbatch just causes the JVM to suffer from many 
NoClassDefFoundError exceptions in com.ibm.xml.parser.Token.getName.  
Very odd error.  I can only conclude that -Xbatch does not work in 
this version of the JVM.  This test was run many times with the 
same result.  The good news is the JVM does not SIGILL when using 
-Xbatch, but I don't think it gets far enough to receive the SIGILL point.

Additionally, the stack trace created from the NoClassDefFoundError 
is 34 methods deep. Unless HotSpot was able to inline 10 methods to 
reduce the stack depth, this NoClassDefFoundError cannot be the problem 
we are seeing in the other 3 tests.  On top of this, absolutely no 
request completes successfully with -Xbatch enabled, so this is really 
not of any help.

Test 5:
I split the client servlet (SimpleMapTestPlus) into its own JVM, 
seperate from the MapXtreme Java server servlet.  Both ran on the same 
machine underneath the same Apache server.

The result was the same as all other tests (except test #4).  The 
server JVM process died with a SIGILL and its stack trace (as reported 
by GDB) shows 24 HotSpot functions on the call stack.

Test 6:
Paul Jossman suggested using MapXtreme Java 4.0 build 20 as the XML 
parser has been switched away from IBM's XML parser to Apache's Xerces 
parser.  This test was run with a single virtual user allowed to 
make 4 concurrent connections to MapXtreme Java.  Both client and 
server servlets were in the same JVM, and no think time was allowed 
between requests.

The JVM made it through about 4,000 requests before it died.  Its 
death yielded the same SIGILL and 24 frame stack trace as every 
other crash.

Test 7:
MapXtreme Java 4.0 build 20 was tested with -Xbatch enabled to see 
if this cleared up the NoClassDefFoundError seen in Test 4.

After 186 successfully requests, the JVM died with a SIGSEGV (signal 11).  
At least with the Xerces XML parser we do not see the ClassDefNotFoundError.
Examining the core file in gdb reveals that the thread which caught 
the signal was executing in a JVM internal method:

int PhaseChaitin::strech_base_pointer_live_rangs(ResourceArea*)

Some of the classes calling this method seemed to refer to the runtime 
HotSpot compiler.  Perhaps the reason for the crash is an invalid 
pointer dereference in the HotSpot compiler itself.

Test 8:
Under the assumption that test 7's error was a result of trying to 
increase the size of the heap during runtime, I resized the initial 
heap to be the same as the maximum.  However, since the call stack 
contained references to the "Compiler" object, I doubt this is the case.

The test ran successfully for an hour, completing 18,278 image requests 
for one virtual user, no think time and 4 concurrent connections.  
12 hours after the test completed (around 4:20 am) when there was no 
load on the server the JVM randomly SIGSEGVd.

Conclusions:
It would seem as though this particular version of the JVM has some 
errors in its HotSpot code generator (the runtime compiler).  On Solaris, 
after a period of time we see the 32 bit JVM crash with a SIGILL 
having been delivered to the process.  The SIGILL is always issued at 
the same point in our software:  24 stack frames down on the runtime 
stack.  Each of these entries is a dynamicly created method, with only 
the HotSpot error trapping code on one end of the stack frame and the 
JVM thread root functions on the other.  There appear to be 9 functions 
assocaited with the Tomcat call stack, leaving the last 15 functions 
to be (possibly) ones from MapXtreme Java's server servlet.

It would seem as though the crash occurs after handling about 533 
requests. Typically the easiest way to cause the crash is to run 
MapXtreme Java with 1 user stress test loading all 13 images in 
the MapXtreme Java 3.1 test Shawn B. created.  In this test, Silk 
is running 4 concurrent connections per user to the server.

Perhaps the issue with multiple threads is that HotSpot has recreated 
the instructions for the method, but has somehow screwed up in copying 
the new instructions into the method's storage in memory.  As a result, 
some other thread calls into the new method before the new method is 
truely ready for execution, and trys to execute a partially complete 
instruction, or something which was not an instruction (but rather older 
data laying in memory).  Is the invalid instruction a null word or 
something silly like that?

It would seem as though load has no bering on when this crash will 
occur, as even one user (with no think time) can bring the JVM down with 
this error.  I now have 3 core dumps showing identical stack traces from 
a thread dying with this SIGILL.  Unfortunately, since HotSpot 
generates the code on the fly, there is no symbol table associated 
with the stack frames to uncover what method of MapXtreme Java is 
causing the error.  If we can identify the current method of the thread 
that received the SIGILL signal, perhaps we can give Sun a test case 
which can reproduce the error.

With newer releases of the 1.4 JVM, we have to wonder if this 
particular HotSpot bug has been fixed, or if the bug is still present 
but other changes to HotSpot's runtime compiler will cause the number 
of stack frames seen and their alignment in memory to be shifted such 
that it doesn't appear to be the same error.

All fingers point to MapXtreme Java as the software causing HotSpot 
to generate illegal machine code, as the NullServlet test with Tomcat 
did not suffer from this runtime problem.  Since we can run several 
hundred successful requests through the server before the crash, I am 
lead to believe that either HotSpot recompiles the victim method 
improperly as a performance improvement, or that the victim method 
really is just called only infrequently by MapXtreme Java (and as it 
happens is only called once every few hundred requests as a background 
cleanup process for example).

Update:

After examining most of the core files from the tests, it would appear 
as though the memory has been zeroed around the instruction which is 
causing the illegal instruction.  I examined the stacks of every 
active thread visible in the core files, and it would appear as though 
the memory was zeroed while the victim thread was sleeping.  When it 
woke up it died with a SIGILL.  It is not known when the memory zeroing 
occured - it may have occured while the thread was waiting for IO or a 
system call (or AWT) call to complete, and then it called a HotSpot'd 
method which had been zeroed over.  Or it was preempted, and while 
waiting for control one or more methods were zeroed behind its back.

This really looks like a race condition, and the method being zeroed 
looks like its also a MapXtreme Java server side method.