Bug ID: JDK-4134584 In 1.1.6: *** panic: 16-bit string hash table overflow

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 1.1.6,1.1.7

Priority: P1
Status: Closed
Resolution: Fixed
OS: generic,windows_nt
CPU: x86

Submitted: 1998-05-03
Updated: 2012-10-08
Resolved: 1998-07-15

Other	Other	Other
1.1.6_007 007Fixed	1.1.7Fixed	1.1.8Fixed


Name: tb29552			Date: 05/02/98


Users report (and we have verified) that running
our document indexing program under Windows NT 4.0
with the JIT compiler supplied with JDK1.1.6 produces
the following error message for relatively small text
collections (10 MB or so of HTML) :

*** panic: 16-bit string hash table overflow

abnormal program termination

Would appreciate info regarding the source and
possible resolution of this problem, which 
appears to be new under 1.1.6 .
(Review ID: 29559)
======================================================================

CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: 1.1.7 FIXED IN: 1.1.6_007 1.1.7 1.1.7b_004 1.1.8 INTEGRATED IN: 1.1.6_007 1.1.7 1.1.7b_004 1.1.8
14-06-2004
PUBLIC COMMENTS .
10-06-2004
EVALUATION This bug will be forwarded to Symantec for further investigation. deepa.viswanathan@eng 1998-05-04 This bug was reproducible without the JIT (See the message below from the user) Since it is not clear what is causing the bug, I am re-assigning the bug to "other" sub-category. Date: Tue, 12 May 1998 14:11:50 -0700 (PDT) From: Timothy Bell <tim.bell@sealserver> Subject: Re: BugID 4134584 (opaque error message: * panic: 16-bit string hash table overflow) To: Deepa Viswanathan <dviswana@taller> MIME-Version: 1.0 Content-Transfer-Encoding: QUOTED-PRINTABLE Content-MD5: oIpDasoFYYdMRUEIaMZesA== FYI: Looks like this bug (eBugID 4134584) may not be against the JIT. Note, however, that he was unable to run the symcjit_g.dll because it wanted "MSVCRTD.DLL" Tim x53241 ------------- Begin Forwarded Message ------------- Date: Tue, 12 May 1998 16:44:27 -0400 From: "Mike O'Connor" <###@###.###> MIME-Version: 1.0 To: Timothy Bell <###@###.###> Subject: Re: BugID 4134584 (opaque error message: * panic: 16-bit string hash table overflow) Content-Transfer-Encoding: quoted-printable X-MIME-Autoconverted: from 8bit to quoted-printable by earth.sun.com id NAA06226 Hi Tim, I just reproduced the bug in Sun-JVM-modulo-JIT (-nojit). I infer that the problem is Sun's and not Symantec's. JDK1.1.5K/green-threads/nojit on Ultrasparc doesn't have this problem. Nor does 1.2beta3 on Ultrasparc. I will try the latest WinNT 1.2 beta tomorrow morning and let you know if it is also bug-free. I was also able to reproduce the bug under the new symcjit.dll you supplied. I was not able to use symcjit_g.dll; it wants "MSVCRTD.DLL", which doesn't seem to be on our NT 4 Server box, and I suspect it wouldn't shed any light on this particular bug. I would be happy to supply a .jar file containing our product and an example document collection which will tickle the bug. The total size of that test package would be on the order of 5 MB. Please let me know if you'd like that as an email attachment or if I should point you at an ftp site. Please note that even for a document collection of only 10 MB, it is perfectly normal for our program to create 150 000 Strings, as we require a String for every unique word. Our software is actually designed for terabyte-sized document collections, and the number of Strings grows linearly with the size of the document collection in the worst case (mostly due to misspelled words). If there is an intentional hard limit on the number of String objects we can allocate in your 1.1 JVM, could you please tell us about it so that we can code around it? We hate to have to refer people to Microsoft's JVM, and will go to almost any length to accomodate your JVM's idiosyncrasies. is all. Timothy Bell wrote: > > Hello Mike > > I've been asked to pass this new drop of the JIT > compiler to you. > > If you could spend some time running your test scenario > using this new JIT, I'd appreciate it. This JIT is not > ready to go out to the world at large... as I understand > it, it's just the results of the weekly build, so please > don't pass it on to anyone. > > please unzip the attached file to get the symcjit.dll and > symcjit_g.dll files. > > To test with the new jit, just replace the current symcjit.dll in > the 1.1.6 bin dir with the new one. I suggest that you save the > old files first by renaming them (EG: to symcjit.dll_01 and > symcjit_g.dll_01). Then run the app as usual. I am also > attaching a symcjit_g.dll to get more debugging info. > > ------------- Begin Forwarded Message ------------- > > Date: Fri, 8 May 1998 13:05:32 -0700 (PDT) > From: Deepa Viswanathan <dviswana@taller> > Subject: pass the new symcjit.dll to your customers > To: ellis.yu@Canada, tim.bell@Eng, sandhya.vora@Eng, > dale.mcduffie@Eng > Cc: dbowen@taller > MIME-Version: 1.0 > > Hi all, > > We are actively investigating the 1.1.6 Symantec JIT bugs. The > following > bugs submitted by you don't have information to be reproduced. > But we do > have a new drop of the Symantec JIT that could potentially fix the > problems. > > bugid synopsis > submitter > 4135304 Retreiving Float/Double data thru JDBC-ODBC bridge causes > an error (ellis) > 4134584 opaque error message: * panic: 16-bit string hash table > overflow (tbell) > 4134585 getThreadContext failed (errcode: 57) jre116 hard crash > (tbell) > 4122416 internal JIT (3.00.030(x)) error 'Unuse64' has occured > (sandhya) > 4133602 1.1.6 jit crashes when reading double from ResultSet > (dalem) > 4133712 Internal JIT error in 1.1.6 ('StackFPU: FPU0') > (dalem) > > Can you please pass the attached symcjit.dll to your customers and > ask them > to verify if their bugs have been fixed? This input would be > critical > in deciding if the new drop is good enough to be put out as a > patch on the Web. > So, can you please try to get this info by Monday? > > To test with the new jit, just replace the current symcjit.dll in > the 1.1.6 bin > dir with the new one and run the app. as usual. I am also > attaching a symcjit_g.dll > to get more debugging info. > > Thanks for all your efforts, > Deepa > (Responsible engineer for Symantec JIT bugs) > > p.s please unzip the attached file to get the symcjit.dll and > symcjit_g.dll > > ------------- End Forwarded Message ------------- > > Tim Bell ###@###.### Source Integration > Engineering > Java Software Division of Sun Microsystems, Inc. > > "JavaSoft(TM), Java(TM), and Java(TM) Development Kit > are all trademarks of Sun Microsystems, Inc." > "JavaSoft(TM), Java(TM), et Java(TM) Development Kit > sont des marques d��pos��es ou enregistr��es de Sun > Microsystems, Inc. aux Etats-Unis et dans d'autres pays." > > --------------------------------------------------------------------------- --------------- > > Name: x116b043.zip > x116b043.zip Type: Zip Compressed Data (application/x-zip-compressed) > Encoding: BASE64 > Description: x116b043.zip ------------- End Forwarded Message ------------- Tim Bell ###@###.### Source Integration Engineering Java Software Division of Sun Microsystems, Inc. "JavaSoft(TM), Java(TM), and Java(TM) Development Kit are all trademarks of Sun Microsystems, Inc." "JavaSoft(TM), Java(TM), et Java(TM) Development Kit sont des marques d��pos��es ou enregistr��es de Sun Microsystems, Inc. aux Etats-Unis et dans d'autres pays." deepa.viswanathan@eng 1998-05-19 I located the problem with the miracle of grep technology. Check out src/share/java/util/util.c, line 276. Happily, this message does not still appear anywhere in the 1.2 sources. david.stoutamire@Eng 1998-05-19 See the comment section for feasibility/risk assessment. david.stoutamire@Eng 1998-06-03 ------------------------------------------------------- The following code still gives the error on 1.1.7l class Bug { public static void main(String[] args) { int i=0; String x; while (true) { x=(""+i).intern(); System.out.println(i); i++; } } } Output : ... ... 26760 26761 26762 * panic: 16-bit string hash table overflow SIGABRT 6* abort (generated by abort(3) routine) si_signo [6]: SIGABRT 6* abort (generated by abort(3) routine) si_errno [0]: Error 0 si_code [-1]: SI_LWP [pid: 23632, uid: 56526] stackbase=EFFFE138, stackpointer=EFFFDE78 Full thread dump: "Finalizer thread" (TID:0xee300210, sys_thread_t:0xef371db8, state:R) prio=1 "Async Garbage Collector" (TID:0xee300258, sys_thread_t:0xef471db8, state:R) prio=1 "Idle thread" (TID:0xee3002a0, sys_thread_t:0xef541db8, state:R) prio=0 "Clock" (TID:0xee300088, sys_thread_t:0xef571db8, state:CW) prio=12 "main" (TID:0xee3000b0, sys_thread_t:0x6dc90, state:R) prio=5 current thread Bug.main(Bug.java:9) Monitor Cache Dump: Registered Monitor Dump: Thread queue lock: <unowned> Name and type hash table lock: <unowned> String intern lock: owner "main" (0x6dc90, 1 entry) JNI pinning lock: <unowned> JNI global reference lock: <unowned> BinClass lock: <unowned> Class loading lock: <unowned> Java stack lock: <unowned> Code rewrite lock: <unowned> Heap lock: <unowned> Has finalization queue lock: <unowned> Finalize me queue lock: <unowned> Dynamic loading lock: <unowned> Monitor IO lock: <unowned> Child death monitor: <unowned> Event monitor: <unowned> I/O monitor: <unowned> Alarm monitor: <unowned> Waiting to be notified: "Clock" (0xef571db8) Sbrk lock: <unowned> Monitor registry: owner "main" (0x6dc90, 1 entry) Thread Alarm Q: Abort (core dumped) mick.fleming@Ireland 1998-09-22 We now allow Java programs to create as many interned strings as they want. However, other literals such as class name, method name, and method signature are still subject to the 30K limitation. dale.mcduffie@Eng 1999-06-02
02-06-1999
WORK AROUND Name: tb29552 Date: 05/02/98 We are recommending that our customers use Microsoft's JVM until this problem is resolved. EDITOR's NOTE: you could also run the 1.1.6 JVM with the JIT turned off by using: java -nojit [other flags...] <class name> or jre -nojit [other flags...] <class name> Refer to this URL for the Windows version of the Tools Reference Pages: http://www.javasoft.com/products/jdk/1.1/docs/tooldocs/win32/java.html ====================================================================== The workarounds are: load less classes or upgrade to JDK 1.2.x. dale.mcduffie@Eng 1999-06-02
02-06-1999
SUGGESTED FIX IBM have tried out the fix for Sunbug 4134584 and say that it doesn't actually make any difference. They tried it on 1.1.7 as well as backporting it to 1.1.6. I put the testcase in the evaluation. From an IBM note : # I think this is because the extra test added # to the if startement isn't checking the right value, but I'd like to be what # the intention was. # I'm also concerned about whether there any other ramifications to the fix. # # Background.... The Str2ID function returns a unique 16-bit if number for the # string (or whatever we are hashing). Hash tables (struct StrIDHash) are used # to & the id is the slot number the item was hashed to. The table contains # 2003 entries and the baseid of this first table is 1 - i.e. it's first slot # has id=1. When the table is full, a new table is created and linked to the # previous one. The baseid of the new table is base-of-previous+size-of-previous # so table 2 has ids 2004 - 4006, etc. # # Problem.... In 1.1.6, the code in Str2ID in util.c calls a panic if a hash # table fills up & its baseid is > 32000. (This seems a bit of an odd number as # if its base was 32000, its last id would be 34002 which would have wrapped # already. # # The 1.1.7 fix.... just checks whether the hash table is stringhash, and, if # so, it doesn't call panic even though the hash table's baseid is > 32000. This # doesn't help because stringHash is the first string table & by the time the # baseid is > 32000 we are many tables down the line. # # If the intention was to just not panic if it's a string we're hashing, then # that can be done by checking whether hash_ptr is stringHash rather than # whether h is stringHash. (hash_ptr points to the first table in the chain. # h points to the current one.). E.g. # # if (h && h->baseid > 32000 && hash_ptr != stringHash) { # # This would certainly fix the panic but will give some pretty odd answers as # the ID returned is h->baseid + i where h is the table & i is the slot. Once # baseid goes negative (pity it isn't using unsigned short throughout) the # calculation won't give the expected answer. After 65536 strings, (or probably # less due to this anomally), the ids will be reused. How much does this matter? # What ramifications will there be? # # Could you let me know whether I've deduced the fixer's intention correctly? # and whether he/she feels there will be knock-on problems? mick.fleming@Ireland 1998-09-24 ------------------------------------------------------------------------------ This seems to be a dup of 4035345. IBM have fixed this in their 1.1.6 JDK. Here are the diffs, and an explaination : # In the Sun VM there are a number of hash tables for holding unique strings # (stringHash, nameTypeHash and some in the verifier). NameTypeHash is used # during class loading/resolution and stores class and method names and # field-block name/types. StringHash is used to hold constant strings # contained within classes and for interned String objects. # # The Str2ID function (contained in src/share/java/util/util.c) is used to # return a unique 16-bit value for a string (if the string is not already in # the hash table it is added). This seems to limit us to 65536 strings in # each hash table. However, Sun has implemented the tables by chaining # together smaller tables using a short (i.e. signed) baseid (in struct # StrIDhash). They therefore have a check to see if the baseid is > 32,000. # If it is, the VM panics with a hash table overflow. # # The simple program below reproduces this by creating interned strings until # the string hash overflows: # # public class test { # public static void main(String[] args){ # # for(int i = 0; true; i++) # System.out.println(String.valueOf(i).intern()); # } # } # # Sun's JDK 1.1.7B gets to 26755 and then panics. # # It has been fixed in IBM's Java VM for AIX and Windows NT. This fix makes # the baseid unsigned, thereby roughly doubling the possible ID values. It # also removes the panic, using an emergency "extension" to the hash table to # enable the VM to produce an OutOfMemoryError exception when the hash table # overflows (the VM may need to load extra classses to throw the extension - # if the nameTypeHash is full, this will lead to recursion). This requires # small changes to src/share/java/runtime/classinitialize.c and # src/share/java/runtime/classresolver.c. Using the program above, the AIX # JDK gives: # # 50784 # java.lang.OutOfMemoryError String intern table overflow # at test.main(Compiled code) # # $ diff new/src/share/java/runtime/classresolver.c # old/src/share/java/runtime/classresolver.c # # 164c164 # < static char # --- # > static void # 179,181d178 # < if (fb->ID == 0) { # < return JAVAPKG "OutOfMemoryError"; # < } # 207d203 # < return NULL; # 228,230d223 # < if (mb->fb.ID == 0) { # < return JAVAPKG "OutOfMemoryError"; # < } # 834,835c827,828 # < if ((ret = ResolveFields(cb, slot)) \|\| # < (ret = ResolveMethods(cb))) { /* May return # "OutOfMemoryError" / # --- # > ResolveFields(cb, slot); # > if ((ret = ResolveMethods(cb))) { / May return "OutOfMemoryError / # 1184c1177 # < SignalError(0, JAVAPKG "OutOfMemoryError", "nameType hash table # overflow"); # --- # > SignalError(0, JAVAPKG "OutOfMemoryError", 0); # # # # # $ diff new/src/share/java/runtime/classinitialize.c # old/src/share/java/runtime/classinitialize.c # # 320c320 # < SignalError(0, JAVAPKG "OutOfMemoryError", "String # intern table overflow"); # --- # > SignalError(0, JAVAPKG "OutOfMemoryError", 0); # 666c666 # < SignalError(0, JAVAPKG "OutOfMemoryError", "String intern table # overflow"); # --- # > SignalError(0, JAVAPKG "OutOfMemoryError", 0); # # # # # $ diff new/src/share/java/util/util.c old/src/share/java/util/util.c # # 31d30 # < #include <limits.h> # 146d144 # < #define HTLASTSIZE 1439 / last hash size / # 277,288c275,278 # < int warning = 0; # < if (h && h->baseid + h->size > USHRT_MAX - HTSIZE + 1) { # < if (h->baseid + h->size > USHRT_MAX - HTLASTSIZE + 1) { # < return 0; / Should never happen / # < } else { # < / Use space left for clean-up / # < next = createHash(HTLASTSIZE); # < warning = 1; # < } # < } else { # < next = createHash(HTSIZE); # < } # --- # > if (h && h->baseid > 32000) { # > panic("16-bit string hash table overflow"); # > } # > next = createHash(HTSIZE); # 307d296 # < if (warning) return 0; # # # ------------------------------------------------------------------------------ Here are the diffs for 1.1.8 : diff -b old/src/share/java/runtime/classresolver.c new/src/share/java/runtime/classresolver.c 170a171,173 > if (fb->ID == 0) { > return JAVAPKG "OutOfMemoryError"; > } 189d191 < 200c202 < return NULL; --- > return NULL; 220a223,225 > if (mb->fb.ID == 0) { > return JAVAPKG "OutOfMemoryError"; > } 828c833,834 < if ((ret = ResolveFields(cb, slot))) { / May return "InternalError" / --- > if ((ret = ResolveFields(cb, slot)) \|\| > (ret = ResolveMethods(cb))) { / May return "OutOfMemoryError" / 834,839d839 < if ((ret = ResolveMethods(cb))) { < / May return "OutOfMemoryError" or "InternalError" / < detail = cbName(cb); < CCSet(cb, Error); < return ret; < } 840a841 > 1184c1185 < SignalError(0, JAVAPKG "OutOfMemoryError", 0); --- > SignalError(0, JAVAPKG "OutOfMemoryError", "nameType hash table overflow"); -------------------------------------------------------- diff -b old/src/share/java/runtime/classinitialize.c new/src/share/java/runtime/classinitialize.c 448c456 < SignalError(0, JAVAPKG "OutOfMemoryError", "String intern table overflow"); --- > SignalError(0, JAVAPKG "OutOfMemoryError", 0); 658c666 < SignalError(0, JAVAPKG "OutOfMemoryError", 0); --- > SignalError(0, JAVAPKG "OutOfMemoryError", "String intern table overflow"); ------------------------------------------------ diff -b old/src/share/java/util/util.csrc/share/java/util/util.c new/ > #include <limits.h> 30d38 < #include <limits.h> 139a148 > 268,269c277,284 < if (h && h->baseid > 30000 && hash_ptr != stringHash) { < panic("16-bit string hash table overflow"); --- > int warning = 0; > if (h && h->baseid + h->size > USHRT_MAX - HTSIZE + 1) { > if (h->baseid + h->size > USHRT_MAX - HTLASTSIZE + 1) { > return 0; / Should never happen / > } else { > / Use space left for clean-up / > next = createHash(HTLASTSIZE); > warning = 1; 270a286 > } else { 271a288 > } 289a307 > if (warning) return 0; mick.fleming@Ireland 1999-04-09 Following is the revised fix: /share/java/util/util.c 2c2 < @(#)util.c 1.52 98/07/14 --- > * %W% %E% 157c157 < short baseid; /* ID for item in slot[0] / --- > unsigned short baseid; / ID for item in slot[0] / 267,268c267,268 < if (h && h->baseid > 32000 && h != stringHash) { < panic("16-bit string hash table overflow"); --- > if (h && h->baseid > 64000 && hash_ptr != stringHash) { > panic("16-bit nametype hash table overflow"); 409c409 < while ((long)(ID - h->baseid) >= h->size) { --- > while (ID >= (h->baseid + h->size)) { ###@###.### 1999-05-14
14-05-1999

Duplicate :	JDK-4186369 - (1.1) change 32K string table limit to 64K
Relates :	JDK-4233230 - (1.1) In 1.1.8 *** panic: 16-bit string hash table overflow
Relates :	JDK-4185113 - *** panic: 16-bit string hash table overflow
Relates :	JDK-4035345 - String.intern() fails after about 100,000 calls