JDK-6697161 : Compiler memory leak
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 5.0u14
  • Priority: P2
  • Status: Closed
  • Resolution: Duplicate
  • OS: solaris_10
  • CPU: sparc
  • Submitted: 2008-05-02
  • Updated: 2010-04-03
  • Resolved: 2009-04-30
Related Reports
Duplicate :  
Description
Customer reported native memory leak during compilation.
(refer to radiance case 11209644 for additional details)

* Observed growing native heap eventually leads to OOM

root@kmcicap2 # more pmap-x-29561.log

29561: /usr/jdk1.5.0_14/bin/sparcv9/java -server -Dweblogic.Name=mcis3
-Xms20
-Xms20
Address Kbytes RSS Anon Locked Mode Mapped File
0000000100114000 2992 1760 1256 - rwx-- [ heap ]
0000000100400000 3956736 2575432 1961984 - rwx-- [ heap ]
00000001F1C00000 3891200 2453624 1802240 - rwx-- [ heap ]
00000002DF400000 4190208 3452952 3145728 - rwx-- [ heap ]
00000003DF000000 3579904 2973528 2621440 - rwx-- [ heap ]
00000004B9800000 4190208 2988688 2314240 - rwx-- [ heap ]
00000005B9400000 1327104 1327104 1327104 - rwx-- [ heap ]


* At that time, there were many brk() system calls on t@19 to allocate memory

root@kmcicap2 #truss -dD -p 29561

Base time stamp:  1208961051.2012  [ Wed Apr 23 23:30:51 KST 2008 ]
/22:	184.5903	184.5903	pollsys(0x00000000, 0, 0x7FFFFFFF713FF970, 0x00000000) = 0
/19:	184.6064	184.6064	brk(0x1C073E000)   = 0
/19:	184.6206	 0.0142 brk(0x1C074E000)   = 0
/19:	184.6336	 0.0130 brk(0x1C075E000)   = 0
/19:	184.6402	 0.0066 brk(0x1C076E000)   = 0
/22:	184.6468	 0.0565 pollsys(0x00000000, 0, 0x7FFFFFFF713FF970, 0x00000000) = 0
/19:	184.6527	 0.0125 brk(0x1C077E000)   = 0
/19:	184.6662	 0.0135 brk(0x1C078E000)   = 0
/53:	184.6668	184.6668	pollsys(0x7FFFFFFF5EAFEE68, 1, 0x7FFFFFFF5EAFED90, 0x00000000) = 0
/19:	184.6839	 0.0177 brk(0x1C079E000)   = 0
/19:	184.6989	 0.0150 brk(0x1C07AE000)   = 0
/22:	184.7069	 0.0601 pollsys(0x00000000, 0, 0x7FFFFFFF713FF970, 0x00000000) = 0
/19:	184.7136	 0.0147 brk(0x1C07BE000)   = 0
/19:	184.7305	 0.0169 brk(0x1C07CE000)   = 0
/19:	184.7447	 0.0142 brk(0x1C07DE000)   = 0
/19:	184.7609	 0.0162 brk(0x1C07EE000)

* t@19 from pstack:

-----------------  lwp# 19 / thread# 19	 --------------------
7fffffff7e5fe3cc __1cENodeFclone6kM_p0_ (26b7fff20, 30, 26b7ffff0, 26b7fff38, 3fffffffbfc849, 7fffffff719febc0) + 11c
7fffffff7e657480 __1cHAddNodeFIdeal6MpnIPhaseGVN_i_pnENode__ (26b7fff88, 7fffffff719fe2a8, 26b7fff20, 0, e, 26b7fff50) + 614
7fffffff7e5eb2ec __1cMPhaseIterGVNNtransform_old6MpnENode__2_ (7fffffff719fe2a8, 26b7fff88, 1, 1d1358020, 7ffffff, 7fffffff7e662118) + a4
7fffffff7e6e4480 __1cMPhaseIterGVNIoptimize6M_v_ (7fffffff719fe2a8, 26b7fff88, 26b7fff88, 0, 7fffffff719fe518, ffffffffffffffff) + b8
7fffffff7e784218 __1cHCompileIOptimize6M_v_ (7fffffff719febc0, 10050f470, 6a1ea4, 7fffffff7e63c738, fc00, 7fffffff7eca8abf) +  c8
7fffffff7e78747c __1cHCompile2t5B6MpnFciEnv_pnKC2Compiler_pnIciMethod_ii_v_ (7fffffff719febc0, 0, 7fffffff719ffb28, 103dd3ff8, 0, d 80) + c3c
7fffffff7e77daec __1cKC2CompilerOcompile_method6MpnFciEnv_pnIciMethod_i_v_ (7fffffff719ffb28, 7fffffff7ec91290, 1040d4560, 25, 1005 39d50, 7fffffff719fed60) + b4
7fffffff7e77e0b4 __1cNCompileBrokerZinvoke_compiler_on_method6FpnLCompileTask__v_ (1040d4560, 10050f470, 1002266d0, 0, 7fffffff7eec 7e10, 7fffffff7ee26000) + 504
7fffffff7e84b3d0 __1cNCompileBrokerUcompiler_thread_loop6F_v_ (7fffffff7eec7e10, 10050f600, 1002266d0, 10054c720, 7fffffff7ee  b3548, 7fffffff7eeb354c) + 450
7fffffff7e7e4734 __1cKJavaThreadDrun6M_v_ (1002266d0, f000, 7fffffff7eebdd0c, 0, 0, 7fffffff7ee26000) + 2ac
7fffffff7ebbd0d0 __1cG_start6Fpv_0_ (1002266d0, d800, b000, b388, 7fffffff7eec8f64, 7fffffff7ee26000) + 210
7fffffff7f5d2298 _lwp_start (0, 0, 0, 0, 0, 0)

* Successful workaround by excluding one method from compilation

-XX:CompileCommand=exclude,org/aspectj/apache/bcel/classfile/Attribute,readAttribute
Looks like a bug in JIT optimization


* Customer has also opened an incident
  http://webbugs.sfbay/rt/incidentDisplay?incidentID=1229942

  where they provide a testcase and instructions to reproduce the problem
  (with tomcat/BCEL)

  Here's summary of customer's observation:
  (for details, refer to the above incident)

  1. This native memory leak did not occur when using -Xint option.
  2. During the booting stage, if class that initially compiles the hotspot
     (by registering .hotspot_complier and excluding it) using the
     PrintCompliation option, the problem did not occur.
  3. The problem did not occur consistently, the problem occurred occasionally
     after multiple restarts.
  4. In another system, I found cases that are related to this problem.
     In those c ases the native memory leak occurred 10+ hours after the
     server was rebooted.  (Solaris 32bit/64bit, Linux 32bit, etc...)
  5. java heap memory did not increase while monitoring using jconsole.
  6. I couldn't determine the root-cause using gcore/mdb:: findleaks.
  7. I noticed that there were a lot of brk() methods used when native memory
     leak occurs after Tomcat' booting is completed.
  8. I do not think that this is a bytecode logic problem because test program
     doesn't transform the original class.
  9. The native memory leak happened on the latest Solaris 10(SPARC 64bit)
     with al l fixes installed, so it is probably not related to OS
     environment problem.
 10. After 3~30 minutes passes from the time native memory leak occurs, the
     JAVA process shuts down with message
     "Exception in thread "CompilerThread1" java.lang .OutOfMemoryError: requested 134217736 bytes for Chunk::new. Out of swap space"


* In addition, a core file has been collected
  /net/cores.central/cores/dir20/11209644 with pkgcore.
escalated to sustaining Apollo 1-23815642

Comments
EVALUATION this is dup of 6676462
30-04-2009