JDK-6747415 : JVM: java 6u4~ use larger C heap if there are many threads
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 6u4
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: linux_redhat_4.0
  • CPU: x86
  • Submitted: 2008-09-11
  • Updated: 2013-07-18
  • Resolved: 2013-07-18
Related Reports
Duplicate :  
Relates :  
Description
In 6739920, networking change to address the specific symptom is in progress.  However, we need JVM ObjectMonitor management code change too so that customers can avoid or work-around similar situations.

Problem from the original CR:
On RHEL 4.6 64-bit 4G mem, two instances of jakarta-jmeter with 6000 threads worked fine on 6u3 64-bit.
But after upgrading to 6u6, java processes soon cause memory thrashing and become very slow.

By monitoring virtual and resident size of java processes, 6u4 and later use larger memory.  In a case,
6u7 uses 600MB or larger C heap.

By checking malloc related path on gdb, it turned out the most memory allocations come from 
ObjectMonitor allocation.

On 6u4, per-thread ObjectMonitor freelist was introdued (_omFreeList).  In some test, I saw more than
800 ObjectMonitors in average on 6u7.  ObjectMonitor size is 168 bytes on 64-bit JVM for Linux.

6000*800*168=806400000

6u3 can run the same app with 100m to 200m C heap, but 6u4 and later require much larger C heap.

C heap usage levels off at some point, but we cannot know how much memory is really needed to
a specific run of application.  Currently there is no clear limit for the ObjectMonitors on
omFreeList.  

To suppress the excessive C heap use, we need a switch to turn off the per-thread ObjectMonitor
free list or a good mechanism to control the memory usage (Or simply a better implementation).
Switching off the per-thread object monitor caches is relatively easy and only involves a few lines of code, but a quick test with the offending application supports my claim above that there's no benefit to this path.  Only a tiny fraction of the extant objectmnitors are sequestered in the per-thread caches.  Adding the switch is a fool's errand and provides no relief from the root problem, which has existed since day one.

Comments
This was fixed with two flags: MonitorBound and MonitorInUseLists
18-07-2013

EVALUATION Discussed with Dave. The most workable change to the VM to alleviate the problem that is observed in 6739920 is to add an XX switch to enable stop-the-world GC if the number of monitors exceeds a given threshold. This switch wouldn't be useful in the general case as it would cause the application to stop, but would be useful to give to people who have this crash. Using C heap for inflated object monitors and having allocation to a per thread list is something we are not going to change in the current Hotspot VM. Therefore this bug is now an RFE to add an XX switch to relieve C heap usage for specific applications that may be swamped with object monitors. (Dave will provide this code). I'm going to change the priority of this (and give it away..)
02-10-2008