JDK-8043766 : CMM Testing: 8u40 Decommit auxiliary data structures
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 8u40,9
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2014-05-22
  • Updated: 2015-06-03
  • Resolved: 2014-10-10
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
8u40 b11Fixed 9Fixed
Related Reports
Relates :  
Test checking that native memory is returning to the system.

---- Details from  Thomas Schatzl -----

Main testing should be in the area of 
a) not breaking existing functionality - the feature is completely
transparent to the VM and
b) memory usage reductions.

What the feature does:
- uncommits parts of the Java heap and auxiliary data according to the
current sizing policy (e.g. HeapMin/MaxFreeRatio) at full GCs.

Full GCs are the only point in time where the heap is decreased.

The release (uncommit) of this data is based on regions. I.e. the unit
of memory freeing is regions, when a region is freed, all its associated
auxiliary data should go away as required.

There is one complication in this matter, and that is that the unit of
commit/uncommit is always OS memory pages, but an OS page worth of
auxiliary data may cover multiple regions. If one of these regions is
still not uncommitted, the auxiliary data cannot be freed either.

This ratio between OS memory page size and region size is not fixed too,
both OS memory page size and region size can vary depending on flags
(UseLargePages etc).

The change is G1 only too.

So, what are the relevant memory consumers:
 - Java heap: split into disjoint regions, given G1RegionSize (range 1M
to 32M, power of two increments); can be managed either with "OS page
size" or large pages (-XX:+UseLargePages active).

 - Auxiliary data:
   contents of auxiliary data always map directly to a disjoint
contiguous section of the Java heap. Their memory is *always* managed in
native page size.

   - Two mark bitmaps, where each bit maps to an area of
MinObjAlignmentInBytes * BitsPerByte bits.
MinObjAlignmentInBytes can vary from 8 (default) to 256 iirc (actually I
cannot find a bound, but it must be a power of two).

This amounts to ~3% of heap space by default; if 4k pages are used (e.g.
on x86), a single page spans 256k of heap (i.e. only a fraction of a
region in any case) - but with increasing MinObjAlignmentInBytes a
single page may span multiple regions.

  - Three "tables" (card table, offset table, hot card cache count)
where in each element of them spans an area of 512 bytes. This is
hardcoded at this time.

The hot card cache count table is only used if it is enabled
(G1ConcRSLogCacheSize is larger than zero).

E.g. with 4k pages, each of them span 2 MB of heap.

In total they make up 0.6% of the Java heap.

Current status:
- there is a prototype available (for creating tests) that manages the
Java heap, the mark bitmaps, and one of the mentioned tables.

So I think you can already start working on tests.

Planned date that this feature to be finished and in the repos is 9th of
June. It may be delayed slightly because we did not plan my vacation (I
talked to Paul about this already) into that time.

I will start getting reviews for the first changes next week.

Difficulties I see here:
- getting reliable information of actual uncommitted memory. I.e. it is
not only important that the VM reports that the heap shrank, but actual
memory usage (e.g. RSS size or whatever) adjusted correctly (i.e. that
the commit/uncommit calls are made correctly).
Also somehow the amount of memory committed/uncommitted what the VM
thinks should probably be checked.

- there is a fairly large amount of possible combinations of settings
that should be checked.

- getting the VM to reliably uncommit a particular area. Possibly best
achievable using large object allocations.

SQE recommends to take this fix in 8u40. SQE is running these tests for 8u40 anyway even if they are not integrated.

- Justification������������������������������ : test cases for functionality already in 8u40. - Risk Analysis����������������������������: no product problems, at most test failures. Already pushed to jdk 9 a few days ago, nightlies do not show problems. - Webrev�������������������������������������������� : http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2014-September/010868.html - Testing (done/to-be-done)������ : This is a collection of jtreg test cases. - Fix For Release������������������ : JDK 8 CPU/PSU

review: http://mail.openjdk.java.net/pipermail/hotspot-gc-dev/2014-September/010868.html

Suggested fix: new test: test/gc/g1/TestAuxiliaryDataDecommit.java The test should: - allocate enough memory to cause extra memory commit - release allocated objects - invoke GC - make sure that memory is decommitted The test should be invoked with: - Min/MaxHeapFreeRatio flags set to cause decommit - Various combinations of MinObjAlignmentInBytes and G1ConcRSLogCacheSize flags The test should be also be executed on host with large pages (with various settings). This could be done as part of a separate RFE

Original umbrella CR