The default MaxHeapSize is MIN(memory/4, 1GB). It works fine for all GCs except CMS. It looks that it is about 128M by default for Linux with 4G of memory and does not increase when I use the memory.  I checked it for different versions of Java (6 and 7), archs and platforms and find the same behaviour. All works fine when I explicitly set MaxHeapSize for CMS. 

This issue is critical for CMS testing. The VM SQE tests does not set change any memory settings so they does not test CMS on really large memory size. Please evaluate it if it is expected behaviotr or you do not plan to fix it. In this case the SQE tests should be updated.

I attached the liitle memory eater.

Here is the oputput:

lm153972@vmsqe-core2q-30:~//ws/bugs>
-->/set/vmsqe/jdk/re/6/archive/fcs/binaries/linux-amd64/fastdebug/bin/java -server  -XX:+UseConcMarkSweepGC -verbose:gc -XX:+PrintHeapAtGC  LogTest 2 >&1   | more
{Heap before GC invocations=0 (full 0):
VM option '+UseConcMarkSweepGC'
VM option '+PrintHeapAtGC'
 par new generation   total 19136K, used 10580K [0x00002aaaae410000, 0x00002aaaaf8d0000, 0x00002aaaaf8d0000)
  eden space 17024K,  62% used [0x00002aaaae410000, 0x00002aaaaee65218, 0x00002aaaaf4b0000)
  from space 2112K,   0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000, 0x00002aaaaf6c0000)
  to   space 2112K,   0% used [0x00002aaaaf6c0000, 0x00002aaaaf6c0000, 0x00002aaaaf8d0000)
 concurrent mark-sweep generation total 62656K, used 0K [0x00002aaaaf8d0000, 0x00002aaab3600000, 0x00002aaab3810000)
 concurrent-mark-sweep perm gen total 21248K, used 2647K [0x00002aaab3810000, 0x00002aaab4cd0000, 0x00002aaab8c10000)
[GC 10580K->10399K(81792K), 0.0323520 secs]
Heap after GC invocations=1 (full 0):
 par new generation   total 19136K, used 157K [0x00002aaaae410000, 0x00002aaaaf8d0000, 0x00002aaaaf8d0000)
  eden space 17024K,   0% used [0x00002aaaae410000, 0x00002aaaae410000, 0x00002aaaaf4b0000)
  from space 2112K,   7% used [0x00002aaaaf6c0000, 0x00002aaaaf6e7500, 0x00002aaaaf8d0000)
  to   space 2112K,   0% used [0x00002aaaaf4b0000, 0x00002aaaaf4b0000, 0x00002aaaaf6c0000)
 concurrent mark-sweep generation total 62656K, used 10242K [0x00002aaaaf8d0000, 0x00002aaab3600000, 0x00002aaab3810000)
 concurrent-mark-sweep perm gen total 21248K, used 2647K [0x00002aaab3810000, 0x00002aaab4cd0000, 0x00002aaab8c10000)
}