Bug ID: JDK-7176479 G1: JVM crashes on T5-8 system with 1.5 TB heap

Type: Bug
Component: hotspot
Sub-Component: gc

Priority: P3
Status: Resolved
Resolution: Fixed
OS: solaris_11
CPU: sparc

Submitted: 2012-06-12
Updated: 2014-03-12
Resolved: 2013-05-09

JDK 7	JDK 8	Other
7u40Fixed	8Fixed	hs24Fixed

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (concurrentG1Refine.cpp:114), pid=1385, tid=2
#  guarantee(_max_cards < max_card_num) failed: card_num representation
#
# JRE version: 7.0_04-b20
# Java VM: Java HotSpot(TM) 64-Bit Server VM (23.0-b21 mixed mode solaris-sparc )
# Core dump written. Default location: /root/kganesan/4thJune_after_finalcut/trunk/target/core or core.1385
#
# An error report file with more information is saved as:
# /root/kganesan/4thJune_after_finalcut/trunk/target/hs_err_pid1385.log
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.sun.com/bugreport/crash.jsp
#

Commandline used:
LD_PRELOAD=libumem.so ../../solaris-sparcv9/bin/java -Xbootclasspath/p:cryper.jar:jceseq.jar:jceseqman.jar:sesman.jar -d64 -Xmx1560g -Xms1560g  -Xmn1360g -XX:+UseG1GC -XX:+AlwaysPreTouch -XX:+UseCondCardMark -XX:+UseLargePages -XX:+UseNUMA -XX:+AggressiveOpts -XX:-UseBiasedLocking

Implemented the strategy laid out above: * The hot card cache is in its own file(s) * The flag that controls the number of entries in the hot card cache (and whether to allocate the hot card cache) is now available in the product. * Simpler interface for the hot card cache. The insertion routine will return the card to be refined (which may or may not be the current card) or NULL. * The code in g1RemSet has been refactored so that a single card is refined. * The card counts table has been placed into its own files(s). * The hot card cache and the card counts have been decoupled, i.e. the hot card cache can exist while the card counts table may not. * Simpler interface for the counts table: initialize(), resize(), is_hot(), clear_all(), clear_range(). * Allows us to change the representation of the card counts table in to a more sparse representation in the future. * The flag that specifies the number of times a card has to be refined before being considered hot is no a product flag. If the value of this flag is 0 (i.e. all the cards are hot) then the counts table is no long allocated. * Counts table is currently a 1:1 correspondence with the card table. During initialization space is reserved for the counts table. When the heap is expanded (including initial expansion) the appropriate amount in the space reserved for the counts table is committed. If the expansion of the committed space for the counts table fails, cards which would map to the uncommitted portion of the counts table are always considered cold. * Counts are only zeroed when regions are freed - i.e. during a GC or a cleanup. If a card is hot before a GC or cleanup then it's considered hot after the GC or cleanup - unless the region spanning the card was collected/freed. * Since we don't refine cards in young regions, we only need to clear the counts for cards in non-young regions during the GC. * The entire card counts table is cleared during a full GC as the entire heap has been compacted/maniplauted. With these changes I was able to allocate a heap > 1TB. Sanity testing on a T5 system: SunOS pae-t58-02 5.11 11.1 sun4v sparc sun4v -bash-4.1$ /sbin/psrinfo | wc 1024 5120 40874 -bash-4.1$ /sbin/swap -s -h total: 1.8G allocated + 772M reserved = 2.5G used, 1.7T available with the amount of heap in the original report (-Xms1560g -Xmx1560g -Xmn1360g) I got the following (with the default # of GC threads): -bash-4.1$ ./bin/java -d64 -server -XX:+UseG1GC -Xms1560g -Xmx1560g -Xmn1360g -version Java HotSpot(TM) 64-Bit Server VM warning: os::commit_memory failed Error occurred during initialization of VM Failed to allocate initial heap. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. Or: -bash-4.1$ ./bin/java -d64 -server -XX:+UseG1GC -Xms1560g -Xmx1560g -Xmn1360g -version Java HotSpot(TM) 64-Bit Server VM warning: os::commit_memory failed Error occurred during initialization of VM Failed to allocate initial heap. Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. With a single GC thread we get: -bash-4.1$ ./bin/java -d64 -server -XX:+UseG1GC -Xms1560g -Xmx1560g -Xmn1360g -XX:ParallelGCThreads=1 -version ### Clearing range: [0, 3271557120) java version "1.8.0-ea-fastdebug" Java(TM) SE Runtime Environment (build 1.8.0-ea-fastdebug-b71) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b15-internal-201301101745.jcuthber.g1-hot-card-cache-fastdebug, mixed mode) and a swap history of: total: 1.5G allocated + 16384E reserved = 1.1G used, 1.7T available total: 1.8G allocated + 3.0G reserved = 4.8G used, 1.7T available total: 4.1G allocated + 1.6T reserved = 1.6T used, 110G available total: 7.5G allocated + 1.6T reserved = 1.6T used, 107G available total: 8.2G allocated + 1.6T reserved = 1.6T used, 107G available total: 10G allocated + 1.6T reserved = 1.6T used, 107G available total: 12G allocated + 1.6T reserved = 1.6T used, 107G available total: 7.7G allocated + 1.5T reserved = 1.5T used, 157G available With 100 GC threads, we get: -bash-4.1$ ./bin/java -d64 -server -XX:+UseG1GC -Xms1560g -Xmx1560g -Xmn1360g -XX:ParallelGCThreads=100 -version ### Clearing range: [0, 3271557120) java version "1.8.0-ea-fastdebug" Java(TM) SE Runtime Environment (build 1.8.0-ea-fastdebug-b71) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b15-internal-201301101745.jcuthber.g1-hot-card-cache-fastdebug, mixed mode) and the following swap history: total: 40G allocated + 52G reserved = 92G used, 1.6T available total: 40G allocated + 52G reserved = 92G used, 1.6T available total: 40G allocated + 52G reserved = 92G used, 1.6T available total: 40G allocated + 1.6T reserved = 1.6T used, 72G available total: 44G allocated + 1.6T reserved = 1.6T used, 69G available total: 45G allocated + 1.6T reserved = 1.6T used, 69G available with 256 threads we are able to successfully initialize the JVM: -bash-4.1$ ./bin/java -d64 -server -XX:+UseG1GC -Xms1560g -Xmx1560g -Xmn1360g -XX:ParallelGCThreads=256 -version ### Clearing range: [0, 3271557120) java version "1.8.0-ea-fastdebug" Java(TM) SE Runtime Environment (build 1.8.0-ea-fastdebug-b71) Java HotSpot(TM) 64-Bit Server VM (build 25.0-b15-internal-201301101745.jcuthber.g1-hot-card-cache-fastdebug, mixed mode) with the swap history: total: 100G allocated + 52G reserved = 152G used, 1.5T available total: 100G allocated + 52G reserved = 152G used, 1.5T available total: 104G allocated + 1.6T reserved = 1.7T used, 9.9G available total: 106G allocated + 1.6T reserved = 1.7T used, 9.9G available total: 106G allocated + 1.6T reserved = 1.7T used, 9.9G available total: 108G allocated + 1.6T reserved = 1.7T used, 9.6G available total: 1.7G allocated + 16384E reserved = 1.5G used, 1.7T available total: 1.7G allocated + 16384E reserved = 1.5G used, 1.7T available So it looks like between 128 and 256 threads can be used for running an actual sanity test.

11-01-2013

A few observations that might help fixing this: * We don't add cards spanning young regions to RSets. The card table entries spanned by a young region are explicitly dirtied when the region is allocated. As a result the G1 post-write-barrier does not enqueue cards that are in young regions. * If we have a 'hot' card before a GC then it's likely that the card will still be hot after the GC unless the object(s) spanned by the card are evacuated by the GC. This suggests that we only need to clear the counts for the cards spanned by a region when the region is freed (either at the end of a GC or cleanup pause) - not the whole card count table. * Cards that have been refined only a few times are considered hot. Ideally I would also like to decouple the existence of the hot card card cache from the card count table. This would mean that a count threshold of 0 would force every card to be pushed into and go through the hot card cache. I would also like to ensure that the cache_insert routine returns exactly one card (the card to be refined) - which could be a card evicted from the hot cache or the current card. What I propose is to go back to a table that has a 1:1 correspondence between cards and their refinement counts. Since the count threshold is small we would use a single byte (or fewer bits) to hold the count for the card. (It could be possible to use the upper bits of the entries in the actual card table to track the counts but that would mean using masks to check the card entry value in the G1 post-write barrier.) We might look into this option if footprint becomes an issue. Further we would eliminate the epoch value stored with every count - reducing footprint but raising the problem of clearing the count table. To reduce this overhead we would only clear the count table entries for cards in regions that free (after the collection or after a cleanup). Further we would only absolutely need to clear the count table for cards in old regions that we free. After a full GC we would have to clear the entire count table. Since we would again have a 1:1 correspondence between cards and their counts the cache insertion routine would return exactly one card - cleaning up ConcurrentG1Refine::refine_card(). If the number of entries in the hot card cache is 0, we don't allocate the hot card cache nor the counting table (regardless of what the count threshold has been set to). If the number of entries in the hot card cache is non-zero then we don't allocate the count table if the count threshold is 0 (this means that we'll treat every card we enqueue as hot). I would prefer to have the count table in its own file with its own defined operations.

04-12-2012