It is the bug in original implementation of JDK-8043766.

The test tries to run with -XX:G1ConcRSLogCacheSize=30, while the max acceptable value is 27:

  product(size_t, G1ConcRSLogCacheSize, 10,                                 \
          "Log base 2 of the length of conc RS hot-card cache.")            \
          range(0, 27)                                                      \

It only happens on large machines, because smaller machines get the test skipped with:

        int maxCacheSize = Math.max(0, Math.min(31, getMaxCacheSize()));
        if (maxCacheSize < hotCardTableSize) {
            throw new SkippedException(String.format(
                    "Skiping test for %d cache size due max cache size %d",
                    hotCardTableSize, maxCacheSize));
        }

getMaxCacheSize() is derived from the Runtime.getRuntime().freeMemory():

    private static int getMaxCacheSize() {
        long availableMemory = Runtime.getRuntime().freeMemory()
                - ShrinkAuxiliaryDataTest.getMemoryUsedByTest() - 1l;
        if (availableMemory <= 0) {
            return 0;
        }

        long availablePointersCount = availableMemory / Unsafe.ADDRESS_SIZE;
        return (63 - (int) Long.numberOfLeadingZeros(availablePointersCount));
    }

So, on smaller machines, for example, my 128G desktop:

jtreg.SkippedException: Skiping test for 30 cache size due max cache size 26
	at gc.g1.TestShrinkAuxiliaryData.test(TestShrinkAuxiliaryData.java:77)

On my 1T server, this test is not skipped and reliably fails like:

 stdout: [];
 stderr: [size_t G1ConcRSLogCacheSize=30 is outside the allowed range [ 0 ... 27 ]
Improperly specified VM option 'G1ConcRSLogCacheSize=30'
Error: Could not create the Java Virtual Machine.
Error: A fatal exception has occurred. Program will exit.
]
 exitValue = 1