When running promotion testing with -XX:+UseG1GC the 32 bit VM crashes with:
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 32756 bytes for ChunkPool::allocate
Looking at the pmap of the crashing process it has filled its address space with 2048k blocks of memory, so I suspected that some thread has been leaking handles.
UTE rerun cmd:
ute -jdk $my_jdk -testlist nsk.sysdict.testlist -test $test_name -vmopts "-server -Xmixed -XX:+UseG1GC"
(may need a few attempts)
I built a modified VM that prints how many handles each thread has allocated in the hs_err file thread list and it looks like one of the test's worker threads had allocated 734677070 handles.
When the faulty condition is triggered the VM's virtual memory usage grows rapidly and leads to native-oom, so it does not appear to be a slow leak but rather a weird condition or loop that the VM gets caught in.