TestOptionsWithRanges shows that illegal values for red and yellow pages are possible.
In JavaThread::create_stack_guard_pages(),
setting StackRedPages=92233720368547753 and StackYellowPages=1 yields
len = (92233720368547753 + 1) * os::vm_page_size()
= 0x8000000000000000 * os::vm_page_size()
= 0
Thus not protecting any pages of the stack.
The check in os::init_2() succeeds, as there the Shadow pages are considered, too, so that the overflowed value is > 0.
On linux, the VM stops with the misleading message
"OpenJDK 64-Bit Server VM warning: Attempt to allocate stack guard pages failed."
On aix, we get assertions and SIGSEGVs.