In PSMarkSweep::invoke function(hotspot/src/share/vm/gc/parallel/psMarkSweep.cpp module) MarkSweepAlwaysCompactCount temporary set to 'count' variable and restored after function completion(by UIntXFlagSetting flag_setting):
  uint count = maximum_heap_compaction ? 1 : MarkSweepAlwaysCompactCount;
  UIntXFlagSetting flag_setting(MarkSweepAlwaysCompactCount, count);

The problem is that MarkSweepAlwaysCompactCount is an uintx and 64 bits long on 64 bit system and 'count' is uint and 32 bits long. Therefore 'count' can overflow if MarkSweepAlwaysCompactCount is greater than maximum 32 bit uint. For example if MarkSweepAlwaysCompactCount=4294967296, then count will be 0 and after that MarkSweepAlwaysCompactCount is also will be zero. And after that divide by zero error can occur because MarkSweepAlwaysCompactCount is used in denominator.

Here an example:
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGFPE (0x8) at pc=0x00007f28e668c124, pid=10809, tid=10836
#
# JRE version: Java(TM) SE Runtime Environment (9.0) (build 9-internal+0-2015-12-09-140432.dmitry.jdk9-hs-rt-filter)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (9-internal+0-2015-12-09-140432.dmitry.jdk9-hs-rt-filter, mixed mode, tiered, compressed oops, parallel gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x1176124]  PSMarkSweepDecorator::precompact()+0x44
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" (or dumping to /export/local/aurora/sandbox/results/workDir/runtime/CommandLine/OptionsValidation/TestOptionsWithRanges/core.10809)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -XX:+UseParallelGC -XX:-UseParallelOldGC -XX:MarkSweepAlwaysCompactCount=4294967296 optionsvalidation.JVMOptionsUtils

Host: scaaa277, Intel(R) Xeon(R) CPU E5-2690 0 @ 2.90GHz, 32 cores, 252G, Oracle Linux Server release 6.3
Time: Wed Dec  9 20:17:55 2015 UTC elapsed time: 0 seconds (0d 0h 0m 0s)
...

I think that fix is trivial - define 'count' as 'uintx':
  uintx count = maximum_heap_compaction ? 1 : MarkSweepAlwaysCompactCount;

I think that new test is not needed, because this flag will be tested for non-G1 GC mode after fixing JDK-8144578.