test java/nio/Buffer/LimitDirectMemory.sh tests the use of the -XX:MaxDirectMemorySize flag with the VM. Some invocations are expected to be invalid due to invalid parameter settings ie:

# Various bad values fail to launch the VM.
launchFail foo
launchFail 10kmt
launchFail -1

First bug: -1 is actually the default value for this flag, it is not an illegal value - it means "use the default". This error was not noticed because the "launch" still fails due to bug 2.

Second bug: launchfail is defined as:

launchFail() {
  echo "Testing: -XX:MaxDirectMemorySize=$* -cp ${TESTCLASSES} \
     LimitDirectMemory true DEFAULT DEFAULT+1M"
  ${TESTJAVA}/bin/java -XX:MaxDirectMemorySize=$* -cp ${TESTCLASSES} \
     LimitDirectMemory true DEFAULT DEFAULT+1M > ${TMP1} 2>&1
  cat ${TMP1}
  cat ${TMP1} | grep -s "Unrecognized VM option: \'MaxDirectMemorySize="
  if [ $? -ne 0 ]
    then echo "--- failed as expected"
  else
    echo "--- failed"
    exit 1
  fi
}

There are two problems here:

1. If the grep succeeds the test passes, so we should be checking for $? == 0. Instead we report failure for all invocations that don't output the message. This turns out to be all invocations because ...
2. The error message output by the VM does not match the string being grep'd for.

Expected: 

Unrecognized VM option: 'MaxDirectMemorySize=

Actual:

Unrecognized VM option 'MaxDirectMemorySize=

Note there is no colon. The format of this message has likely changed between VM versions.