FULL PRODUCT VERSION :
$ java -version
java version "1.6.0_24"
Java(TM) SE Runtime Environment (build 1.6.0_24-b07)
Java HotSpot(TM) 64-Bit Server VM (build 19.1-b02, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux web1 2.6.24-27-server #1 SMP Wed Mar 24 11:32:39 UTC 2010 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
NativeThreadSet maintains a growable array of thread IDs. When the add() method is called to add another thread id to the internal array, there is a conditional comparison to see if the array should be resized. The problem in the Oracle JDK is that a "greater than" operator is used when it compares the number of slots in use to the total length of the array.

if (used > elts.length) {

Instead, "greater than or equals" should have been used. In OpenJDK, the above line of code is:

if (used >= elts.length) {

Let's say that the elts array is currently of length 2 (this is the default size used by sun.nio.ch.FileChannelImpl) and both array elements contain a thread ID, so used = 2. If add() is called again, the comparison will fail, because used is not greater than elts.length. The code then looks for an element in the array with a value of 0, indicating a free slot. None will be found, so assuming that assertions are enabled, an AssertionError is thrown:

assert false;

and then -1 is returned as the array index for the element in which the thread ID was stored:

return -1;

If the client code stores -1 as the array index and then later passes -1 to the remove() method, remove() will immediately return with no indication of an error.

if (i < 0)
    return;

So, if assertions are not enabled, the thread ID will fail to be stored by NativeThreadSet and the client will likely not know that the thread is not being tracked. If the client uses NativeThreadSet.signal() to try to signal all the threads, it is possible that some threads will not be signaled.

REGRESSION.  Last worked in version 7

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Create a NativeThreadSet object with a fixed size, e.g., new NativeThreadSet(n).

Call add() on the NativeThreadSet n + 1 times

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The nth + 1 call to add() returns n + 1
ACTUAL -
The nth + 1 call to add() returns -1

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Caused by: java.lang.AssertionError
        at sun.nio.ch.NativeThreadSet.add(NativeThreadSet.java:46)
        at sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:610)

REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Turning off assertions stops the AssertionError from being thrown, though, there is, of course, still the possibility that threads are not properly signaled.