FULL PRODUCT VERSION :
java version "1.6.0_21"
Java(TM) SE Runtime Environment (build 1.6.0_21-b06)
Java HotSpot(TM) 64-Bit Server VM (build 17.0-b16, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux elsoxfappp102 2.6.18-194.3.1.el5 #1 SMP Sun May 2 04:17:42 EDT 2010 x86_64 x86_64 x86_64 GNU/Linux


A DESCRIPTION OF THE PROBLEM :
We constantly get the following error when using JMX:
Dec 25, 2010 7:55:49 PM ClientCommunicatorAdmin restart
WARNING: Failed to restart: java.io.IOException: Illegal state.

I instrumented the method com.sun.jmx.remote.internal.ClientNotifForwarder.preReconnection to print out a debug message when entered and got the following:

19:55:49.714 ClientListenerInfo  [Thread-3] state = 3, beingReconnected = false
19:55:49.722 ClientListenerInfo  [RMI TCP Connection(516)-145.36.209.222] state = 3, beingReconnected = false
19:55:49.726 ClientListenerInfo  [RMI TCP Connection(518)-145.36.209.222] state = 3, beingReconnected = true

Note that three threads enter this code in quick succession (within 12ms). The last thread enters in an illegal state.

In reading the code there is a scenario where the synchronization does the wrong thing if 3 threads attempt to go through the code at the same time. Consider the code in com.sun.jmx.remote.internal.ClientCommunicatorAdmin.restart, the first thread will set the state to RE_CONNECTING and then leave the sychronization block, the second thread will find the state to be RE_CONNECTING and wait on the lock. When thread 1 finishes and sets the state to CONNECTED, then thread 2 can leave the synchronization block - but fails to set the state to RE_CONNECTING because that code is incorrectly in the else branch. Thus thread 2 starts the reconnecting and thread 3 wakes to find the state not RE_CONNECTING so it believes it can safely start the reconnect and it also starts reconnecting. The bad mode is discovered in the preReconnection method.

I cannot provide a reproducible case because I don't know how to cause the connection to close, thus forcing the reconnect. But it is clear that the code will fail if 3 threads try to simultaneously execute the restart method.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Cannot provide a way to reproduce this error.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
Dec 25, 2010 7:55:49 PM ClientCommunicatorAdmin restart
WARNING: Failed to restart: java.io.IOException: Illegal state.


REPRODUCIBILITY :
This bug can be reproduced always.