We are encountering a bug with the charset encoder when attempting to flush it in case of an OVERFLOW. The following scenario reproduces the problem. A testcase is also provided.

(BEA bug id - CR259766)
Create an encoder for ISO-2022-JP. Encode some characters and then flush the encoder into a buffer (ensure that this buffer has insufficient space available). It returns OVERFLOW. An attempt to flush the encoder again with a sufficient buffer space throws the following IllegalStateException -

Exception in thread "main" java.lang.IllegalStateException: Current state =
FLUSHED, new state = FLUSHED
        at
java.nio.charset.CharsetEncoder.throwIllegalStateException(CharsetEncoder.java:941)
        at java.nio.charset.CharsetEncoder.flush(CharsetEncoder.java:630)
        at CharsetEncoderTest.main(CharsetEncoderTest.java:55)

According the javadocs for java.nio.charset.CharsetEncoder -
"If there is insufficient room in the output buffer then it returns CoderResult.OVERFLOW. If this happens then this method must be invoked again, with an output buffer that has more room, in order to complete the current encoding operation."

$ java -version
java version "1.5.0_04"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0_04-b05)
Java HotSpot(TM) Client VM (build 1.5.0_04-b05, mixed mode)


Testcase
---------------------------------------------------------------
import java.nio.ByteBuffer;
import java.nio.CharBuffer;
import java.nio.charset.Charset;
import java.nio.charset.CharsetEncoder;
import java.nio.charset.CoderResult;
import java.nio.charset.CodingErrorAction;

public class CharsetEncoderTest {
    private static final String ENCODING_NAME = "ISO-2022-JP";
    private static final String TESTSTR = "\u65e5\u672c\u8a9e";

    public static void main(String[] args) throws Exception {
        Charset charset = Charset.forName(ENCODING_NAME);
        CharsetEncoder encoder = charset.newEncoder().
                onMalformedInput(CodingErrorAction.REPLACE).
                onUnmappableCharacter(CodingErrorAction.REPLACE);

        CharBuffer cb = CharBuffer.wrap(TESTSTR);
        ByteBuffer bb = ByteBuffer.allocate(1024);
        encoder.encode(cb, bb, true);

        // flush encoder into a very small buffer
        ByteBuffer smallBuffer = ByteBuffer.allocate(1);
        CoderResult result = encoder.flush(smallBuffer);

        // flush should return OVERFLOW
        assert(result == CoderResult.OVERFLOW);

        // allocate big buffer, and attempt to flush again
        ByteBuffer bigBuffer = ByteBuffer.allocate(1024);

        // the following call fails - BUG
        result = encoder.flush(bigBuffer);
    }

}

---------------------------------------------------------------

Compile and run the testcase

    $javac CharsetEncoderTest.java

    $java CharsetEncoderTest