JDK-5101128 : (cs) CoderResult.isOverflow() returns false when UTF-16 overflows
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio.charsets
  • Affected Version: 6
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: windows_xp
  • CPU: x86
  • Submitted: 2004-09-13
  • Updated: 2004-09-24
  • Resolved: 2004-09-24
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other JDK 6
5.0u3Fixed 6 mustangFixed
Related Reports
Relates :  
Description

Name: rmT116609			Date: 09/13/2004


FULL PRODUCT VERSION :
java version "1.4.2_05"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04)
Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode)

java version "1.5.0-beta"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-beta-b32c)
Java HotSpot(TM) Client VM (build 1.5.0-beta-b32c, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]

A DESCRIPTION OF THE PROBLEM :
While decoding UTF-16 streams when CoderResult coderResult = decoder.decode(in, out, false) is called and if the out buffer is indeed smaller than than needed, the coderResult.isOverflow() returns false while it should return true. OTOH,  out.hasRemaining() returns false thus implying that buffer has overflowed.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Run the given program herewith. It returns the correct result for UTF-8 while yields incorrect value for UTF-16


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Encoding            UTF-8
out.hasRemaining()  false
result.isOverFlow() true
Encoding            UTF-16
out.hasRemaining()  false
result.isOverFlow() true
ACTUAL -
Encoding            UTF-8
out.hasRemaining()  false
result.isOverFlow() true
Encoding            UTF-16
out.hasRemaining()  false
result.isOverFlow() false

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
/**
 * @author Chandra Patni
 * @version 1.0
 */
public class JDKBug {
  public static void main(String[] args) throws Exception {
    //create a long enough string.. text contains 177
    String text = "Java technology is a " +
      "portfolio of products that are based on the power of " +
      "networks and the idea that the same software should run " +
      "on many different kinds of systems and devices.";
    decode(text, "UTF-8"); // ok
    decode(text, "UTF-16"); // buggy
  }
  public static void decode(String text, String encoding) throws Exception {
    System.out.println("Encoding            "+encoding);
    byte buffer[] = text.getBytes(encoding);
    ByteBuffer in = ByteBuffer.wrap(buffer);
    CharBuffer out = CharBuffer.allocate(128); //allocate a buffer < 177
    Charset utf16 = Charset.forName(encoding);
    CharsetDecoder decoder = utf16.newDecoder();
    CoderResult result = decoder.decode(in, out, true);
    System.out.println("out.hasRemaining()  "+out.hasRemaining());
    System.out.println("result.isOverFlow() " + result.isOverflow());
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Use out.hasRemaining() instead of CoderResult.isOverflow()
boolean overFlow = coderResult.isOverflow();
if("UTF-16".equalsIgnoreCase(charset.name())) {
  //if there is nothing more to add to "out" buffer
  //it means it has overflowed.
  overFlow = !out.hasRemaining();
}


The fix for this should go into sun.nio.cs.UnicodeDecoder class

94		if (!dst.hasRemaining())
95		    return CoderResult.UNDERFLOW;

Should return CoderResult.OVERFLOW
(Incident Review ID: 305705) 
======================================================================

Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: dragon mustang FIXED IN: mustang INTEGRATED IN: mustang
25-09-2004

SUGGESTED FIX --- /u/martin/ws/mustang/src/share/classes/sun/nio/cs/UnicodeDecoder.java 2004-08-27 16:00:24.296824000 -0700 +++ /u/martin/ws/unicode/src/share/classes/sun/nio/cs/UnicodeDecoder.java 2004-09-16 21:22:18.277363000 -0700 @@ -92,7 +92,7 @@ } if (!dst.hasRemaining()) - return CoderResult.UNDERFLOW; + return CoderResult.OVERFLOW; mark += 2; dst.put(c);
25-09-2004

PUBLIC COMMENTS (cs) CoderResult.isOverflow() returns false when UTF-16, UTF-16LE, UTF-16BE overflows
25-09-2004

EVALUATION The submitter's diagnosis is correct. All the UTF-* charsets are affected. Here is a more complete test case: ------------------------------------------------------------- import java.util.*; import java.nio.*; import java.nio.charset.*; public class Bug { public static void main(String[] args) throws Exception { //create a long enough string.. text contains 177 String text = "Java technology is a ..."; Iterator it = Charset.availableCharsets().entrySet().iterator(); while (it.hasNext()) { Map.Entry e = (Map.Entry) it.next(); String csn = (String) e.getKey(); Charset cs = (Charset) e.getValue(); if (cs.canEncode()) try {test(text, csn);} catch (Throwable t) {t.printStackTrace();} } } public static void test(String text, String encoding) throws Exception { Charset cs = Charset.forName(encoding); if (! cs.canEncode()) return; CharsetEncoder encoder = cs.newEncoder(); CharsetDecoder decoder = cs.newDecoder(); if (! cs.newEncoder().canEncode('.')) return; ByteBuffer in = ByteBuffer.wrap(text.getBytes(encoding)); CharBuffer out = CharBuffer.allocate(text.length()/2); CoderResult result = decoder.decode(in, out, true); if (out.hasRemaining() || ! result.isOverflow()) System.out.println("out.hasRemaining()=" + out.hasRemaining() + " result.isOverflow()=" + result.isOverflow() + " " + encoding); } } ------------------------------------------------------------- ###@###.### 2004-09-16
16-09-2004