JDK-4457851 : (cs) JCK13a api/java_io/CharacterEncoding/primarySBCS.html#ISO_8859_1 Solaris sp
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.io
  • Affected Version: 1.4.0
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS:
    generic,linux,solaris,solaris_2.6,solaris_7,solaris_8,windows_nt,windows_2000 generic,linux,solaris,solaris_2.6,solaris_7,solaris_8,windows_nt,windows_2000
  • CPU: generic,x86,sparc
  • Submitted: 2001-05-11
  • Updated: 2001-10-09
  • Resolved: 2001-10-03
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
1.4.0 beta3Fixed
Related Reports
Duplicate :  
JDK-4402765 - (cs) api/java_lang/String/i18n.html#i18nCtorEncodings, Merlin, Sparc, B46
Duplicate :  
JDK-4402886 - (cs) JCK13: api/java_io/PrintWriter/index.html --> PrWWriter1,3,4,5,Merlin,Sparc
Duplicate :  
JDK-4408438 - (cs) Strings with German characters will not compile on Merlin-beta build 49
Duplicate :  
JDK-4463205 - (cs) jdk regression:OutputStreamWriter.write incorrectly writes valid chars
Duplicate :  
JDK-4482116 - (cs) outputstream prints wrong values for non-ascii characters
Duplicate :  
JDK-4407455 - (cs) JCK13a :api/java_io/FileReader ,Merlin-b48 ,Solaris
Duplicate :  
JDK-4417760 - (cs) Reg-test:sun/io/CharToByteConverter/MalformedSurrogates threw an exception
Duplicate :  
JDK-4424300 - (cs) jdk1.4.0beta-b54 java.lang.String: getBytes("ASCII") returns wrong array of
Duplicate :  
JDK-4424392 - (cs) CharsetEncoder.encode() misses char after nonmapped one
Duplicate :  
JDK-4464220 - (cs) print_chars_02_x (pm) test throws Unexpected Exception
Relates :  
JDK-4503732 - (cs) Charset API redesign
Description
JDK            : 1.4.0-beta-b64  ( passed using jdk1.4-b63)
JCK            : jck1.3a
Platform[s]    : Solaris Sparc/x86
switch/Mode    : all switches 
JCK test owner : http://javaweb.eng/jct/sqe/JCK-tck/usr/owners.jto
Failing Test   : 

 api/java_io/CharacterEncoding/primarySBCS.html#ISO_8859_1


Test source location:
=====================
/net/jdk/export/disk8/local.java/jck1.3a/JCK-runtime-13a/tests/api/java_io/CharacterEncoding/TestMap.java Test8859_1.java

jtr file location:
==================
file:///net/jtgb4u4c/export/sail16/results/merlin/b64/jck13a/sparc/s6_client_jtg-s116/workDir/api/java_io/CharacterEncoding/primarySBCS_ISO_8859_1.jtr

jtr also attached.

How to reproduce:
====================

run the attched scritped.
   
   io_CharacterEncoding.ksh
--------Script START---------------------
#!/bin/ksh
SWITCH=${1+$@}
JDK=/net/jdk/export/disk8/local.java/jdk1.4
TESTBASE=/net/jdk/export/disk8/local.java/jck1.3a
JCK=${TESTBASE}/JCK-runtime-13a
executeClass="javasoft.sqe.tests.api.java.io.CharacterEncoding.Test8859_1"
excludeCmd=
executeClassArgs="-Encoding 8859_1"
executeTestURL=
headless=
#executeContextArgs is used for vm testing.
executeContextArgs=

case `uname -s` in
  SunOS)
   ARCH=`uname -p`
   case $ARCH in
    sparc)
      sharedJDK=$JDK/solsparc
      ;;
    i386)
      sharedJDK=$JDK/solx86
      ;;
   esac
   ;;
  Linux)
   sharedJDK=$JDK/linux-i386
   ARCH=linux
   ;;
esac

CLASSPATH=${JCK}/classes:${JCK}/javatest.jar
DISPLAY=${DISPLAY-$HOST:0.0}

if `echo $SWITCH|grep "\-d64" >/dev/null`; then
  LD_LIBRARY_PATH=${JCK}/lib/sparcv9
else
  LD_LIBRARY_PATH=${JCK}/lib/${ARCH}
fi


PATH=$JDK/bin:$sharedJDK/bin:$PATH

export PATH CLASSPATH DISPLAY LD_LIBRARY_PATH


echo
java ${SWITCH} -version
echo
java ${SWITCH} -verify -Xfuture -Djava.security.policy=${JCK}/lib/jck.policy -Djava.rmi.activation.port=${executeClass} -Djava.awt.headless=${headless} ${executeClass} ${excludeCmd}  ${executeClassArgs} ${executeContextArgs} ${executeTestURL} 
--------Script END----------------------

Test output:
=============

ByteConverterLoad: Passed. OKAY
TestByteMapping: Passed. OKAY
LoadCharToByte: Passed. OKAY
TestNoSubMapping: Passed. OKAY
TestUnmappable: Failed. Incorrect number of bytes written: 4
TestHighSurrogate: Passed. OKAY
TestLowSurrogate: Passed. OKAY
TestSurrogatePair: Passed. Incorrect number of byte written: 2
STATUS:Failed.tests: 8; passed: 7; failed: 1; first test case failure: TestUnmappable


Specific Machine Info:
=====================

SunOS jtg-s120 5.8 Generic sun4u sparc SUNW,Ultra-60

SunOS jtg-i112 5.8 Generic i86pc i386 i86pc




takwing.cheung@Eng 2001-05-11
Comments
CONVERTED DATA BugTraq+ Release Management Values COMMIT TO FIX: generic FIXED IN: merlin-beta3 INTEGRATED IN: merlin-beta3 VERIFIED IN: merlin-beta3
14-06-2004
EVALUATION From duplicate bug 4463205: This is a duplicate of 4424392. java.nio.charset.CharsetEncoder always skips ahead by at least a single char when a non-mapped character is encountered on the input stream. The result is lost data. Ian.Little@Ireland 6/6/2001 The following jck tests are impacted by this bug: jck1.3: api/java_lang/String/i18n.html#i18nCtorEncoding[String2211] - 4402765, 4424300 jck1.3a: api/java_io/FileReader/index.html#FRRead1Test[Read1Test0007] - 4407455 api/java_io/FileReader/index.html#FRRead3Test[Read3Test0007] - 4407455 The following tests are taken from the various bugs that are essentially caused by the same problem. Theyshould be representative of every varient of this bug (unmappable character vs. surrogate pair, one character skipped vs. multiple characters skipped, etc.). // Watered down failing jck test for unmappable characters. Note that // '\ufff0' is an unmappable character (not a high surrogate) but two // characters are consumed. // OUTPUT // // jdk1.3 fcs: // source chars: // \u41 \u42 \ufff0 \u61 \u62 // written bytes: // 0x41 0x42 0x3f 0x61 0x62 // OKAY // // jdk1.4-beta_refresh-b71: // source chars: // \u41 \u42 \ufff0 \u61 \u62 // written bytes: // 0x41 0x42 0x3f 0x62 // Exception in thread "main" java.lang.Exception: Incorrect number of bytes written: 4 // at Unmappable.main(Unmappable.java:36) // // this test covers the case reported in 4464220 /** * @test %W% %E% * @bug 4457851 */ import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStreamWriter; import java.io.UnsupportedEncodingException; public class Unmappable { private static final String encoding = "8859_1"; private static final char chars[] = { 'A', 'B', '\uFFF0', 'a', 'b' }; public static void main(String [] args) throws Exception { byte bytes[] = null; System.out.println("source chars:"); for (int i = 0; i < chars.length; i++) System.out.print(" \\u" + Integer.toHexString((int) chars[i])); System.out.println(); ByteArrayOutputStream baos = new ByteArrayOutputStream(); OutputStreamWriter writer = new OutputStreamWriter(baos, encoding); writer.write(chars); writer.flush(); bytes = baos.toByteArray(); System.out.println("written bytes:"); for (int i = 0; i < bytes.length; i++) System.out.print(" 0x" + Integer.toHexString((int) bytes[i])); System.out.println(); if (bytes.length < chars.length) throw new Exception("Incorrect number of bytes written: " + bytes.length ); /* If the conversion worked, and the resulting byte array is at least * the same length as the character array, then * the test passed. */ System.out.println("OKAY"); } } // Watered down failing jck test for vallid surrogate pairs. In jdk1.4, is // appears that substitution occurs for the surrogate pair and the character // following the pair is skipped. // // OUTPUT // // jdk1.3 fcs: // source chars: // \u61 \ud8ac \udcff \u78 \u79 // written bytes: // 0x61 0x3f 0x78 0x79 // OKAY // // jdk1.4-beta_refresh-b71: // source chars: // \u61 \ud8ac \udcff \u78 \u79 // written bytes: // 0x61 0x3f 0x79 // Exception in thread "main" java.lang.Exception: Incorrect number of byte written: 3 // at SurrogatePair.main(SurrogatePair.java:40) /** * @test %W% %E% * @bug 4457851 */ import java.io.ByteArrayOutputStream; import java.io.IOException; import java.io.OutputStreamWriter; import java.io.UnsupportedEncodingException; public class SurrogatePair { private static final String encoding = "8859_1"; private static final char chars[] = { 'a', '\uD8AC', '\uDCFF', 'x', 'y' }; public static void main(String [] args) throws Exception { byte bytes[] = null; System.out.println("source chars:"); for (int i = 0; i < chars.length; i++) System.out.print(" \\u" + Integer.toHexString((int) chars[i])); System.out.println(); ByteArrayOutputStream baos = new ByteArrayOutputStream(); OutputStreamWriter writer = new OutputStreamWriter(baos, encoding); writer.write(chars); writer.flush(); bytes = baos.toByteArray(); System.out.println("written bytes:"); for (int i = 0; i < bytes.length; i++) System.out.print(" 0x" + Integer.toHexString((int) bytes[i])); System.out.println(); /* Assume that if no exception was thrown, and the size of bytes is at * least 4, then the test passed. */ if (bytes.length >= 4) System.out.println("OKAY"); else throw new Exception("Incorrect number of byte written: " + bytes.length); } } // test from 4424392 // // Note these two subsequences in testString "\u07ffB" and "\uffffD" are not // surrogate pairs since '\u07ff' and '\uffff' do not fall into the range of // high surrogates. The following unicode character is consumed as if // they are incorrectly interpreted as high surrogates. // // OUTPUT (jdk1.4-beta_refresh-b71): // // Charset: ISO-8859-1 // Source string chars: \u41 \u7ff \u42 \u43 \uffff \u44 \u45 // Bytes after encoding: 0x41 0x3f 0x43 0x3f 0x45 // (0x3f corresponds to '?' which is used for a substitution of a single // character) /** * @test %W% %E% * @bug 4457851 */ import java.nio.*; import java.nio.charset.*; public class SurrogateSubstitution { final static String charsetName = "ISO-8859-1"; final static String testString = "A\u07ffBC\uffffDE"; public static void main(String args[]) throws Exception { Charset cs = Charset.forName(charsetName); CharsetEncoder encoder = cs.newEncoder(); System.out.println("Charset: " + cs.name()); byte[] bytes = new byte[64]; ByteBuffer bb = ByteBuffer.wrap(bytes); System.out.print("Source string chars:" ); char[] chars = testString.toCharArray(); for (int i = 0; i < chars.length; i++) { System.out.print(" \\u" + Integer.toHexString((int)chars[i])); } System.out.println(""); CharBuffer cb = CharBuffer.wrap(testString); while (cb.hasRemaining()) { encoder.encode(cb, bb, false); } encoder.flush(bb); bb.flip(); int n = bb.remaining(); System.out.print("Bytes after encoding:"); for (int i = 0; i < n; i++) { System.out.print(" 0x" + Integer.toHexString((int)bytes[i])); } System.out.println(""); bb.clear(); // XXX insert check here to detect expected number of bytes if (n != ???) throw new Exception("expected: " + ??? + " found: " + n); } } /** * @test %W% %E% * @bug 4457851 */ // OUTPUT // $ jdk13; javac -encoding 8859_1 Unmappable.java && java Unmappable // Unmappable.java:2: unclosed string literal // String s = "?, ?, ? // ^ // Unmappable.java:2: ';' expected // String s = "?, ?, ? // ^ // 2 errors // $ jdk14; javac -encoding 8859_1 Unmappable.java && java Unmappable // Unmappable.java:2: unclosed string literal // String s = "??? ^ // Unmappable.java:2: ';' expected // String s = "??? ^ // 2 errors // A similar program without the obvious bug also fails as the '"' is skipped. public class Unmappable { String s = "��, ��, �� } /** * @test %W% %E% * @bug 4457851 */ // from regression test sun/io/CharToByteConverter/MalformedSurrogates import java.io.*; public class Skip { public static void main(String[] args) throws Exception { System.out.println("Testing string conversion..."); // 1 char unrecognized, 'e' (and other unmappable chars) skipped // String t = "abc\uD800\uDB00\uD800\uDB00efghijklm"; // 1 char unrecognized, "ef" (and other unmappable chars) skipped String t = "abc\uD800\uDB00efghijklm"; String t2 = t.substring(2); byte[] b = t2.getBytes(); System.out.println("number of bytes found: " + b.length); for (int i = 0; i < b.length; i++) System.out.print(" 0x" + Integer.toHexString((int) b[i]) + "(" + (char) b[i] + ")"); System.out.println(); if (b.length != 11) { // XXX need to put in correct length for string throw new Exception("Bad string conversion for bad surrogate"); } } } iag@eng 2001-07-19 More jck1.3 failures: api/java_io/PrintWriter/index.html#PrWWrite1Test[Write1Test0002] - 4402886 api/java_io/PrintWriter/index.html#PrWWrite3Test[Write3Test0001] - 4402886 api/java_io/PrintWriter/index.html#PrWWrite4Test[Write4Test0001] - 4402886 api/java_io/PrintWriter/index.html#PrWWrite5Test[Write5Test0002] - 4402886 madbot@eng's distilled test for these failures: /** * @test %W% %E% * @bug 4457851 */ import java.io.*; public class Blah { private static PrintWriter pw = null; private static ByteArrayOutputStream baos = null; public static void main(String[] args) throws Exception { baos = new ByteArrayOutputStream(); pw = new PrintWriter(baos); char[] chars = { (char)0, (char)32, (char)65, (char)127,(char)128, (char)255 }; writeTest(chars, chars.length); } private static void writeTest(char[] streamChars, int num) throws Exception { pw.write(streamChars, 0, num); if (pw.checkError()) throw new RuntimeException("FAILED: checkError()"); String expectedText = new String(streamChars, 0, num); String outText = baos.toString(); if (outText.length() != expectedText.length()) throw new RuntimeException("FAILED: " + outText.length()); } } iag@eng 2001-07-20 Fixed as part of the Charset API redesign (4503732). -- ###@###.### 2001/10/3
20-07-2001