JDK-4486307 : (spec) Need to document deviation from standards in Japanese charsets
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio.charsets
  • Affected Version: 1.4.0,5.0
  • Priority: P4
  • Status: Resolved
  • Resolution: Won't Fix
  • OS: generic
  • CPU: generic
  • Submitted: 2001-07-31
  • Updated: 2017-08-09
  • Resolved: 2017-08-09
Related Reports
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
The  SJIS, EUC_JP and JISX0201 converters in the Java platform deviate
from the published standards for a few special cased codepoints. 
The deviation is necessary to overcome and handle issues relating
to yen/overline/tilde characters within the ASCII or JIS roman range
and their logical interpretation within applications.
This needs to be clearly documented within the Java language spec.
The specification requirement was initially recorded in bug 4483377
which records apparent jck failures which occurred when the JCK
io api tests used the published SJIS specification mapping table
from Unicode.org as a means of testing standards conformance for
the Japanese character encodings in the Java platform.

---------carryover info from bug 4483377 --------------------------------------
From 4483377:

Name: ooR10006			Date: 07/24/2001


jdk1.4.0beta-b72's methods:
OutputStreamWriter(ByteArrayOutputStream, encodingName).write(char[]) and
InputStreamReader(ByteArrayInputStream, encodingName).read(char[], int, int)
incorrectly write/read chars/bytes for the encoding "SJIS". 
Specifically, the problems are:

1. The character 0x005C is encoded as (byte)0x5C instead of the 2 byte
sequence (byte)0x81, (byte)0x5F.

2. The byte 0x5C is decoded as char 0x005C instead of 0x00A5.

3. 2 byte sequence (byte)0x81, (byte)0x5F is decoded as char 0xFF3C
instead of char 0x005C.

4. The byte 0x7E is decoded as char 0x007E instead of 0x203E, however
the char 0x203E is correctly encoded as byte 0x7E.

The following test shows this:

import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.InputStreamReader;
import java.io.OutputStreamWriter;
import java.io.IOException;
import java.io.UnsupportedEncodingException;

public class test { 
    public static void main(String[] args){
        byteToChar();
        charToByte();
    }
    static void byteToChar(){
        char[] receivedChars = new char[4];
        byte[] inputBytes = {(byte)0x5c, (byte)0x81, (byte)0x5F, (byte)0x7e};
        char[] expectedChars = {(char)0xA5, (char)0x5C, (char)0x203E};
        try {
            ByteArrayInputStream bais = new ByteArrayInputStream(inputBytes);
            InputStreamReader reader = new InputStreamReader(bais, "SJIS");
            reader.read(receivedChars, 0, receivedChars.length);
            System.out.println("byte sequence for decoding: (byte)0x5C " 
                    + "(byte)0x81, (byte)0x5F, (byte)0x7e");
            System.out.println("decoded chars: " 
                    + "0x" + Integer.toHexString(receivedChars[0]) + ", "
                    + "0x" + Integer.toHexString(receivedChars[1]) + ", "
                    + "0x" + Integer.toHexString(receivedChars[2]));
            System.out.println("expected chars: " 
                    + "0x" + Integer.toHexString(expectedChars[0]) + ", "
                    + "0x" + Integer.toHexString(expectedChars[1]) + ", "
                    + "0x" + Integer.toHexString(expectedChars[2]));
                    
        } catch(UnsupportedEncodingException e) {
            return;
        } catch(IOException ex) {
            return;
        }
    }      
    static void charToByte(){
        char[] inputChars = {(char)0x5C};
        byte[] expectedBytes = {(byte)0x81, (byte)0x5F};
        try {
            ByteArrayOutputStream baos = new ByteArrayOutputStream(10);
            OutputStreamWriter writer = new OutputStreamWriter(baos, "SJIS");
            writer.write(inputChars);
            writer.flush();
            byte[] bytes = baos.toByteArray();
            System.out.println("chars for encoding: (char)0x5C ");
            System.out.println("encoding bytes: " 
                    + "0x" + Integer.toHexString(bytes[0] & 0xFF));
            System.out.println("expected bytes: " 
                    + "0x" + Integer.toHexString(expectedBytes[0] & 0xFF) + ", "
                    + "0x" + Integer.toHexString(expectedBytes[1] & 0xFF));

        } catch (UnsupportedEncodingException e) {
            return;
        } catch (IOException ex) {
            return;
        }
    }
}
% jdk1.4.0beta-b72/solsparc/bin/java test
byte sequence for decoding: (byte)0x5C (byte)0x81, (byte)0x5F, (byte)0x7e
decoded chars: 0x5c, 0xff3c, 0x7e
expected chars: 0xa5, 0x5c, 0x203e
chars for encoding: (char)0x5C 
encoding bytes: 0x5c
expected bytes: 0x81, 0x5f
%

Due to this the following JCK test fails:

api/java_io/mbCharEncoding/index.html#ShiftJIS

The test is correct and actually fails since 1.1.6

======================================================================

Comments
EVALUATION The mapping tables from Unicode.org are not real standards. For Japanese encodings, follow the JIS standards. See 4251698 for the JIS X 0208 specific issue. (The Yen Sign problem is still applicable to the JIS standards. In JIS X 0201, 0x5c is Yen Sign.) masayoshi.okutsu@Eng 2001-08-01
01-08-2001