JDK-8160939 : TrueType Fonts that use a CJK CMAP encoding are broken by 8008386.
  • Type: Bug
  • Component: client-libs
  • Sub-Component: 2d
  • Affected Version: 7u80,8,9,10
  • Priority: P3
  • Status: Open
  • Resolution: Unresolved
  • OS: windows_7
  • CPU: x86_64
  • Submitted: 2016-07-01
  • Updated: 2018-09-05
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
Relates :  
Description
FULL PRODUCT VERSION :
java version "1.8.0_92"
Java(TM) SE Runtime Environment (build 1.8.0_92-b14)
Java HotSpot(TM) 64-Bit Server VM (build 25.92-b14, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows [Version 6.1.7601] 
(Windows 7 Enterprise Service Pack 1)

A DESCRIPTION OF THE PROBLEM :
Some True Type fonts (YGO13D.ttf and YMjO24ks.ttf) that worked in Java 6 no longer work in Java 8.

REGRESSION.  Last worked in version 6u45

ADDITIONAL REGRESSION INFORMATION: 
java version "1.6.0_45"
Java(TM) SE Runtime Environment (build 1.6.0_45-b06)
Java HotSpot(TM) 64-Bit Server VM (build 20.45-b01, mixed mode)

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1) Install the YGO13D.ttf and YMjO24ks.ttf fonts.
2) Compile the testFont.java program using JDK 1.6
3) Run testFont and get expected results.
4)Compile the testFont.java program using JDK 1.8
4)Run testFont and the fonts do not display. Also the call to "font.canDisplayUpTo(name)" fails.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Console window output:

Serif Plain
Serif Italic
Serif Bold
Serif Bold & Italic
YGO13D Plain
YGO13D Italic
YGO13D Bold
YGO13D Bold & Italic
YMjO24ks Plain
YMjO24ks Italic
YMjO24ks Bold
YMjO24ks Bold & Italic

ACTUAL -
Output:

Serif Plain
Serif Italic
Serif Bold
Serif Bold & Italic
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>
<unsupported character boxes>

ERROR MESSAGES/STACK TRACES THAT OCCUR :
E:\Java\FontTest>echo Java home: c:/Applications/Java/jdk1.8.0_92
Java home: c:/Applications/Java/jdk1.8.0_92

E:\Java\FontTest>javac testFont.java

E:\Java\FontTest>java testFont
There is at least one character in text 'YGO13D Plain' that is not contained in
font YGO13D: Y
There is at least one character in text 'YGO13D Italic' that is not contained in
 font YGO13D: Y
There is at least one character in text 'YGO13D Bold' that is not contained in f
ont YGO13D: Y
There is at least one character in text 'YGO13D Bold & Italic' that is not conta
ined in font YGO13D: Y
There is at least one character in text 'YMjO24ks Plain' that is not contained i
n font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Italic' that is not contained
in font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold' that is not contained in
 font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold & Italic' that is not con
tained in font YMjO24ks: Y
There is at least one character in text 'YGO13D Plain' that is not contained in
font YGO13D: Y
There is at least one character in text 'YGO13D Italic' that is not contained in
 font YGO13D: Y
There is at least one character in text 'YGO13D Bold' that is not contained in f
ont YGO13D: Y
There is at least one character in text 'YGO13D Bold & Italic' that is not conta
ined in font YGO13D: Y
There is at least one character in text 'YMjO24ks Plain' that is not contained i
n font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Italic' that is not contained
in font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold' that is not contained in
 font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold & Italic' that is not con
tained in font YMjO24ks: Y
There is at least one character in text 'YGO13D Plain' that is not contained in
font YGO13D: Y
There is at least one character in text 'YGO13D Italic' that is not contained in
 font YGO13D: Y
There is at least one character in text 'YGO13D Bold' that is not contained in f
ont YGO13D: Y
There is at least one character in text 'YGO13D Bold & Italic' that is not conta
ined in font YGO13D: Y
There is at least one character in text 'YMjO24ks Plain' that is not contained i
n font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Italic' that is not contained
in font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold' that is not contained in
 font YMjO24ks: Y
There is at least one character in text 'YMjO24ks Bold & Italic' that is not con
tained in font YMjO24ks: Y

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
File: testFont.java

import java.awt.*;
import java.awt.event.*;
import javax.swing.*;

public class testFont extends JPanel {
      //String[] type = { "Serif","SansSerif"}; 
      //String[] type = { "Serif","Kredit","Yu Gothic","YGO13D","YMjO24ks"}; 
      String[] type = { "Serif","YGO13D","YMjO24ks"};
      int[] styles = { Font.PLAIN, Font.ITALIC, Font.BOLD,
      Font.ITALIC + Font.BOLD };
      String[] stylenames = 
      { "Plain", "Italic", "Bold", "Bold & Italic" };

      public void paint(Graphics g) {
         for (int f = 0; f < type.length; f++) { 
            for (int s = 0; s < styles.length; s++) { 
               Font font = new Font(type[f], styles[s], 18);
               g.setFont(font); 
               String name = type[f] + " " + stylenames[s];

               int indexUnsupportedCharacter = font.canDisplayUpTo(name);
               if (indexUnsupportedCharacter != -1)
               {
                   System.out.println("There is at least one character in text '" + name + "' that is not contained in font " +    

                    font.getFontName() + ": "+ name.substring(indexUnsupportedCharacter, indexUnsupportedCharacter+1));

               }
               
               g.drawString(name, 20, (f * 4 + s + 1) * 20); 
            }
         }
      }

      public static void main(String[] a) {
         JFrame f = new JFrame();
         f.addWindowListener(new WindowAdapter() {
            public void windowClosing(WindowEvent e) {
               System.exit(0);
            }
         }
      );

      f.setContentPane(new testFont());
      f.setSize(400,400);
      f.setVisible(true);
   }
}


-------------------------------------------
*Command File to compile using JDK 1.6 45:
set JAVA_HOME=c:/Applications/Java/jdk1.6.0_45
set PATH=c:\Applications\Java\jdk1.6.0_45\bin;%PATH%
set INCLUDE=c:\Applications\Java\jdk1.6.0_45\include;%INCLUDE%;
echo Java home: %JAVA_HOME%
javac testFont.java

--------------------------------------------
*Command file to compile using JDK 1.8 92
set JAVA_HOME=c:/Applications/Java/jdk1.8.0_92
set PATH=c:\Applications\Java\jdk1.8.0_92\bin;%PATH%
set INCLUDE=c:\Applications\Java\jdk1.8.0_92\include;%INCLUDE%;
echo Java home: %JAVA_HOME%
javac testFont.java



---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
The only workaround is reverting from JDK 1.8 to JDK 1.6.

SUPPORT :
YES


Comments
The current implementation made for JDK-8008386 in charsets/DoubleByte.java is the appropriate/reasonable solution to handle the situation (where there is an unmappable leading byte). CJK font CMAP encoding handing code has to be updated to handle this change (to probably feed the corresponding charset to build the camp encoding table correctly, don't mix the singlebyte and doublebyte, don't mix unmappble/illegal byte with mappable, for example).
02-11-2017

I don't know how we can get the old and new behaviours simultaneously from the charsets APIs, I'll defer to the charsets owners on that. To be compatible with old + new we'd need to re-write how this mapping is done, but in a way that localises the changes, and does not require embedding knowledge of all the supported valid code points for these encodings. So a reversal of how it is built may be needed. Also testing may require access to fonts use these encodings. Wansung ones are attached to the bug. Another approach might be if the solution can result in the same mapping array to be able to compare these results to expected results generated by JDK 7 GA .. quite how that test should be implemented is TBD. Either accessing internals or written as a stand alone test that is a copy of the internal code ?
13-10-2017

email sent to the bug submitter: ============================================================================================= From the given test case,I have checked this JDK 6,7,8,9ea on Windows 7 but could not reproduce the issue. Steps to reproduce: - Install yandermo.ttf [http://www.1001freefonts.com/yfonts.php] - Run the test case Result : Please find the attached file. Would you please kindly check & confirm if there is any other requirement to reproduce this issue? ==============================================================================================
20-09-2016

This is caused by an incompatible change in behaviour outside the font code. Namely the way character conversion works for multi-byte encodings : https://bugs.openjdk.java.net/browse/JDK-8008386 The updates in the bug to remove "8" as affected version and imply it was introduced in an 8-update are wrong. It was fixed in 8 b115 (pre-GA). It was then backported to 7u80 b11 (again the note that it was introduced in 7u80 b15 is wrong) It was also backported to 6u65 (and maybe BPR builds for earlier updates) so again the claim it can't be reproduced in 6ux is wrong. The reason for all these backports is that the original issue had a support customer escalation. More about what is happening: Just about any TrueType or OpenType font you will encounter these days have a unicode CMAP. But the TT spec allows for several other encodings usually seen only in older CJK fonts. Unicode was not at all widely used when the spec. was created so this was necessary. In our case the font has a 3,5,4 encoding where the 5 means "Wansung" - In order to map a request by a Java application to render a unicode character, using most fonts the Java font system has no special translation to do since both use unicode. But in the case of this font the font code needs to convert this character into that encoding in order to be able to request the correct character. The font system depends on the built-in Java platform character conversion APIs to do this. For the most part this means using the java.nio.charset APIS. Given a unicode code point, you would lookup/find the needed charset encoder and invoke it to encode the character. In order to do this efficiently the font code has a list of the encodings used in TrueType fonts and the range of encoded characters in that encoding. It then maps these *into* unicode. Now it uses that to build a reverse mapping of unicode->charset, cached in an array. Then when the font is asked to map a unicode char it can just consult that array. EUC-KR is a double byte encoding meaning that it does not encode ASCII characters which are mapped as single bytes. Previously if you requested a pair of bytes to be mapped to unicode they were treated as a pair. After the bug fix if the first byte is "ascii" it is replaced as unmappable and something similar will happen to the next byte. Now the number of unicode chars in the output does not match what we expected .. and the reverse mapping table is all messed up. This change in behaviour can be demonstrated by extracting the relevant code from the font implementation and creating a short standalone program that uses *entirely* supported 1.4 and later public API. It does one thing in JDK 1.4->1.7 and another in JDK 8 and later. So regardless of whether the 1.4->1.7 behaviour was "wrong" it is a fairly significant incompatible behavioural change in 8, outside the font code, which is just a client of this code. We would need to rework the way the font code does the conversion but we would not want to do this unless we were sure there was no other change or fix coming to the char converter code that enables backwards compatibility in a way that can be backported. Also this will be tricky to test for all the encodings because such fonts are now rare .. as evidenced by the fact this has apparently gone unreported for some time .. and the code has been left alone as much as possible.
18-07-2016

@Prahalad, 2a. if the font is not installed, then a substitution font is automatically provided. 2b. yes, the boxes usually indicate a missing glyph.
13-07-2016

Quick Observation From Testing The Issue: 1. The test creates a font with new Font ( "face-name", "style", and point size); & uses the font for a string rendering. 2. I tested the issue with JDK-7 GA and JDK-8 GA b132 on Windows 7. a. When the submitter provided fonts aren't installed, it appears like a fallback font is getting used. b. With the submitter provided fonts installed in windows font directory, we do see black boxes (missing glyph?) appearing with Jdk 8 GA but working fine on Jdk 7 GA.
11-07-2016

does it affect 9 ?
07-07-2016

This issue is reproducible from JDK 7u80 onward. so, it looks like a bug to me. Therefore,moving this up for dev.
07-07-2016

Based upon the bug submitter's instructions, checked this for 6u45,8u92,9ea on windows 7 and could confirm the issue as reported by the submitter. Steps to reproduce : ******************** - Install the attached font(YGO130D.ttf and ymj240.ttf) - Run the attached test case(testFont.java) with JDk Result: ******* OS: Windows 7 64 bit JDK: 6u45 b132 b06 : Pass 7 b147 : Pass 7u51 b14 : Pass 7u71 b14 : Pass 7u76 b13 : Pass 7u79 b15 : Pass =>7u80 b15 : Fail <======// Introduced version 8 b132: Fail 8u92 b31 : Fail 9ea+124 : Fail
07-07-2016

Bug Submitter's reply: ============================================================================================================================= Please try with the attached fonts, YGO130D.ttf and ymj240.ttf. These fonts were the specific ones that do not work anymore. I have attached my test results. =============================================================================================================================
07-07-2016

Checked this for 8, 8u92, 9ea on Windows 7 and could not able to reproduce this issue. Steps to reproduce: ************************* Run the attched test case(testFont.java) with JDK. Result: ********* OS : Windows 7 64 bit JDK : ******* 6u45 b06 : Pass 8 b132 : Pass 8u92 b31 : Pass 9ea+b125 : Pass
04-07-2016