JDK-8147002 : [macosx] Arabic character cannot be rendered on MacOS X
  • Type: Bug
  • Component: client-libs
  • Sub-Component: 2d
  • Affected Version: 9
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: x86_64
  • Submitted: 2016-01-13
  • Updated: 2018-02-15
  • Resolved: 2017-02-10
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10 JDK 8 JDK 9
10Fixed 8u162Fixed 9 b158Fixed
Related Reports
Blocks :  
Relates :  
Relates :  
Description
Arabic character alef cannot be rendered on MacOS X
A rectangle is displayed instead.

The behavior affects Java SE 9 only.
Java SE 8 is fine: Java 1.8.0_66 was tried.

The behaviour is strictly reproducible.

Here is a reproducer:

% more FontDrawingTest.java
import javax.swing.*;
import java.awt.*;

public class FontDrawingTest {
    public static void main(String[] args) {
        SwingUtilities.invokeLater(() -> {
            JFrame frame = new JFrame("Test");
            frame.add(new MyComponent());
            frame.setDefaultCloseOperation(WindowConstants.EXIT_ON_CLOSE);
            frame.setSize(200, 200);
            frame.setLocationRelativeTo(null);
            frame.setVisible(true);
        });
    }

    private static class MyComponent extends JComponent {
        private final Font font = new Font("Menlo", Font.ITALIC, 100);
        private final String text = "\u0627"; // Arabic letter alef

        @Override
        protected void paintComponent(Graphics g) {
            if (font.canDisplayUpTo(text) == -1) {
                g.setColor(Color.black);
                g.setFont(font);
                g.drawString(text, 70, 110);
            }
        }
    }
}
Comments
CopyCTFallbackFontAndGlyphForJavaGlyphCode(..) calls ... CTFontRef CTS_CopyCTFallbackFontAndGlyphForUnicode (const AWTFont *font, const UTF16Char *charRef, CGGlyph *glyphRef, int count) { CTFontRef fallback = JRSFontCreateFallbackFontForCharacters((CTFontRef)font->fFont, charRef, count); if (fallback == NULL) The JRS* method in there is surely adding the jre/lib/fonts to the fall back list. This is the piece of information that was missing. We should not need to make the glyph code negative (I was hoping to get away from the JRS magic) but what we can do is add the JDK fonts to the cascade list we get from CoreText .. making sure it is ahead of at least the Apple LastResort font. It may be sufficient to add Lucida Sans Regular since we are in "fall back" mode. No other font has code points that it does not. You won't get an "italic" version, nor a "serif" version but you will get the key thing that matters - alef instead of missing glyph. This code must be resilient against there being no such font (as is the case for OpenJDK), although in that case you will get missing glyph still but that would be the same in OpenJDK 8
08-02-2017

Investigation [adding CtFontCopyFullName(fallback) in CoreText,m#CTS_CopyCTFallbackFontAndGlyphForUnicode] reveals that in jdk9b95 alef character is falling back to LucidaBrightRegular.ttf font present in <builddir>/images/jdk/lib/fonts. I could see the cursive line at the top for alef character while opening LucidaBrightRegular font in fontforge utility, which I could not see in any system fonts, claiming to have alef glyph, shown in the "character viewer" utility. If I remove LucidaBrightRegular, then it falls back to LucidaSansRegular, if that is removed, it falls back to LucidaSansDemibold. If all ttf fonts are removed from jdk/lib/fonts, it falls back to GeezaProInterfaceRegular. Same for jdk8b132 (jdk8GA) In jdk9, the glyphcode being passed to CTS_CopyCTFallbackFontAndGlyphForJavaGlyphCode() is 0 so it is not able to find any fallback and shows [box] When it works (as in b95), the glyphcode being passed to this function is -1575. If I hardcode glyphCode = -1575 in AWTStrike.m#Java_java_sun_font_CStrike_getNativeGlyphOutline() before calling CTS_CopyCTFallbackFontAndGlyphForJavaGlyphCode(), it works for jdk9 too. So, we need to find out why glyphCode 0 is being passed and not -1575.
08-02-2017

While trying to find out which font is used for italic style in working case, I am comparing the working and non-working case and I found the fallback routine is handled by CTS_GetGlyphsAsIntsForCharacters() http://hg.openjdk.java.net/jdk9/client/jdk/file/94038948b70f/src/java.desktop/macosx/native/libawt_lwawt/font/CoreTextSupport.m#l118 For non-working case, this is called 30 times [GlyphLayout.layout()#425->CCompositeGlyphMapper.charToGlyph-> CCompositeGlyphMapper.convertToGLyph->CCharToGlyphMapper.charToGlyph(int)->CCharToGlyphMapper.charToGlyph(char)] with fallback "true" for each case and CTFontGetGlyphsForCharacters() returns glyph= 240 in 26 out of 30case, 735 in 2 case and 935 in remaining 2 case. Now, the unicode value passed is always 1575. And, glyphsAsInts[i] shows -1575 for all 30 cases For working case, this is called 326 times [SUnLayoutEngine.shape()->Font2D.charToGlyph->CCharToGlyphMapper.charToGlyph(int)->CCharToGlyphMapper.charToGlyph(char)] with fallback "true" for each case and CTFontGetGlyphsForCharacters() returns different glyphsid for each case [like glyph=735 with glyphsAsInts=-1575, 734 with glyphsAsInts=-1574, 950 with glyphsAsInts=-65163, 736 with glyphsAsInts=-1576, 956 with glyphsAsInts -65169 etc] and unicode value passed is also different[same as glyphAsInts]
07-02-2017

For Plain, CCompositeGlyphMapper#convertToGlyph(unicode=value 1575) gets the glyph code(909) from slot[4] which is CourierNewPSMT Slot[0] Menlo-Regular Slot[1] Monaco Slot[2] AppleSymbols SLot[3] LucidaGrande Slot[4] CourierNewPSMT Slot[5] Ayuthaya Slot[6] Kailasa Slot[7] HiraginoSans-W3 Slot[8] HiraginoSansGB-W3 Slot[9] AppleSDGothicNeo-Regular Slot[10] PingFangHK-Regular SLot[11] PingFangSC-Regular Slot[12] PingFangTC-Regular Slot[13] kohinoorBangla-Regular Slot[14] kohinoorDevanagiri-Regular Slot[15] GujaratiSangamMN Slot[16] GurnukhiMN Slot[17] KannadaSangamMN Slot[18] KhmerSangamMN Slot[19] LaoSangamMN SLot[20] MalayalamSangamMN Slot[21] MyanmarSangamMN Slot[22] OriyaSangamMN Slot[23] SinhalaSangamMN Slot[24] TamilSangamMN SLot[25] KohinoorTelegu-regular Slot[26] Mshtakan Slot[27] EuphemiaUCAS Slot[28] PlantagenetCherokee SLot[29] AppleColorEmoji For Italic, glyphCode is -ve (-1575) which is not present in any slots so there is missing glyph SLot[0] Menlo-Italic Slot[1] Monaco Slot[2] AppleSymbols Slot[3] LucidaGrande SLot[4] CourierNewPS-ItalicMT SLot[5] Ayuthaya SLot[6] Kailasa SLot[7] HiraginoSans-W3 SLot[8] HiraginoSansGB-W3 SLot[9] AppleSDGothicNeo-Regular SLot[10] PingFangHK-Regular Slot[11] PingFangSC-Regular Slot[12] PingFangTC-Regular Slot[13] kohinoorBangla-Regular Slot[14] kohinoorDevanagiri-Regular Slot[15] GujaratiSangamMN Slot[16] GurnukhiMN Slot[17] KannadaSangamMN Slot[18] KhmerSangamMN Slot[19] LaoSangamMN SLot[20] MalayalamSangamMN Slot[21] MyanmarSangamMN Slot[22] OriyaSangamMN Slot[23] SinhalaSangamMN Slot[24] TamilSangamMN SLot[25] KohinoorTelegu-regular Slot[26] MshtakanOblique Slot[27] EuphemiaUCAS-Italic Slot[28] PlantagenetCherokee SLot[29] AppleColorEmoji
31-01-2017

The test seems to work for "Avenir", "Palatino","GeezaPro","Optima" (same in b95 and b96) fonts although it differs from "Menlo" alef character (when seen in b95). "AquaKana", "PingFang" , "Kohinoor" even though does not show rectangle but it differs in b95 and b96. It seems this rectangle problem is only for "Menlo" font.
31-01-2017

The test works with Font.PLAIN and Font.BOLD. It has problem with Font.ITALIC and Font.BOLD | Font.ITALIC I tried on windows 7 where in both b95 and b96, the arabic character is not displayed, rather a slanted line is shown, even if I change the font to "Arabic Typesetting Regular".
30-01-2017

This is odd : Menlo Regular has the glyph, but it is not mapped by the CMAP. The 0,3 (apple unicode) CMAP skips a whole bunch of code point ranges that exist in the font. It has a morx table, but not any OpenType layout tables. A comment in a name table entry says it is based on Bitstream Vera and Deja Vu. So it seems they mapped only the glyphs they wanted to .. but either accidentally left in, or found it easier to leave in, the unwanted glyphs. So it looks likely the glyph was not coming from Menlo in the first place .. rather some other fallback. More investigation needed.
05-01-2017

The test uses ITALIC. If I switch to PLAIN it renders alef. Menlo Regular has over 700 more glyphs than Menlo Italic but neither font claims to support Arabic so these must be coming from fallbacks. I will have to examine the fallbacks reported to us by Core Graphics/Text for this case. There could also be more to it than that ...
13-01-2016

Java SE 9 b100 fails
13-01-2016

9b86 works fine
13-01-2016