JDK-8256372 : [macos] Unexpected symbol was displayed on JTextField with Monospaced font
  • Type: Bug
  • Component: client-libs
  • Sub-Component: 2d
  • Affected Version: 12.0.2,13,14,15,16
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: os_x
  • CPU: x86
  • Submitted: 2020-11-16
  • Updated: 2022-07-26
  • Resolved: 2021-05-20
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 15 JDK 17
11.0.13-oracleFixed 13.0.13Fixed 15.0.9Fixed 17 b24Fixed
Related Reports
Duplicate :  
JDK-8227828 - [macos] Sub/Superscript text is rendered with incorrect spacing between characters
Duplicate :  
JDK-8212744 - [macos] Overlapping glyphs with harfbuzz + scaled AAT fonts
Duplicate :  
JDK-8180613 - Layout is incorrect for rotated text
Relates :  
JDK-8273321 - java/awt/font/GlyphVector/NLGlyphTest.java fails on Linux
Relates :  
JDK-8144012 - [PIT][macosx] Failure of closed/java/awt/FontClass/FontStringBounds.java
Relates :  
JDK-8210782 - Upgrade HarfBuzz to the latest 2.3.1
Relates :  
JDK-8247872 - Upgrade HarfBuzz to the latest 2.7.2
Description
When I used Surrogate Pair character on JTextField on macOS, unexpected symbol (square) was displayed at the end of line.

I used following build.
$ ~/jdk-16.jdk/Contents/Home/bin/java -version
openjdk version "16-ea" 2021-03-16
OpenJDK Runtime Environment (build 16-ea+24-1553)
OpenJDK 64-Bit Server VM (build 16-ea+24-1553, mixed mode, sharing)

Please see attached screen shot and testcase.

According to my investigation, I could recreate this issue after JDK13.
I could not recreate this issue on JDK 12.0.1, but I could recreate it on JDK 12.0.2.
I assume this issue came from HarfBuzz upgrade (JDK-8220392)

Base update might be JDK-8210782.
In these days, HarfBuzz was upgrade by JDK-8247872, it should be in 16-b24.
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/jdk13u-dev/pull/377 Date: 2022-07-26 13:43:52 +0000
26-07-2022
Fix request (15u, 13u): this is a necessary prerequisite for HarfBuzz upgrade backport. There are two context-related conflicts and a rollback of a logging improvement in one place.
26-07-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk15u-dev/pull/240 Date: 2022-07-26 09:31:43 +0000
26-07-2022
Fix Request (11u): Fix was backported to 11.0.13-oracle and it's still missing in OpenJDK 11.0.14. It doesn't apply cleanly. Review: https://github.com/openjdk/jdk11u-dev/pull/317
09-09-2021
Changeset: 005d8a7f Author: Phil Race <prr@openjdk.org> Date: 2021-05-20 19:49:03 +0000 URL: https://git.openjdk.java.net/jdk/commit/005d8a7fca8b4d9519d2bde0a7cdbbece1cd3981
20-05-2021
>> Maybe another possibility is to force the coretext shaper like was implicitly done before the harfbuzz upgrade Or maybe not. In the source I spotted a reference to https://github.com/harfbuzz/harfbuzz/issues/1478 This confirms that now the OT shaper is called for AAT fonts and even fonts created with the HB API to create a CT font, and that to get the coretext shaper you need to force it AND that the plan is to remove it. The isAAT() method and resulting boolean flag passed to native was meant to mean "since it is AAT, create a CT font ptr so that HB can use its coretext shaper". Now that's moot. Whether a font is an AAT font or not does not need to be specified. HB will figure it out. Does this flag even serve a purpose any more ? It doesn't even mean "use CT" unless we forced the shaper they plan to remove ...
17-05-2021
On windows and Linux char-> glyph mapping for OpenType + TrueType fonts is done via Java-level code in sun.font.CMap.java which parses the cmaps of the fonts.. TrueTypeGlyphMapper.java calls this code. For Type1, TypeGlyphMapper calls into freetype code that has consumed the font] on macoS for TrueType the CFont class uses CCharToGlyphMapper which calls down to macOS specific code. And when doing text rendering that does not use layout and no special circumstances apply, we end up calling a standard core text API to map chars to glyphs CTFontGetGlyphsForCharacters((CTFontRef)font->fFont, unicodes, glyphs, count); When doing layout we ordinarily pass to layout call backs that refer to these same internal APIs But on macOS harfbuzz had no native support for AAT fonts so it was using CoreText for layout in such cases and we create and pass a CoreText font handle. We did not supply the call backs, since it should not need the JDK ones, HB 2.3.1 added it's own support for AAT so now when we create a CoreText font it seems to use its own AAT support and obtains and parses the cmap itself since it is no longer calling into CoreText at all (SFAICS). The cmap for Menlo has no mapping for the newline codepoint, so it finds none and returns the missing glyph As to why just Menlo .. well there are surely other fonts that would have the same issue but the primaries (slot 0) for the other logical fonts DO have mappings for control chars so it all works out fine there. Whereas when we call the above mentioned core text API CTFontGetGlyphsForCharacters, we get back a reference to glyph ID 3, which is an invisible glyph (no rendering, no advance). Similarly the JDK's Java-level code has a special function in CMap.java final int getControlCodeGlyph(int charCode, boolean noSurrogates) { if (charCode < 0x0010) { switch (charCode) { case 0x0009: case 0x000a: case 0x000d: return CharToGlyphMapper.INVISIBLE_GLYPH_ID; } } else if (noSurrogates && charCode >= 0xFFFF) { return 0; } return -1; } So we ensure that anyone requesting to render one of these common control chars won't get those nasty boxes. But without harfbuzz having similar logic, once we send the text containing new line off to harfbuzz, then it ends up with that missing glyph. Fonts have multiple CMap subtables and whilst they should all map the same code points to the same glyphs it is not guaranteed and when you call an API like CTFontGetGlyphsForCharacters you can't control which cmap it uses. There are even platform-specific cmap subtables and platform policies. Ideally whether you use layout to render a newline or not you'd get the same result but there's certainly a theoretical possibility you won't but much more immediate is that there's an actual important case. The Swing text edit controls seem to all append this and are incapable of functioning if it is not there. This is a design mistake but not one that is easily rectified. Right now the only thing I can think of is to start to us treat the AAT fonts handed down to harfbuzz like we do the OpenType ones - no need to use the hb API call to create a coretext font - just create it like any other font and supply the call backs to get the info HB is looking up itself. This will then end up in that same CT code we use to map in the non-layout case and so have the same results. Of course we are there dependent on CT to do what it is doing today, but in the event of any further problems with some other font we own the code path so can intercept and adjust what is returned. Maybe another possibility is to force the coretext shaper like was implicitly done before the harfbuzz upgrade, but that is - from where we are now - a bigger change.
17-05-2021
To be more direct you can use Font font = new Font("Menlo", Font.PLAIN, 60); since Menlo is the base font for Monospaced. Also look at Menlo in Font2DTest and toggle to TextLayout.draw and several code points change from rendering as invisible to the missing glyph box.
16-05-2021
Demonstrating the problem can be reduced to this : import java.awt.*; import java.awt.font.*; public class GV { public static void main(String[] args) { Font font = new Font(Font.MONOSPACED, Font.PLAIN, 60); FontRenderContext frc = new FontRenderContext(null, true, true); GlyphVector cgv = font.createGlyphVector(frc, "\n"); char[] chs = { '\n' }; GlyphVector lgv = font.layoutGlyphVector(frc, chs, 0, 1, 0); int c_code = cgv.getGlyphCode(0); int l_code = lgv.getGlyphCode(0); System.out.println("create code="+c_code + " layout code="+l_code); } } % ~/jdk12/Contents/Home/bin/java GV create code=3 layout code=3 % ~/jdk16/Contents/Home/bin/java GV create code=3 layout code=0 Note: These two APIs *can* legitimately return different glyphs even different numbers of glyphs - since the whole point of layout is to transform the text into laid out text whereas create by specificiation just does 1:1 mapping. But in this case it is not so clear why that would happen.
16-05-2021
So .. Swing is really asking for this. The content instance for the editor is initialised with a newline. GapContent is used here but you can see StringContent does the same. public GapContent(int initialLength) { super(Math.max(initialLength,2)); char[] implied = new char[1]; implied[0] = '\n'; replace(0, 0, implied, implied.length); public StringContent(int initialLength) { if (initialLength < 1) { initialLength = 1; } data = new char[initialLength]; data[0] = '\n'; count = 1; } ////// This seems to be necessary because other places in the Swing text code like rendering and caret rendering seem to be unable to handle truly empty content. So far as I can tell this newline is always present at the end and for whatever resaon is part of of the content Swing hands off to LineBreakMeasurer which produces TextLayout instances. The particular text used here is enough to trigger this. So we ALWAYS get back 2 glyphs here. Including in JDK 12 and earlier where it looks fine. But because of whatever reason with the monospaced font and/or the maybe coretext usage by HarfBuzz we end up with glyph id 0 in later JDK's whereas we had glyph ID 3 in earlier JDKs. I suppose that newline was being mapped to that glyph before but now it is mapped to the missing glyph which renders as a box instead of glyph ID 3 which renders as invisible. But the problem isn't just limited to JTextFIeld. If you construct a JTextArea with embedded newlines, and that also triggers layout, then we get missing glyphs for those too. So we need to understand why it is now being mapped to the missing glyph.
14-05-2021
The problem doesn't appear with a JLabel only the edit component. You can't move the caret past it and chars get appended afer the ideograph but before this missing glyph which keeps shifting to the end of the line Deleting the CJK doesn't make it go away whch is interesting. So this looks like it is specifically an issue with Swing editing. I don't know why monospaced specifically but likely because of a different font being used for the ideograph in this case. But I've tried directly using the 3 JA fonts listed in the cascade list for monospaced and the problem does not manifest, so either I did not get that right or part of the trigger is that it be a composite font.
12-05-2021
I can confirm that issue started from jdk13-b14 which has harfbuzz upgraded fix.
18-11-2020
I could recreate this issue with JDK 12.0.1 + JDK-8220392 (JDK-8210782) "Upgrade HarfBuzz to the latest 2.3.1" on Japanese macOS.
18-11-2020
[~itakiguchi] Why did you relate it to JDK-8210782 and JDK-8247872? Do yousee the issue only after this harfbuzz upgrade?
17-11-2020