JDK-8177332 : The presence of a file with a Japanese ShiftJIS name can cause javac to fail
  • Type: Bug
  • Component: tools
  • Sub-Component: javac
  • Affected Version: 9
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2017-03-17
  • Updated: 2017-04-18
  • Resolved: 2017-04-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10 JDK 9
10Fixed 9 b165Fixed
Related Reports
Relates :  
JDK-8169069 - Module system implementation refresh (11/2016)
Relates :  
JDK-8177314 - java VM fails to start with a Japanese ShiftJIS locale
Description
FULL PRODUCT VERSION :
$ java -version
java version "9-ea"
Java(TM) SE Runtime Environment (build 9-ea+159)
Java HotSpot(TM) 64-Bit Server VM (build 9-ea+159, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux rhxxxx.microfocus.com 2.6.32-131.0.15.el6.x86_64 #1 SMP Tue May 10 15:42:40 EDT 2011 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
No locale to be set.

A DESCRIPTION OF THE PROBLEM :
If there is a file present in the current directory whose name consists of ShiftJIS characters, and my locale is not set, 'javac HelloWorld.java' (obviously a trivial Java program) fails.

REGRESSION.  Last worked in version 8u121

ADDITIONAL REGRESSION INFORMATION: 
java version "9-ea"
Java(TM) SE Runtime Environment (build 9-ea+159)
Java HotSpot(TM) 64-Bit Server VM (build 9-ea+159, mixed mode)


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
First we must create a file with a Japanese ShiftJIS name.
You need a terminal that supports Japanese ShiftJIS first. I do...

$ LANG=ja_JP.SJIS xterm&

... which creates an xterm. I cannot read or write Japanese, so I then do 'ls notexist' to get an error in Japanese from 'ls', and cut and paste some Japanese characters from that error and use 'touch' to create the file...

$ touch <pasted-Japanese-characters>

There is now a file with a name consisting on ShiftJIS Japanese characters. 
In another terminal (or the Japanese one - it doesn't really matter) unset your language settings, and check your locale - it should now default to 'POSIX'

$ unset LANG
$ unset LC_ALL
$ locale
LANG=
LC_CTYPE="POSIX"
LC_NUMERIC="POSIX"
LC_TIME="POSIX"
LC_COLLATE="POSIX"
LC_MONETARY="POSIX"
LC_MESSAGES="POSIX"
LC_PAPER="POSIX"
LC_NAME="POSIX"
LC_ADDRESS="POSIX"
LC_TELEPHONE="POSIX"
LC_MEASUREMENT="POSIX"
LC_IDENTIFICATION="POSIX"
LC_ALL=

Next we'll be trying to compile 'HelloWorld.java' - here it is...

public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello, World");
    }

}

... now to compile it


$ javac HelloWorld.java


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
HelloWorld.java compiles without any warning or error.
ACTUAL -
$ javac HelloWorld.java
An exception has occurred in the compiler (9-ea). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: ?????????????y
	at java.base/sun.nio.fs.UnixPath.encode(UnixPath.java:145)
	at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
	at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:280)
	at java.base/java.nio.file.Path.resolve(Path.java:402)
	at jdk.compiler/com.sun.tools.javac.file.RelativePath.resolveAgainst(RelativePath.java:60)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$DirectoryContainer.list(JavacFileManager.java:478)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:713)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:722)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:648)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:529)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:293)
	at jdk.compiler/com.sun.tools.javac.code.Symtab.lambda$addRootPackageFor$7(Symtab.java:758)
	at jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:354)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:285)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:300)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:570)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:554)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1073)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:944)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:302)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:162)
	at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:55)
	at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:41)


ERROR MESSAGES/STACK TRACES THAT OCCUR :
$ javac HelloWorld.java
An exception has occurred in the compiler (9-ea). Please file a bug against the Java compiler via the Java bug reporting page (http://bugreport.java.com) after checking the Bug Database (http://bugs.java.com) for duplicates. Include your program and the following diagnostic in your report. Thank you.
java.nio.file.InvalidPathException: Malformed input or input contains unmappable characters: ?????????????y
	at java.base/sun.nio.fs.UnixPath.encode(UnixPath.java:145)
	at java.base/sun.nio.fs.UnixPath.<init>(UnixPath.java:69)
	at java.base/sun.nio.fs.UnixFileSystem.getPath(UnixFileSystem.java:280)
	at java.base/java.nio.file.Path.resolve(Path.java:402)
	at jdk.compiler/com.sun.tools.javac.file.RelativePath.resolveAgainst(RelativePath.java:60)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$DirectoryContainer.list(JavacFileManager.java:478)
	at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:713)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.list(ClassFinder.java:722)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.scanUserPaths(ClassFinder.java:648)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.fillIn(ClassFinder.java:529)
	at jdk.compiler/com.sun.tools.javac.code.ClassFinder.complete(ClassFinder.java:293)
	at jdk.compiler/com.sun.tools.javac.code.Symtab.lambda$addRootPackageFor$7(Symtab.java:758)
	at jdk.compiler/com.sun.tools.javac.code.Symbol.complete(Symbol.java:633)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.visitTopLevel(Enter.java:354)
	at jdk.compiler/com.sun.tools.javac.tree.JCTree$JCCompilationUnit.accept(JCTree.java:529)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:285)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.classEnter(Enter.java:300)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.complete(Enter.java:570)
	at jdk.compiler/com.sun.tools.javac.comp.Enter.main(Enter.java:554)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.enterTrees(JavaCompiler.java:1073)
	at jdk.compiler/com.sun.tools.javac.main.JavaCompiler.compile(JavaCompiler.java:944)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:302)
	at jdk.compiler/com.sun.tools.javac.main.Main.compile(Main.java:162)
	at jdk.compiler/com.sun.tools.javac.Main.compile(Main.java:55)
	at jdk.compiler/com.sun.tools.javac.Main.main(Main.java:41)
$ 


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
public class HelloWorld {

    public static void main(String[] args) {
        System.out.println("Hello, World");
    }

}


---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
If the locale is 'en_GB.UTF8' the compilation works.... and, yep ja_JP.utf8 also. C does not.

$ export LANG=en_GB.UTF8
$ javac HelloWorld.java
Comments
Fix Approved
04-04-2017
David - my comment was just an FYI that we are still trying to figure out if this locale is supported. Nothing to do with files with content in this encoding.
04-04-2017
We haven't established yet whether this is a supported configuration or not, see JDK-8177314.
04-04-2017
These are the most important three lines in the stacktrace: at jdk.compiler/com.sun.tools.javac.file.RelativePath.resolveAgainst(RelativePath.java:60) at jdk.compiler/com.sun.tools.javac.file.JavacFileManager$DirectoryContainer.list(JavacFileManager.java:478) at jdk.compiler/com.sun.tools.javac.file.JavacFileManager.list(JavacFileManager.java:713) If you look at RelativePath.resolveAgainst, the signature is: public Path resolveAgainst(Path directory) throws /*unchecked*/ InvalidPathException { public Path resolveAgainst(FileSystem fs) throws /*unchecked*/ InvalidPathException { What I take from that is that it is deliberate that these methods do not catch IllegalPathException and convert it to IOException. That leaves the other two methods in the trace as possibilities for handling IPE. My inclination is to put it on the more deeply nested method i.e. DirectoryContainer.list, where we have more contextual info for the rethrown exception. if (isValidFile(fname, fileKinds)) { + try { RelativeFile file = new RelativeFile(subdirectory, fname); JavaFileObject fe = PathFileObject.forDirectoryPath(JavacFileManager.this, file.resolveAgainst(directory), userPath, file); resultList.append(fe); + } catch (IllegalPathException e) { + throw new IOException("error accessing directory " + directory. e); + } }
27-03-2017
Regression started in 9 b148. Looks like changes made to ClassFinder.java in the following changeset triggered this regression: 8169069: Module system implementation refresh (11/2016) http://hg.openjdk.java.net/jdk9/jdk9/langtools/rev/d516975e8110 I also believe the new behavior should *probably* be considered a bug. (But this is not my product area...)
21-03-2017
Another related issue JDK-8177314 with Japanese locale, we are unable to reproduce at our end...
21-03-2017