On Windows the canonicalize_md.c file contains methods to canonicalize paths in wide character format (e.g. wcanonicalize, wcanonicalizeWithPrefix). These are used by the WinNTFileSystem implementation.

In addition it contains methods to canonicalize paths given as char* (canonicalize, canonicalizeWithPrefix). While canonicalizeWithPrefix doesn't seem to be used at all (any more), the canonicalize method is used in hotspot (classLoader.cpp) and in the native library code of module java.instrument. There's no usage of it in libjava or the java.base module, though.

There's a problem with canonicalize/canonicalizeWithPrefix in its current implementation: As Windows has a ~255 character limitation on paths when using the char* Win32 APIs, the current canonicalize() method checks, if the path is larger than the limit and if yes calls the wide char wcanonicalize() method, which is able to handle longer paths.

The problem of the current canonicalize method is when a relative path is given, which has a length < 255, but corresponds to an absolute path >= 255. In this case the _fullpath() method will return the > 256 char path. When the method then iterates over the parts of the path, the FindFirstFile() is called for the various prefixes of the whole path. If one of these exceeds the 255 char limit, the FindFirstFile() will return with an error ERROR_PATH_NOT_FOUND. The code then assumes the error was because the path does not yet exist, so it stops canonicalizing erroneously.

The proposed fix is to just always forward to the wcanonicalize method, which would also fix other problems with the canonicalize method (e.g. the canonical path gets larger).

canonicalizeWithPrefix has the same problem (and more, since it will never forward to the wide character version). This method shall be removed completely.