Test execution depends on the current setting of locale-related environment variables, namely:
LANG
LC_ALL
LC_CTYPE
If all variables are unset, the vm initialization mechanics, as seen by calls to Charset.defaultCharset().name(), default to UTF-8. That charset is obviously handled correctly. The created directory is accessible from the Setup program, the javac compiler, the boot classloader, and from shell commands. Interestingly enough, the directory name which causes the test to fail (see ISO-8859-1 case) is part of the (much longer) name used here.
Setting LANG=en-US.ISO-8859-1 seems to be a special case. The default charset then is ISO-8859-1, selecting a directory name with a Spanish special character. This name can be handled from within the setup program. Shell commands and javac can't see / can't write to this directory, causing the test to fail.
When setting LANG and LC_CTYPE to any other tested codepage of the ISO-8859 family, vm initialization mechanics default the charset to US-ASCII. No special dirname string (with non-ASCII characters) is defined for that charset. It is clear that the test will succeed then.
I'm not sure where to locate the misbehaviour. Is it just a test bug? Or is it an issue in the java/OS interaction? Or just a vm initialization issue?
I could reproduce the effect on MacOS 10.14 and 10.15, using a Java 8 vm as well as an OpenJDK head revision vm. I would assume the issue does not depend on the Java release. I did not test the releases in between, though.
The test did not fail on other platforms (linuxx86_64, linuxs390x, rs6000_64 (AIX)). On Linux, the default encoding always was US-ASCII, introducing no special characters into the directory name. On AIX, however, LANG=en_US.UTF-8 as well as LANG=en_US.ISO-8859-1 resulted in the default encoding to be ISO-8859-1. The test succeeded despite the special character in the directory name.
Attached to this bug are
the test files modified to obtain more debugging output (as individual files and as importable diff)
a jtr file of a successful run on AIX
a jtr file of a successful run on MacOS
a jtr file of a failing run on MacOS
Please contact the author if you need any additional, clarifying information.