Bug ID: JDK-8072130 java/lang/instrument/BootClassPath/BootClassPathTest.sh fails on Mac OSX

JDK 12
12 b13Fixed

This test started failing for the same CNFE "AgentSupport" reason on JDK 18 due to changes made by JDK-8260265. See JDK-8273188. JDK-8260265 does not appear to have been backported yet, but I thought I'd still point it out because JDK-8273188 findings might prove useful here.
08-09-2021
URL: http://hg.openjdk.java.net/jdk/jdk/rev/3c6d285c8168 User: bchristi Date: 2018-09-24 17:42:55 +0000
24-09-2018
The problem boils down to different decisions being made about the default/platform encoding in the standard java main() environment, versus the jvmti/OnLoad/premain() environment, under particular conditions on Mac. In the test, Setup.java sees a defulaultEncoding[1] of UTF-8 (based on the file.encoding system property), and constructs the full 'unicode' Boot-Class-Path manifest entry for Agent.jar. When Agent.jar is OnLoad-ed, Boot-Class-Path is read (as a UTF-8 string), and converted to the native platform encoding, which comes up as US-ASCII (based, I believe, on env var settings). 'iconvToPlatform' is set to convert UTF-8 to US-ASCII[2], but the Boot-Class-Path contains (many) characters that can’t convert to US-ASCII, so iconv() fails (errno says: "Illegal byte sequence"). appendBootClassPath() continues[3] without calling AddToBootstrapClassLoaderSearch. No path on bootclasspath -> ClassNotFoundException. The discrepancy in determining the encoding was introduced in 8 (backported to 7u60) with the fix for JDK-8011194 ("Apps launched via double-clicked .jars have file.encoding value of US-ASCII on Mac OS X”) [4]. In order for files to be displayed correctly in double-clicked apps under non-ASCII languages, if the native encoding is reported as US-ASCII, but no values are set for LANG/LC_ALL/LC_CTYPE, we use UTF-8 instead, as it's known to be supported on Mac. This adjustment is not made when OnLoad-ing an agent jar. One of the main things that made this difficult to reproduce is that the 8011194 fix looks specifically for env vars which do not have any value set (getenv(...) == NULL). I now know this is not the same as being set to an empty value. In Bourne-style shells, I've always used "export VARNAME=" to clear vars, which has always done what I needed up until now. The real way to clear/remove a var is 'unset'. This is why I couldn't reproduce this bug, even when logged into the test machine and setting up seemingly matching 'locale' output. It didn't help that an empty value and no/NULL value look the same in the 'locale' output, or that what 'locale' reports sometimes doesn't match the env values (e.g. if LANG and LC_ALL are set to empty values, 'locale' reports LC_ALL as having a value of "C"). To reproduce this bug reliably, one should confirm (using unset!) that LANG, LC_ALL, and LC_CTYPE are not listed by 'env’. In my view, there are a couple things worth fixing here: 1. On Mac, the Agent OnLoad-ing environment should default to UTF-8 under the same conditions as ParseLocale() in java_props_md.c. 2. I would like to see ParseLocale() also check LANG/LC_ALL/LC_CTYPE for an empty value (""), as it communicates just as little information as the var being set to NULL. We might also consider hard-coding a UTF-8-capable locale in the BootClassPathTest.sh script on Mac - it would ensure that the test is properly exercised. But then again, leaving it as is would expose the test to running in more environments (such as the one that uncovered this problem). -- 1. http://hg.openjdk.java.net/jdk/jdk/file/0517bd2a0eda/test/jdk/java/lang/instrument/BootClassPath/Setup.java#l98 2. http://hg.openjdk.java.net/jdk/jdk/file/0517bd2a0eda/src/java.instrument/unix/native/libinstrument/EncodingSupport_md.c#l87 3. http://hg.openjdk.java.net/jdk/jdk/file/0517bd2a0eda/src/java.instrument/share/native/libinstrument/InvocationAdapter.c#l873 4. http://hg.openjdk.java.net/jdk8/jdk8/jdk/rev/883cc296ec89
20-08-2018
The above code (sun_jnu_encoding = UTF-8 on Mac) has been introduced with the fix to JDK-8003228. Brent, would you please take a look?
25-04-2018
Assigning this to core-libs because this is an issue with JDK handling of locales on Mac OSX, or a bug in the test, or an issue with the Mach5 locale setup of its Mac OSX machines. It's not a VM bug. The VM correctly handles UTF-8 unicode directory names specified using -Xbootclasspath/a. The javap issue discussed above is javap-specific and not related to the VM handling of unicode directory names.
23-04-2018
The test also passes on Mach5 Mac OSX machines if the following is added to BootClassPathTest.sh: export LC_ALL="C" This changes the value of LC_ALL from "" to "C". This also works: export LC_ALL=$LANG
20-04-2018
The test passes with Mach5 on Mac OSX if the following is added to BootClassPathTest.sh: export LC_ALL="en_US.UTF-8" This changes the locale on the Mach5 Mac OSX machines from "C" to "en_US.UTF-8" and changes LC_ALL="" to LC_ALL="en_US.UTF-8". ==================================================================== It looks like java hard-wires UTF-8 for Macs: In GetJavaProperties() in java.base/unix/native/libjava/java_props_md.c: #ifdef MACOSX sprops.sun_jnu_encoding = "UTF-8"; #else sprops.sun_jnu_encoding = sprops.encoding; #endif In Java_java_lang_System_initProperties() in java.base/share/native/libjava/System.c: #ifdef MACOSX /* * Since sun_jnu_encoding is now hard-coded to UTF-8 on Mac, we don't * want to use it to overwrite file.encoding */ PUTPROP(props, "file.encoding", sprops->encoding); #else PUTPROP(props, "file.encoding", sprops->sun_jnu_encoding); #endif
20-04-2018
This is from JavapTask.java: private JavaFileObject getClassFileObject(String className) throws IOException { try { JavaFileObject fo; if (moduleLocation != null) { fo = fileManager.getJavaFileForInput(moduleLocation, className, JavaFileObject.Kind.CLASS); } else { fo = fileManager.getJavaFileForInput(StandardLocation.PLATFORM_CLASS_PATH, className, JavaFileObject.Kind.CLASS); if (fo == null) fo = fileManager.getJavaFileForInput(StandardLocation.CLASS_PATH, className, JavaFileObject.Kind.CLASS); } return fo; } catch (IllegalArgumentException e) { return null; } }
18-04-2018
I wrote a small test that calls Class.forName("AgentSupport") and ran it with -Xbootclasspath/a:bootXXX on Mac. The forName() call found the AgentSupport class running with both JDK-8 and JDK-11. Also, -J-Xlog:class+path shows that the boot class loader's class path is getting set correctly in the javap case described above, even though AgentSupport is not found in that case. Need to figure out how javap tries to load the class. It looks like the boot loader is not asked to load AgentSupport. The issue is reproduceable on Linux using javap and a -J-Xbootclasspath/a that has just English a-z characters,
18-04-2018
Thanks David!
11-01-2018
Running Tim's test on the sca machine there seems to be quite a difference between the directory name as listed in boot.dir versus what it displayed via ls: (not sure how this will show up in JBS) > more boot.dir /tmp/JDK-8072130/testcase/BootClassPath/BootClassPathTest.d/bootØ§Ù eÅ¡tinaTÃ¼rkÃ§eespaÃ±olà¹ Ø¹Ø±Ø¨ÙØ©ä¸æä¸æÑÑÑÑÐºÐ¸Ð¹à¤¹à¤¿à¤à¤¦à¥ÎµÎ»Î»Î·Î½Î¹ÎºÎ¬×¢××¨××ªæ¥æ¬èªíêµì´LietuviÅ³Ä à¸à¸¢ > ls -al total 56 drwxr-xr-x 9 jprtadm wheel 306 Jan 11 01:43 . drwxr-xr-x 9 jprtadm wheel 306 Jan 11 01:43 .. -rw-r--r-- 1 jprtadm wheel 1393 Jan 11 01:43 Agent.class -rw-r--r-- 1 jprtadm jprtgrp 1421 Jan 11 01:43 Agent.jar -rw-r--r-- 1 jprtadm wheel 263 Jan 11 01:43 DummyMain.class -rw-r--r-- 1 jprtadm wheel 500 Jan 11 01:43 MANIFEST.MF -rw-r--r-- 1 jprtadm wheel 5598 Jan 11 01:43 Setup.class -rw-r--r-- 1 jprtadm wheel 206 Jan 11 01:43 boot.dir drwxr-xr-x 3 jprtadm wheel 102 Jan 11 01:43 bootØ§Ù Ø¹Ø±Ø¨ÙØ©ä¸æä¸æÑÑÑÑÐºÐ¸Ð¸Ìà¤¹à¤¿à¤à¤¦à¥ÎµÎ»Î»Î·Î½Î¹ÎºÎ±Ì×¢××¨××ªæ¥æ¬èªá á ¡á«á á ®á¨á á ¥LietuviuÌ¨cÌesÌtinaTuÌrkcÌ§eespanÌolà¹ à¸à¸¢ So I'm guessing that the unicode characters are in fact being modified.
11-01-2018
It appears to be a bug in the VM's processing of boot classpath entries. Using Tim's test I simply did a cd into: /tmp/JDK-8072130/testcase/BootClassPath/BootClassPathTest.d> where we have the "magic" bootXXX directory, and which contains the AgentSupport.class file from the test. I then ran javap -cp bootXXX AgentSupport, from Java 7 and Java 9 - both worked: Compiled from "AgentSupport.java" public class AgentSupport { public AgentSupport(); } I then tried javap -J-Xbootclasspath/a:bootXXX and Java 7 worked but Java 9 failed: Error: class not found: AgentSupport
11-01-2018
A simple test also worked for JDK 8u92. That makes sense as 7 and 8 uses the SysClassPath class for managing the boot classpath, whereas it was completely rewritten in 9 for the module system. This bug makes more sense if introduced in 9. I reran Tim's test using 8u92 and it also passed. This can be transferred to runtime.
11-01-2018
The test fix Tim gave was for the side-issue of the " Unrecognized option: -J-ea" problem, not the main problem.
11-01-2018
Yes my theory is that the actual creation of the directory may result in the name being modified due to "unicode decomposition" and subsequently when you try to read back the directory it is not found because it appears to have a different name. It may be hard to determine that if the things you use already handle the decomposition. I would try a simple Java test that creates a directory with the "unicode" value from the test and then see if that directory exists via Java APIs. If so it may be a VM bug if the VM is not able to handle that form of directory name.
11-01-2018
Ok, so the root cause is still a mystery then? In your comment here: https://bugs.openjdk.java.net/browse/JDK-8072130?focusedCommentId=13985188&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-13985188 You mention it may be due to the filesystem and unicode decomposition.
11-01-2018
Tim, are you saying this just needs the test fix you recommended above? It wasn't clear to me since you also indicated that this is a regression, and how the regression was introduced has not been uncovered.
11-01-2018
This is a test bug that has been failing since JDK 1.8. It may pass on some Mac OS systems, but fails every time on scaab059.us.oracle.com - refer to my earlier comment.
08-01-2018
Is this an infra bug? If this is a test bug please reassign back to serviceability
08-01-2018
A simplified test case based on BootClassPathTest.sh is attached as JDK-8072130.tar. To reproduce: - Save as /tmp/JDK-8072130.tar - cd /tmp && tar xvf JDK-8072130.tar - export JAVA_HOME=/x/java/jdk-9.jdk/Contents/Home # adjust for your system - bash /tmp/JDK-8072130/testcase/doit.bash
16-12-2017
[~dholmes] Regarding this earlier comment: Though there is an unrelated error: Cleanup... ----------System.err:(3/135)---------- Unrecognized option: -J-ea Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. which doesn't trigger a test failure. This is a test bug. TESTTOOLVMOPTS is passed to $JAVA where TESTVMOPTS should be: diff -r b9a19d1e61f2 test/jdk/java/lang/instrument/BootClassPath/BootClassPathTest.sh --- a/test/jdk/java/lang/instrument/BootClassPath/BootClassPathTest.sh Fri Dec 08 13:04:43 2017 -0800 +++ b/test/jdk/java/lang/instrument/BootClassPath/BootClassPathTest.sh Sun Dec 10 19:32:51 2017 -0800 @@ -29,6 +29,9 @@ # @key intermittent # @run shell/timeout=240 BootClassPathTest.sh if [ "${TESTJAVA}" = "" ] then echo "TESTJAVA not set. Test cannot execute. Failed." @@ -90,6 +93,6 @@ "$JAVAC" ${TESTJAVACOPTS} ${TESTTOOLVMOPTS} -d "${TESTCLASSES}" \ "${TESTSRC}"/Cleanup.java -"$JAVA" ${TESTTOOLVMOPTS} -classpath "${TESTCLASSES}" Cleanup "${BOOTDIR}" +"$JAVA" ${TESTVMOPTS} -classpath "${TESTCLASSES}" Cleanup "${BOOTDIR}" exit $result
11-12-2017
JPRT runs the test with LANG=C in the environment, this changes the default encoding from UTF-8 to US-ASCII. US-ASCII is not in the table defined in the test so the boot dir suffix is just "" and the test passes. Though there is an unrelated error: Cleanup... ----------System.err:(3/135)---------- Unrecognized option: -J-ea Error: Could not create the Java Virtual Machine. Error: A fatal exception has occurred. Program will exit. which doesn't trigger a test failure.
23-11-2016
The HFS filesystem uses unicode decomposition to create file/path names: https://developer.apple.com/legacy/library/technotes/tn/tn1150.html#UnicodeSubtleties If the unicode characters from the test need to be decomposed then I would not expect the test to work. It is possible that it may work if the Unicode is normalized.
15-08-2016
I also tested this in Aurora via rbt. Similar to JPRT the test was executed with LANG=C which sets the default encoding to US-ASCII and so the test trivially passed. I need to be able access a mach5 machine directly and log in as the test user that mach 5 uses so that I can see what is actually occurring in mach 5.
15-08-2016

Relates :	JDK-8256405 - java/lang/instrument/BootClassPath/ tests fail (macOS)
Relates :	JDK-8011194 - Apps launched via double-clicked .jars have file.encoding value of US-ASCII on Mac OS X
Relates :	JDK-8169949 - Remove java/lang/instrument/BootClassPath/BootClassPathTest.sh from ProblemList.txt
Relates :	JDK-8003228 - (props) sun.jnu.encoding should be set to UTF-8 [macosx]
Relates :	JDK-8072371 - Add two failing svc tests to the problem list