JDK-8248879 : SA core file support on OSX has some bugs trying to locate the jvm libraries
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 16
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2020-07-06
  • Updated: 2025-04-07
  • Resolved: 2020-08-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 16
16 b11Fixed
Related Reports
Relates :  
JDK-8232973 - Potential infinite loop in macOS hotspot agent
Description
I tried to attach to a core file that was generated while running the jhsdb command (yes, using SA to debug SA). SA crashed when I did this, and after fixing the crash, it was still unable to attach. 

The issue is with ps_core.c::get_real_path(), which tries to find the path to the java libs. It starts by searching for bin/java in the path to the executable (execpath), and uses that to find the path to the java libraries. However, that doesn't work if the execpath ends in bin/jhsdb, or any other java launcher in the bin directory. It then checks JAVA_HOME. If that is null, it then falls into the following code:

      char* dyldpath = getenv("DYLD_LIBRARY_PATH");
      char* save_ptr;
      char* dypath = strtok_r(dyldpath, ":", &save_ptr);

This gets a SEGV if DYLD_LIBRARY_PATH is null (the crash I mentioned above), so one fix needed is to add the null check. If the DYLD_LIBRARY_PATH check fails, then it gives up, so another change I made that helped with the jhsdb case was as a last resort to have it just search for bin/ instead of bin/java.
Comments
URL: https://hg.openjdk.java.net/jdk/jdk/rev/34d506267a80 User: cjplummer Date: 2020-08-06 20:17:53 +0000
06-08-2020
I made more improvements than originally mentioned in the description. As a bit of background first, get_real_path() is used to locate any library that is referenced from the core file using a relative path. So the core file will, for example, refer to @rpath/libjvm.dylib, and get_real_path() will convert that to a usable path to the file. Usually only JDK libraries and user libraries are specified with @rpath. System libraries all use full path names. get_real_path() had a couple of shortcomings. The way it worked is if the specified execname ended in bin/java or if $JAVA_HOME was set, then it only checked for libraries in subdirs of the first one of those 2 that it found to be valid. It would not look in both directories if both were valid, only in the first to be found valid. Only if neither of those were valid did it look in DYLD_LIBRARY_PATH. So, for example, as long as execname ended in bin/java, that's the only jdk directory that was checked for libraries. If it didn't end in bin/java, and $JAVA_HOME was set, then only it was checked. Then I added (in my first fix for this bug) a 3rd option looking for the existence of any "bin/" in execname so it would also work if, for example, the executable was bin/jhsdb or bin/javac. Only if none of these 3 paths existed did the code defer to DYLD_LIBRARY_PATH. That made is hard to locate non JDK libraries, such as user JNI libraries, or to override the execname search for the JDK by setting $JAVA_HOME. I've fixed this by having get_real_path() check all 3 of the potential JDK locations not only to see if the paths are valid, but also if the library is in any of the paths, and then check all the paths DYLD_LIBRARY_PATH if it failed to find the library in the JDK paths. So now all the potential locations are checked to see if they contain the library. By doing this I was able to make it find the JDK libraries by properly specifying the execname or JAVA_HOME, and still find a user JNI library in DYLD_LIBRARY_PATH.
06-08-2020