There are two issues with the java level ELF file support used by DSO.java, which support all our posix ports. The first is the ELF file support is broken for 64-bit, although that will be fixed JDK-8247272. The second is that currently OSX relies on this DSO.java ELF support, but OSX has mach-o files, not ELF files, so the DSO.java code does not work for it. This is the main issue being addressed by this CR.

In addition to the java level ELF file support for doing address to symbol lookups, there is also native support for doing this via various platform specific JVMDebugger subclasses. For example, LinuxDebuggerLocal.lookup(addr), which goes out to the native LinuxDebuggerLocal.lookupByAddress0() to do the lookup. The ELF version of this for Linux works fine for 64-bit. For OSX there is BsdDebuggerLocal.lookup(addr) which also goes out to native, and understands mach-o files. Note on OSX this only works with core files since there is no OSX support for LoadObjects and no plans to fix that. See JDK-8248875.

So the main goal of this CR is to fix address to name lookups when debugging OSX core files, and to a lesser extent get rid of the linux reliance on the java level ELF file support.

The key to doing address to symbol lookups is the LoadObject interface, which declares closestSymbolToPC() for doing this kind of symbol lookup:

public interface LoadObject
  public ClosestSymbol closestSymbolToPC(Address pc);

A LoadObject is the SA internal representation of a DLL or executable. Different platforms have different implementations:

class sun.jvm.hotspot.debugger.windbg.DLL implements LoadObject
class sun.jvm.hotspot.debugger.posix.DSO implements LoadObject
class sun.jvm.hotspot.debugger.linux.SharedObject extends DSO
class sun.jvm.hotspot.debugger.bsd.SharedObject extends DSO

So LoadObject exports closestSymbolToPC() and it is implemented by DLL for windows and SharedObject for Linux and BSD. Both Linux and BSD rely on the posix DSO class for most of the SharedObject implementation, including closestSymbolToPC(), which relies on a java implementation of ELF support. As pointed out, this won't work for OSX since it has mach-o files, not ELF, and we want to completely get rid of the java level ELF support.

For this CR the reliance on ELF is being stripped out, and instead the pre-exiting native support for doing address to symbol lookups will be used, which on linux supports ELF files and on OSX support mach-o file. The native lookups are done via the platform debuggers' lookup(addr) API. However, it is declared in the following platform debugger classes, not in a common superclass like JVMDebugger:

   LinuxDebuggerLocal
   BsdDebuggerLocal
   WindbgDebuggerLocal

We want to change DSO.closestSymbolToPC() to call this platform debugger lookup() API instead of relying on the java ELF support, but there are two roadblocks. The first is that DSO does not have access to the platform debugger. This is easy to fix by passing it in to the DSO constructor. However, since the DSO class is shared by multiple platforms, it needs to store the debugger using the superclass type, JVMDebugger, which does not declare the lookup() API. We could declare the lookup() API in JVMDebugger, but it's not really appropriate to add the lookup API there since it will also impact RemoteDebuggerClient, which does not need it (and could not implement it). So instead DSO.closestSymbolToPC() will be moved down into the various SharedObject subclassses.