JDK-8235637 : jhsdb jmap from OpenJDK 11.0.5 doesn't work if prelink is enabled
  • Type: Bug
  • Component: hotspot
  • Sub-Component: svc-agent
  • Affected Version: 11.0.5,14
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • Submitted: 2019-11-14
  • Updated: 2020-06-09
  • Resolved: 2019-12-24
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 14 JDK 15 Other
11.0.7Fixed 13.0.4Fixed 14 b30Fixed 15Fixed openjdk8u252Fixed
Related Reports
Relates :  
JDK-8225636 - SA can't handle prelinked libraries
Description
ADDITIONAL SYSTEM INFORMATION :
OS
centos-release-7-6.1810.2.el7.centos.x86_64
and
centos-release-6-10.el6.centos.12.3.x86_64

JDK
openjdk version "11.0.5" 2019-10-15
OpenJDK Runtime Environment 18.9 (build 11.0.5+10)
OpenJDK 64-Bit Server VM 18.9 (build 11.0.5+10, mixed mode)

A DESCRIPTION OF THE PROBLEM :
I can no longer use jhsdb jmap to generate a heap dump from a core dump created by gdb. The issue only occurs if prelink is enabled. Only OpenJDK 11.0.5 is affected. OpenJDK 11.0.4 and Orcale JDK 11.0.5 are not affected.

REGRESSION : Last worked in version 11

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Use CentOS 7 or 6. (I tested with centos-release-7-6.1810.2.el7.centos.x86_64 and centos-release-6-10.el6.centos.12.3.x86_64).
2. Install OpenJDK 11.0.5 (I used https://github.com/AdoptOpenJDK/openjdk11-upstream-binaries/releases/download/jdk-11.0.5%2B10/OpenJDK11U-jdk_x64_linux_11.0.5_10.tar.gz).
3. Install prelink (yum install prelink).
4. Enable it (prelink -a).
5. Start any Java process. I used the default Tomcat 9 installation for example.
6. Create a core dump using gdb. I used the following commands:
gdb
attach <Java PID>
gcore jvm-prelink.core
detach
quit
7. Use jhsdb jmap to convert it to a Java heap dump:
jhsdb jmap --binaryheap --dumpfile dump.hprof --exe <path to java>/java --core jvm-prelink.core


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
dump.hprof is created
ACTUAL -
I'm getting an error. It looks like this with LIBSAPROC_DEBUG set to true:
libsaproc DEBUG: lib_ld = 0x3001614c38, lib_dyn_addr = 0x3001614c38 -> lib_base_diff = 0x0
Error attaching to core file: Can't attach to the core file
sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file
        at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method)
        at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:282)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:674)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:612)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:338)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305)
        at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:157)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:191)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118)
        at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:329)
        at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:488)

CUSTOMER SUBMITTED WORKAROUND :
Use OpenJDK 11.0.4 or Orcale JDK 11.0.5 instead of OpenJDK 11.0.5.

FREQUENCY : always
Comments
Fix request for 13u: This change fixes a regression caused by the fix for JDK-8225636. The change applies cleanly.
28-04-2020
Fix Request for 11u and 8u: This is a regression introduced due to JDK-8225636, fix is to improve the error condition to hanle invalid memory access. we need to fix this issue for LTS version (8u and 11u). The fix is low risk. This change can pass serviceability/sa jtreg tests. The JDK 14 patch applies as is to JDK 11u. For 8u, we need to change the path to ps_core.c , but we can apply it without further change.
26-12-2019
URL: https://hg.openjdk.java.net/jdk/jdk14/rev/cf32454b65f0 User: dbuck Date: 2019-12-24 09:44:49 +0000
24-12-2019
Hi [~ysuenaga], thanks for patch, let me apply and validate the fix...
11-12-2019
According to GDB sourcecode [1], we should allow when lib_base_diff == 0. So I guess we can fix as below. (I haven't tested it yet.) ``` diff --git a/src/jdk.hotspot.agent/linux/native/libsaproc/ps_core.c b/src/jdk.hotspot.agent/linux/native/libsaproc/ps_core.c --- a/src/jdk.hotspot.agent/linux/native/libsaproc/ps_core.c +++ b/src/jdk.hotspot.agent/linux/native/libsaproc/ps_core.c @@ -536,7 +536,7 @@ phbuf = read_program_header_table(lib_fd, elf_ehdr); if (phbuf == NULL) { print_debug("can't read program header of shared object\n"); - return 0L; + return -1L; } // Get the address of .dynamic section from shared library. @@ -552,7 +552,7 @@ if (ps_pdread(ph, (psaddr_t)link_map_addr + LINK_MAP_LD_OFFSET, &lib_ld, sizeof(uintptr_t)) != PS_OK) { print_debug("can't read address of dynamic section in shared object\n"); - return 0L; + return -1L; } // Return the load address which is calculated by the address of .dynamic @@ -665,7 +665,7 @@ if (read_elf_header(lib_fd, &elf_ehdr)) { if (lib_base_diff == 0x0L) { lib_base_diff = calc_prelinked_load_address(ph, lib_fd, &elf_ehdr, link_map_addr); - if (lib_base_diff == 0x0L) { + if (lib_base_diff == -1L) { close(lib_fd); return false; } ``` [1] http://www.sourceware.org/git/?p=binutils-gdb.git;a=blob;f=gdb/solib-svr4.c;h=de765576d0b7fda28028b4899a79ccffb16f4fae;hb=HEAD#l289
11-12-2019
With LIBSAPROC_DEBUG=1 libsaproc DEBUG: lib_ld = 0x3001a14dd8, lib_dyn_addr = 0x3001a14dd8 -> lib_base_diff = 0x0 Error attaching to core file: Can't attach to the core file sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file From the code if (lib_base_diff == 0x0L) { lib_base_diff = calc_prelinked_load_address(ph, lib_fd, &elf_ehdr, link_map_addr); if (lib_base_diff == 0x0L) { close(lib_fd); return false; <== as lib_base_diff is still 0 we return from here, } } for one of the .so file lib_base_diff is turning out to be 0, libsaproc DEBUG: lib_base_diff= 0x0 ] libsaproc DEBUG: lib_ld = 0x3021416d70, lib_dyn_addr = 0x3021416d70 -> lib_base_diff = 0x0 libsaproc DEBUG: reading library /lib64/libnsl.so.1 @ 0x3021200000 [ 0x0 ] In case there is no "address conflict @ 0x0" we should not be doing above calculation.
10-12-2019
Issue started from 14 ea b02 onwards, Due to JDK-8225636, reverting back the changeset http://closedjdk.us.oracle.com/jdk/jdk-cpu/open/rev/da554fdb51d0 could fix the issue
10-12-2019
I am able to reproduce this issue in jdk14 ea b26 == -sh-4.2$ /scratch/fairoz/JAVA/jdk14/jdk-14-ea+26/bin/jhsdb jmap --binaryheap --dumpfile dump.hprof --exe /scratch/fairoz/JAVA/jdk14/jdk-14-ea+26/bin/java --core ./core1 Attaching to core ./core1 from executable /scratch/fairoz/JAVA/jdk14/jdk-14-ea+26/bin/java, please wait... Error attaching to core file: Can't attach to the core file sun.jvm.hotspot.debugger.DebuggerException: Can't attach to the core file at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach0(Native Method) at jdk.hotspot.agent/sun.jvm.hotspot.debugger.linux.LinuxDebuggerLocal.attach(LinuxDebuggerLocal.java:345) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attachDebugger(HotSpotAgent.java:674) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebuggerLinux(HotSpotAgent.java:612) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.setupDebugger(HotSpotAgent.java:338) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.go(HotSpotAgent.java:305) at jdk.hotspot.agent/sun.jvm.hotspot.HotSpotAgent.attach(HotSpotAgent.java:157) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.start(Tool.java:193) at jdk.hotspot.agent/sun.jvm.hotspot.tools.Tool.execute(Tool.java:118) at jdk.hotspot.agent/sun.jvm.hotspot.tools.JMap.main(JMap.java:176) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.runJMAP(SALauncher.java:321) at jdk.hotspot.agent/sun.jvm.hotspot.SALauncher.main(SALauncher.java:406) -sh-4.2$
10-12-2019
Assigning myself to reproduce this issue on jdk/jdk
10-12-2019
It would be useful to find out if the issue is the target VM or the VM SA is running on. So try attaching from OracleJDK to OpenJDK core file and also vice versa.
10-12-2019
Deleting my previous comments. None of the failures I mentioned are related to attach failures, only to dumping the heap after attaching.
10-12-2019
I could see, only one related issue JDK-8225715, which got fixed in 11.0.5 OpenJDK and not in 11.0.5 Oracle JDK
10-12-2019
I wonder if OpenJDK 11.0.5 backported something that Oracle JDK 11.0.5 did not, and that backport is the cause of this bug. If so, that could mean it is also broken in later releases, even Oracle JDK. Someone should try out the current JDK 14.
10-12-2019
According to submitter >Only OpenJDK 11.0.5 is affected. OpenJDK 11.0.4 and Orcale JDK 11.0.5 are not affected Issue is only observed on OpenJDK builds.
10-12-2019