JDK-8230233 : JVM is leaking memory
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 8u161
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: solaris_10
  • CPU: x86_64
  • Submitted: 2019-08-15
  • Updated: 2019-08-28
  • Resolved: 2019-08-28
Related Reports
Duplicate :  
JDK-8229345 - Memory leak due to vtable stubs not being shared on SPARC
Description
A DESCRIPTION OF THE PROBLEM :
When using libumem as memory allocator, and enable memory audit, there are a lot of leaks coming from 

fffffb7ffeca5028       1 fffffb7f71a6d6c0 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f7509ea80 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f7499cf00 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f64ceaf00 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f71d4b6c0 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f7f6fee40 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f64c0fa80 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6
fffffb7ffeca5028       1 fffffb7f7f7ba780 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6

This could be caused by changeset 
changeset:   8504:f8a45a60bc6b
summary:     JDK-8174962: Better interface invocations
Comments
vtable stubs are not shared on Solaris. This is the x86 version of JDK-8229345
27-08-2019
I could not reproduce this issue on 8u161/8u231. Below is the output fmatte@llg00twj:...atch/fmatte/8230233/JI-9061945$ mdb -p 17294 Loading modules: [ ld.so.1 libumem.so.1 libc.so.1 libuutil.so.1 libnvpair.so.1 libcmdutils.so.1 libavl.so.1 ] java:17294*> ::findleaks mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address mdb: leaky_subr_add_leak: couldn't read cache at 0: no mapping for address java:17294*> ::findleaks java:17294*>
27-08-2019
Additional information from submitter == I created a simple program using "mvn archetype:generate", pick springframework's aop sample (see attached), and add sleep statement to pause the application. - Build it with "mvn -Dmaven.test.skip=true package" to skip the test. - on solaris, setup the env before running it (assuming bash), export all the following env variables UMEM_LOGGING=transaction UMEM_DEBUG=audit JAVA_OPTS=-d64 -Xss8192k -XX:+PreserveFramePointer -XX:-UseCompressedOops LD_PRELOAD_64=/usr/lib/64/libumem.so - run it with "java $JAVA_OPTS -jar ./JI-9061945-1.0-SNAPSHOT.jar" - you can either gcore it or use mdb to attach to the running process, and then run ::findleaks The following is the output sample from a mdb section: > ::findleaks mdb: [fffffb7ffedc0000, fffffb7ffedc8000): couldn't read 16384 bytes at fffffb7ffedc0000: no mapping for address BYTES LEAKED VMEM_SEG CALLER 4096 14 fffffb7fde9cc000 MMAP 135168 1 fffffb7fe0ba6000 MMAP 94208 1 fffffb7fe01d8000 MMAP 20480 1 fffffb7fe01d2000 MMAP 4096 1 fffffb7fe01d0000 MMAP 4096 1 fffffb7fdf7ef000 MMAP 24576 1 fffffb7fdf7e8000 MMAP 86016 1 fffffb7fdf7d2000 MMAP 8192 1 fffffb7fdf7cf000 MMAP 114688 1 fffffb7fdedd4000 MMAP 12288 1 fffffb7fdedd0000 MMAP 131072 1 fffffb7fdeccf000 MMAP 126976 1 fffffb7fdeacd000 MMAP 118784 1 fffffb7fde9ce000 MMAP ------------------------------------------------------------------------ Total 14 oversized leaks, 884736 bytes CACHE LEAKED BUFCTL CALLER 00000000004ba028 1 0000000000546480 libjli.so`JLI_MemAlloc+0xe 00000000004bd028 1 0000000000529b40 libjli.so`JLI_MemAlloc+0xe 00000000004b8028 1 000000000053ba80 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004b9028 1 000000000053e480 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004b9028 2 000000000053e240 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 .... 00000000004ba028 1 000000000b582cc0 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004ba028 1 00000000083d7f00 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004bd028 1 0000000000948540 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004c2028 1 00000000005a4e40 libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 00000000004ce028 1 0000000000533840 libjvm.so`__1cCosNset_boot_path6Fcc_b_+0x8dd ------------------------------------------------------------------------ Total 158 buffers, 7936 bytes > 0000000008e37f00::whatis mdb: unable to readvar "umem_internal_arena": unknown symbol name 8e37f00 is allocated from umem_bufctl_audit_cache > 0000000008e37f00::bufctl_audit -v Usage: address::bufctl_audit > ::dcmds ! grep audit bufctl_audit - print a bufctl_audit > 0000000008e37f00::bufctl_audit ADDR BUFADDR TIMESTAMP THREAD CACHE LASTLOG CONTENTS 8e37f00 8e34d20 36ea9433ec7489 34 4ba028 47cf00 0 libumem.so.1`umem_cache_alloc_debug+0xfd libumem.so.1`umem_cache_alloc+0xb3 libumem.so.1`umem_alloc+0x64 libumem.so.1`umem_malloc+0x3f libjvm.so`__1cCosGmalloc6FLnKMemoryType_rknPNativeCallStack__pv_+0xb6 libjvm.so`__1cKCompiledICSset_to_megamorphic6MpnICallInfo_nJBytecodesECode_pnGThread__b_+0xc6 libjvm.so`__1cNSharedRuntimeVhandle_ic_miss_helper6FpnKJavaThread_pnGThread__nMmethodHandle__+0xf79 libjvm.so`__1cNSharedRuntimebBhandle_wrong_method_ic_miss6FpnKJavaThread__pC_+0x2f 0xfffffb7fec245c1a 0xfffffb7fec7ff854 Obviously, there are other leaks, it would be great if they can be looked at and fixed as well. ==
27-08-2019