JDK-7183443 : Memory corruption when doing Inet6AddressImpl_lookupAllHostAddr
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 6u21
  • Priority: P3
  • Status: Closed
  • Resolution: Incomplete
  • OS: linux
  • CPU: x86
  • Submitted: 2012-07-11
  • Updated: 2014-08-25
  • Resolved: 2012-07-22
Related Reports
Duplicate :  
JDK-7194275 - Memory corruption when doing Inet6AddressImpl_lookupAllHostAddr
Relates :  
JDK-8021369 - JVM crashes with message " Program terminated with signal 11, Segmentation fault "
Description
FULL PRODUCT VERSION :
java version "1.7.0_05"
Java(TM) SE Runtime Environment (build 1.7.0_05-b05)
Java HotSpot(TM) Server VM (build 23.1-b03, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux chronos 2.6.32-5-686 #1 SMP Sun May 6 04:01:19 UTC 2012 i686 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
GNU C Library (Debian EGLIBC 2.11.3-3) stable release version 2.11.3, by Roland McGrath et al.


A DESCRIPTION OF THE PROBLEM :
I'm running an EAR in a Glassfish that does a lot of HTTP workload and DNS lookups.
The  JDK is reproducably crashing after a while of parallel network banging with always the same error message. It looks like always the same memory region is being corrupted.


REGRESSION.  Last worked in version 7

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
It's difficult to say what causes this problem, but my guess is that

/opt/java/jdk1.7.0_05/jre/lib/i386/libnet.so(Java_java_net_Inet6AddressImpl_lookupAllHostAddr+0xe5)[0x88020605]

is not thread safe or does not handle an error situation correctly.
I've taken a look inside the JDK source code but didn't find anything suspicious.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
The DNS lookups work.
ACTUAL -
The JDK exits with a problem in glibc's free function.
This indicates usually a corrupted memory region.


ERROR MESSAGES/STACK TRACES THAT OCCUR :
*** glibc detected *** /opt/java/jdk/bin/java: free(): invalid pointer: 0x85130a4a ***
======= Backtrace: =========
/lib/i686/cmov/libc.so.6(+0x6b381)[0xb7698381]
/lib/i686/cmov/libc.so.6(+0x6cbd8)[0xb7699bd8]
/lib/i686/cmov/libc.so.6(cfree+0x6d)[0xb769ccbd]
/lib/i686/cmov/libresolv.so.2(__libc_res_nsearch+0x5d0)[0x86727550]
/lib/i686/cmov/libnss_dns.so.2(_nss_dns_gethostbyname4_r+0xd0)[0x86b098a0]
/lib/i686/cmov/libc.so.6(+0xa5555)[0xb76d2555]
/lib/i686/cmov/libc.so.6(getaddrinfo+0x165)[0xb76d4735]
/opt/java/jdk1.7.0_05/jre/lib/i386/libnet.so(Java_java_net_Inet6AddressImpl_lookupAllHostAddr+0xe5)[0x88020605]
[0xb4bf284d]
[0xb4beb9d8]
[0xb4beb9d8]
[0xb4beb45a]
[0xb4beb45a]
[0xb4beb45a]
[0xb4beb45a]
[0xb4beb45a]
[0xb562e838]
[0xb4beb45a]
[0xb4beb45a]
[0xb4d537c0]
[0xb4beb9d8]
[0xb4beb2c7]
[0xb4beb845]
[0xb4beb2c7]
[0xb4beb845]
[0xb4be83d9]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x27ab75)[0xb6f5bb75]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x39f459)[0xb7080459]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x27998f)[0xb6f5a98f]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x27ad22)[0xb6f5bd22]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x2c5259)[0xb6fa6259]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x45ca94)[0xb713da94]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x45cc21)[0xb713dc21]
/opt/java/jdk1.7.0_05/jre/lib/i386/client/libjvm.so(+0x3a5fc1)[0xb7086fc1]
/lib/i686/cmov/libpthread.so.0(+0x5955)[0xb7792955]
/lib/i686/cmov/libc.so.6(clone+0x5e)[0xb76f95ee]
======= Memory map: ========
08048000-08049000 r-xp 00000000 08:04 74848792   /opt/java/jdk1.7.0_05/bin/java
08049000-0804a000 rw-p 00000000 08:04 74848792   /opt/java/jdk1.7.0_05/bin/java
0886a000-094f5000 rw-p 00000000 00:00 0          [heap]
84a00000-84aee000 rw-p 00000000 00:00 0
84aee000-84b00000 ---p 00000000 00:00 0
84b82000-84b85000 ---p 00000000 00:00 0
84b85000-84bd3000 rw-p 00000000 00:00 0
84bd3000-84bd6000 ---p 00000000 00:00 0
84bd6000-84c24000 rw-p 00000000 00:00 0
84c24000-84c27000 ---p 00000000 00:00 0
84c27000-84c75000 rw-p 00000000 00:00 0
84c75000-84c78000 ---p 00000000 00:00 0
84c78000-84cc6000 rw-p 00000000 00:00 0
84cc6000-84cc9000 ---p 00000000 00:00 0
84cc9000-84d17000 rw-p 00000000 00:00 0
84d17000-84d1a000 ---p 00000000 00:00 0
84d1a000-84d68000 rw-p 00000000 00:00 0
84d68000-84d6b000 ---p 00000000 00:00 0
84d6b000-84db9000 rw-p 00000000 00:00 0
84db9000-84dbc000 ---p 00000000 00:00 0
84dbc000-84e0a000 rw-p 00000000 00:00 0
84e0a000-84e0d000 ---p 00000000 00:00 0
84e0d000-84e5b000 rw-p 00000000 00:00 0
84e5b000-84e5e000 ---p 00000000 00:00 0
84e5e000-84eac000 rw-p 00000000 00:00 0
84eac000-84eaf000 ---p 00000000 00:00 0
84eaf000-84efd000 rw-p 00000000 00:00 0
84efd000-84f00000 ---p 00000000 00:00 0
84f00000-84f4e000 rw-p 00000000 00:00 0
84f4e000-84f51000 ---p 00000000 00:00 0
84f51000-84f9f000 rw-p 00000000 00:00 0
84f9f000-84fa2000 ---p 00000000 00:00 0
84fa2000-84ff0000 rw-p 00000000 00:00 0
84ff0000-84ff3000 ---p 00000000 00:00 0
84ff3000-85041000 rw-p 00000000 00:00 0
85041000-85044000 ---p 00000000 00:00 0
85044000-85092000 rw-p 00000000 00:00 0
85092000-85095000 ---p 00000000 00:00 0
85095000-850e3000 rw-p 00000000 00:00 0
850e3000-850e6000 ---p 00000000 00:00 0
850e6000-85134000 rw-p 00000000 00:00 0
85134000-85137000 ---p 00000000 00:00 0
85137000-85185000 rw-p 00000000 00:00 0
85185000-85188000 ---p 00000000 00:00 0
85188000-851d6000 rw-p 00000000 00:00 0
851d6000-851d9000 ---p 00000000 00:00 0
851d9000-85227000 rw-p 00000000 00:00 0
85227000-8522a000 ---p 00000000 00:00 0
8522a000-85278000 rw-p 00000000 00:00 0
85278000-8527b000 ---p 00000000 00:00 0
8527b000-852c9000 rw-p 00000000 00:00 0
852c9000-852cc000 ---p 00000000 00:00 0
852cc000-8531a000 rw-p 00000000 00:00 0
8531a000-8531d000 ---p 00000000 00:00 0
8531d000-8536b000 rw-p 00000000 00:00 0
8536b000-8536e000 ---p 00000000 00:00 0
8536e000-853bc000 rw-p 00000000 00:00 0
853bc000-853bf000 ---p 00000000 00:00 0
853bf000-8540d000 rw-p 00000000 00:00 0
8540d000-85410000 ---p 00000000 00:00 0
85410000-8545e000 rw-p 00000000 00:00 0
8545e000-85461000 ---p 00000000 00:00 0
85461000-854af000 rw-p 00000000 00:00 0
854af000-854b2000 ---p 00000000 00:00 0
854b2000-85500000 rw-p 00000000 00:00 0
85500000-85529000 rw-p 00000000 00:00 0
85529000-85600000 ---p 00000000 00:00 0
8560d000-85610000 ---p 00000000 00:00 0
85610000-8565e000 rw-p 00000000 00:00 0
8565e000-85661000 ---p 00000000 00:00 0
85661000-856af000 rw-p 00000000 00:00 0
856af000-856b2000 ---p 00000000 00:00 0
856b2000-85700000 rw-p 00000000 00:00 0
85700000-857ff000 rw-p 00000000 00:00 0
857ff000-85800000 ---p 00000000 00:00 0
85800000-8586f000 rw-p 00000000 00:00 0
8586f000-85900000 ---p 00000000 00:00 0
85900000-85a00000 rw-p 00000000 00:00 0
85a0d000-85a10000 ---p 00000000 00:00 0
85a10000-85a5e000 rw-p 00000000 00:00 0
85a5e000-85a61000 ---p 00000000 00:00 0
85a61000-85aaf000 rw-p 00000000 00:00 0
85aaf000-85ab2000 ---p 00000000 00:00 0
85ab2000-85b00000 rw-p 00000000 00:00 0
85b00000-85cfa000 rw-p 00000000 00:00 0
85cfa000-85d00000 ---p 00000000 00:00 0
85d00000-85f00000 rw-p 00000000 00:00 0
85f00000-85fe7000 rw-p 00000000 00:00 0
85fe7000-86000000 ---p 00000000 00:00 0
86000000-860f4000 rw-p 00000000 00:00 0
860f4000-86100000 ---p 00000000 00:00 0
8610d000-86110000 ---p 00000000 00:00 0
86110000-8615e000 rw-p 00000000 00:00 0
8615e000-86161000 ---p 00000000 00:00 0
86161000-861af000 rw-p 00000000 00:00 0
861af000-861b2000 ---p 00000000 00:00 0
861b2000-86200000 rw-p 00000000 00:00 0
86200000-86300000 rw-p 00000000 00:00 0
8631a000-8631d000 ---p 00000000 00:00 0
8631d000-8636b000 rw-p 00000000 00:00 0
8636b000-8636e000 ---p 00000000 00:00 0
8636e000-863bc000 rw-p 00000000 00:00 0
863bc000-863bf000 ---p 00000000 00:00 0
863bf000-8640d000 rw-p 00000000 00:00 0
8640d000-86410000 ---p 00000000 00:00 0
86410000-8645e000 rw-p 00000000 00:00 0
8645e000-86461000 ---p 00000000 00:00 0
86461000-864af000 rw-p 00000000 00:00 0
864af000-864b2000 ---p 00000000 00:00 0
864b2000-86500000 rw-p 00000000 00:00 0
86500000-86600000 rw-p 00000000 00:00 0
86600000-866ff000 rw-p 00000000 00:00 0
866ff000-86700000 ---p 00000000 00:00 0
86720000-86730000 r-xp 00000000 08:04 32899464   /lib/i686/cmov/libresolv-2.11.3.so
86730000-86731000 r--p 00010000 08:04 32899464   /lib/i686/cmov/libresolv-2.11.3.so
86731000-86732000 rw-p 00011000 08:04 32899464   /lib/i686/cmov/libresolv-2.11.3.so
86732000-86734000 rw-p 00000000 00:00 0
86734000-86751000 r-xp 00000000 08:04 32508962   /lib/libgcc_s.so.1
86751000-86752000 rw-p 0001c000 08:04 32508962   /lib/libgcc_s.so.1
86774000-867a7000 r-xp 00000000 08:04 74849013   /opt/java/jdk1.7.0_05/jre/lib/i386/libsunec.so
867a7000-867ab000 rw-p 00032000 08:04 74849013   /opt/java/jdk1.7.0_05/jre/lib/i386/libsunec.so
867ab000-867af000 rw-p 00000000 00:00 0
867af000-867b2000 ---p 00000000 00:00 0
867b2000-86800000 rw-p 00000000 00:00 0
86800000-868fd000 rw-p 00000000 00:00 0
868fd000-86900000 ---p 00000000 00:00 0
86900000-869ff000 rw-p 00000000 00:00 0
869ff000-86a00000 ---p 00000000 00:00 0
86a00000-86b00000 rw-p 00000000 00:00 0


REPRODUCIBILITY :
This bug can be reproduced often.
Comments
Some more information provided by submitter. Since it also applies to older releases, I'll change the affects version. It also seems more likely now to be a glibc bug. DIFFERENT JDK VERSIONS ====================== I've made some test runs. The results: has_problem(jdk1.7.0_05) == true has_problem(jdk1.7.0_01) == true has_problem(jdk1.6.0_26) == true has_problem(jdk1.6.0_21) == true This means the problem exists with probably every JDK on Linux x86. DIFFERENT OS VERSIONS ===================== My assumption from before was wrong. The problem also shows on Fedora Linux 17. But it doesn't show so far on OpenSolaris 11. My current working hypothesis is that it's a bug in the glibc, similar to this one: http://sourceware.org/bugzilla/show_bug.cgi?id=10652 Take a look at the date of the bug... >> Do you have suggestions on narrowing this with a test case? > Is there any way you could cut down the EAR to something smaller that > you can send us, but which still shows the problem? I couldn't reproduce it with two simple test programs. Logging doesn't help to show a pattern. It's probably <1% of all ipv6 nameservice lookups on a multithreaded machine.
30-07-2013
EVALUATION I'm marking this bug incomplete because it's not clear whether it has been observed on a supported configuration or not. Please tell us exactly which distribution is involved and whether glibc has been replaced or not. Note that if we have a bug in this area then we would expect to get a lot of bug reports. The last bug report in this area turned out to be a glibc bug that has since been fixed. If this crash is observed with a supported configuration then please provide details as to how to run GF so that we can duplicate the crash.
22-07-2012