Bug ID: JDK-6626677 Error: Unimplemented()/HPI sysMonitorExit is broken on linux

JDK-6626677 : Error: Unimplemented()/HPI sysMonitorExit is broken on linux

Type: Bug
Component: core-libs
Sub-Component: java.lang
Affected Version: 6u10,7

Priority: P2
Status: Closed
Resolution: Fixed
OS: generic,linux
CPU: generic

Submitted: 2007-11-06
Updated: 2012-02-02
Resolved: 2011-05-18

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

Other	JDK 6	JDK 7
1.4.2_27Fixed	6u18Fixed	7 b30Fixed

Related Reports

Duplicate :	JDK-6638105 - Unimplemented() at hpi.cpp:30
Relates :	JDK-6502237 - HPI sysMonitorExit is broken on linux-amd64
Relates :	JDK-6983747 - Remove obsolete dl_mutex lock
Relates :	JDK-6918009 - -XX:+UseConcMarkSweepGC -XX:+CMSIncrementalMode -XX:+UseTLAB causes volano test to crash
Relates :	JDK-6341237 - JVM 1.4.2_08 crashes on RHL AS 3.0
Relates :	JDK-6721093 - -XX:AppendRatio=N not supported
Relates :	JDK-6504660 - HPI panic callback is dead code

Description

Here are other analysis report entries. The hs_err_pid file
is not saved by JavaTest/JTREG so these entries have much
less info:

New MM_REGRESSION failures (from 2007.11.05)
*   java/lang/management/MemoryMXBean/PendingAllGC.sh
*   sun/management/snmp/jvmMemory/JvmMemPoolTest.sh
        These tests failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux AMD64 Server VM (machine vm-amd64-01). There is
        another instance of this assertion failing in
        ValidateOpenTypes.java below.

        Update: See the entry for sourcePath_s002 for the gory details.

New MM_REGRESSION failures (from 2007.11.02)
    sun/management/jmxremote/bootstrap/RmiSslNoKeyStoreTest.sh
        This test failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux AMD64 Server VM (machine em64t-002). There is another
        instance of this assertion failing in ValidateOpenTypes.java
        below.

        Update: See the entry for sourcePath_s002 for the gory details.

New MM_REGRESSION failures (from 2007.10.30)
    sun/management/jmxremote/bootstrap/RmiSslBootstrapTest.sh
        This test failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux AMD64 Server VM (machine vmnightly6). There is another
        instance of this assertion failing in ValidateOpenTypes.java
        below.

        Update: See the entry for sourcePath_s002 for the gory details.

New MM_REGRESSION failures (from 2007.10.10)
    sun/management/snmp/generic/GenericTest.sh
        This test failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:33)
            Error: Unimplemented()

        on Linux IA32 Server VM (machine em64t-002). There is another
        instance of this assertion failing in ValidateOpenTypes.java
        below.

        Update: See the entry for sourcePath_s002 for the gory details.

New MM_REGRESSION failures (from 2007.09.27)
    java/lang/management/ManagementFactory/ValidateOpenTypes.java
        This test failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:33)
            Error: Unimplemented()

        on Linux IA32 Server VM (machine vm-v20z-26).

        This test executed and passed in the 2007.09.26 nightly so I'm
        not sure if the test changed between Dolphin-B20 and B21 or if
        this failure is due to the new VM in the 2007.09.27 nightly.

        Update: See the entry for sourcePath_s002 for the gory details.
We have been seeing an intermittent assertion failure in nightly.
Here is the most complete entry from my nightly analysis report:

New nsk.quick_jdi failures (from 2007.10.17)
    nsk/jdi/Location/sourcePath_s/sourcePath_s002
        This test failed the following assert:

            Internal Error (src/share/vm/runtime/hpi.cpp:33)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine sunfire002). There is another
        instance of this assertion failing in ValidateOpenTypes.java
        below.

        Update: Because this is an NSK test, the hs_err_pid file was
            saved. Here is info for the failing thread:

            ---------------  T H R E A D  ---------------

            Current thread (0x080c0800):  JavaThread "Finalizer" daemon [_thread_in_vm, id=9893, stack(0xedae4000,0xedb35000)]

            Stack: [0xedae4000,0xedb35000],  sp=0xedb33ad4,  free space=318k
            Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
            V  [libjvm.so+0x777553];;  VMError::report_and_die()+0x273
            V  [libjvm.so+0x3b01a6];;  report_unimplemented(const char*, int)+0x56
            V  [libjvm.so+0x408f22];;  unimplemented_panic+0x12
            C  [libhpi.so+0x4537]  sysMonitorExit+0xb7Warning: /export/local/common/jdk/baseline/linux-i586/jre/lib/i386/native_threads/libhpi.so does not exist
            C  [libhpi.so+0x7916]  sysFindLibraryEntry+0x56
            V  [libjvm.so+0x6284fa];;  NativeLookup::lookup_style(methodHandle,
char*, const char*, int, bool, bool&, Thread*)+0x2da
            V  [libjvm.so+0x628649];;  NativeLookup::lookup_entry(methodHandle,
bool&, Thread*)+0x59
            V  [libjvm.so+0x628e67];;  NativeLookup::lookup_base(methodHandle, bool&, Thread*)+0x27
            V  [libjvm.so+0x6290ec];;  NativeLookup::lookup(methodHandle, bool&, Thread*)+0x8c
            V  [libjvm.so+0x43b179];;  InterpreterRuntime::prepare_native_call(JavaThread*, methodOopDesc*)+0x149
            j  java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0
            j  java.lang.ref.Finalizer.runFinalizer()V+45
            j  java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1
            j  java.lang.ref.Finalizer$FinalizerThread.run()V+11
            v  ~StubRoutines::call_stub
            V  [libjvm.so+0x440d00];;  JavaCalls::call_helper(JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x260
            V  [libjvm.so+0x647f18];;  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle*, JavaCallArguments*, Thread*), JavaValue*, methodHandle*, JavaCallArguments*, Thread*)+0x18
            V  [libjvm.so+0x44030f];;  JavaCalls::call_virtual(JavaValue*, KlassHandle, symbolHandle, symbolHandle, JavaCallArguments*, Thread*)+0xff
            V  [libjvm.so+0x4404a7];;  JavaCalls::call_virtual(JavaValue*, Handle, KlassHandle, symbolHandle, symbolHandle, Thread*)+0x67
            V  [libjvm.so+0x505c22];;  thread_entry(JavaThread*, Thread*)+0x82
            V  [libjvm.so+0x72d28c];;  JavaThread::thread_main_inner()+0xcc
            V  [libjvm.so+0x6498e8];;  java_start(Thread*)+0x138
            C  [libpthread.so.0+0x5371]

            Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
            j  java.lang.ref.Finalizer.invokeFinalizeMethod(Ljava/lang/Object;)V+0
            j  java.lang.ref.Finalizer.runFinalizer()V+45
            j  java.lang.ref.Finalizer.access$100(Ljava/lang/ref/Finalizer;)V+1
            j  java.lang.ref.Finalizer$FinalizerThread.run()V+11
            v  ~StubRoutines::call_stub

        This failure is looking like an instance of:

            6341237 2/3 JVM 1.4.2_08 crashes on RHL AS 3.0

        I'll check with Steve Bohne just to be sure.

        Update: Steve requests that a new bug be filed. On further
            analysis, 6341237 is about two distinct issues and Steve
            doesn't want to muddy the waters.
More occurrences of this failure mode in nightly testing:

New MM_REGRESSION failures (from 2007.11.07)
*   java/lang/management/ThreadMXBean/ThreadCpuTime.java
*   sun/management/jmxremote/bootstrap/RmiBootstrapTest.sh
        These tests failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        RmiBootstrapTest.sh failed on Linux AMD64 Server VM (machine
        wowamd) and ThreadCpuTime.java failed on Linux IA32 Server VM
        (machine vm-v20z-26). This failure mode is covered by the
        following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
Another occurrence of this failure mode in nightly testing:

New nsk.quick_jdi failures (from 2007.11.26)
*   nsk/jdi/ThreadReference/forceEarlyReturn/forceEarlyReturn014
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine dip). This failure mode is
        covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
Another occurrence of this failure mode in nightly testing:

New MM_REGRESSION failures (from 2007.12.01)
*   sun/management/snmp/generic/GenericTest.sh
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux AMD64 Server VM (machine wowamd). This failure mode is
        covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
http://sqeweb/nfs/tools/gtee/results/JDK7/NIGHTLY/VM/2008-03-03/GC_Baseline/vm/linux-amd64/server/mixed/vm-linux-amd64_server_mixed_vm.gc.testlist2008-03-04-00-09-53/ResultDir/Juggle32/hs_err_pid21522.log

Test case: gc/ArrayJuggle/Juggle32
           or gc.ArrayJuggle.Juggle32

Tlog file:
-----------------------
#! sh
#
#

test_name="Juggle32"
test_work_dir="/export/local/4661.JDK7.NIGHTLY.VM+linux-amd64_server_mixed_vm.gc.testlist/results/ResultDir/Juggle32"
test_case_name="Juggle32"
SHELL="/bin/ksh"
STRESS_OPTIONS=""
RAS_OPTIONS=""
CLASSPATH="/export/local/common/testbase/6/vm/vm/bin/classes:/export/local/common/jdk/baseline/linux-amd64/lib/tools.jar"
PATH="/export/local/common/jdk/baseline/linux-amd64/bin:/bin:/usr/bin"
JAVA="/export/local/common/jdk/baseline/linux-amd64/bin/java"
TEMP=""
DISPLAY="gtee.sfbay:0"
JAVA_OPTS="-d64 -server -Xmixed -DHANGINGJAVA31766 -XX:-PrintVMOptions"
TESTBASE="/export/local/common/testbase/6/vm/vm"
HOME="/import/gtee"
ARCH="linux-amd64"
SEPARATOR=":"
PS=":"
LIBJSIG_PATH="/export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjsig.so"
SystemRoot=""
LD_LIBRARY_PATH="/export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64:/export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server"
ROOTDIR=""
TIMEOUT="30"
COMMON_LIBS_LOCATION="/export/local/common/testbase/6/vm/vm/bin"
WINDIR=""


#
export SHELL
export DISPLAY
export LIBJSIG_PATH
export SystemRoot
export TESTBASE
export RAS_OPTIONS
export HOME
export ROOTDIR
export LD_LIBRARY_PATH
export CLASSPATH
export TEMP
export WINDIR
export PATH
TEST_DEST_DIR="Juggle32"
TESTNAME="${test_case_name}"
testName="gc/ArrayJuggle//Juggle32"
TESTDIR="${test_work_dir}"
testWorkDir="${test_work_dir}/"
export testWorkDir
tlogOutFile="${test_work_dir}/${test_name}.tlog"
testErrFile="${test_work_dir}/${test_name}.err"
EXECUTE_CLASS="${test_name}"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}${SEPARATOR}${COMMON_LIBS_LOCATION}/lib/${ARCH}/nsk/share/gc/lock/jni"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}${SEPARATOR}${COMMON_LIBS_LOCATION}/lib/${ARCH}/nsk/share/gc/lock/malloc"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}${SEPARATOR}${COMMON_LIBS_LOCATION}/lib/${ARCH}/nsk/share/gc/lock/jvmti"
LD_LIBRARY_PATH="${LD_LIBRARY_PATH}${SEPARATOR}${COMMON_LIBS_LOCATION}/lib/${ARCH}/nsk/share/gc/lock/jniref"
export LD_LIBRARY_PATH
EXECUTE_CLASS="gc.ArrayJuggle.Juggle01.Juggle01"
TEST_ARGS="-gp hashed(objectArr) -ms medium ${STRESS_OPTIONS}"
APPLICATION_TIMEOUT="${TIMEOUT}"
CLASSPATH="${test_work_dir}${PS}${CLASSPATH}"
export CLASSPATH
${JAVA} ${JAVA_OPTS} ${EXECUTE_CLASS} ${TEST_ARGS}
# Test level exit status: 134

-----------------------

Failing thread's stack retrace:
=========
Thread 1 (process 21539):
#0  0x000000318bb2e37d in raise () from /lib64/tls/libc.so.6
#1  0x000000318bb2faae in abort () from /lib64/tls/libc.so.6
#2  0x0000002a95e7053e in os::abort ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#3  0x0000002a9604dd15 in VMError::report_and_die ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#4  0x0000002a95a78516 in report_unimplemented ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#5  0x0000002a95b38c5a in unimplemented_panic ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#6  0x0000002a96510e62 in sysMonitorExit ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/native_threads/libhpi.so
#7  0x0000002a965136ed in sysFindLibraryEntry ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/native_threads/libhpi.so
#8  0x0000002a95e41eea in NativeLookup::lookup_style ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#9  0x0000002a95e42055 in NativeLookup::lookup_entry ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#10 0x0000002a95e42967 in NativeLookup::lookup_base ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#11 0x0000002a95e42c3e in NativeLookup::lookup ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#12 0x0000002a95b82b4a in InterpreterRuntime::prepare_native_call ()
   from /export/local/common/jdk/baseline/linux-amd64/jre/lib/amd64/server/libjvm.so
#13 0x0000002a96cd0426 in ?? ()
#14 0x0000000000001bed in ?? ()
#15 0x0000002a96cd036a in ?? ()
#16 0x00000000000000b8 in ?? ()
#17 0x0000000040c2b998 in ?? ()
#18 0x0000002a9f1b2e18 in ?? ()
#19 0x0000000040c2b9f8 in ?? ()
#20 0x0000002a9f23dd88 in ?? ()
#21 0x0000000000000000 in ?? ()
Another occurrence of this failure mode in nightly testing:

New MM_REGRESSION failures (from 2008.03.24)
*   sun/management/jmxremote/bootstrap/RmiRegistrySslTest.sh
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine pacmanjr). This failure mode
        is covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
Another occurrence of this failure mode in nightly testing:

New JDI_REGRESSION failures (from 2008.03.31)
*   nsk/jdi/StringArgument/isValid/isvalid003
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine vm-v20z-21). This failure mode
        is covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
Another occurrence of this failure mode in nightly testing:

New MM_REGRESSION failures (from 2008.04.15)
*   sun/management/jmxremote/bootstrap/JvmstatCountersTest.java
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine vm-v20z-26). This failure mode
        is covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.
The failure also happened in the following tests:
runtime/ParallelClassLoading/stress-redefine/holdLock/reflect/independent
runtime/ParallelClassLoading/bootstrap/stress-redefine/reflect/inner-simple

See http://sqeweb.sfbay.sun.com/nfs/results/vm/gtee/JDK7/NIGHTLY/VM/2008-04-24/GC_Baseline-Xinc/vm/linux-amd64/server/mixed/vm-linux-amd64_server_mixed_vm.parallel_class_loading.testlist2008-04-24-22-19-17/analysis_new.html
Another occurrence of this failure mode in nightly testing:

New nsk.jvmti failures (from 2008.05.15)
*   nsk/jvmti/scenarios/hotswap/HS104/hs104t002
        This test failed the following assertion:

            Internal Error (src/share/vm/runtime/hpi.cpp:30)
            Error: Unimplemented()

        on Linux IA32 Client VM (machine vmnightly6). This failure mode
        is covered by the following bug:

            6626677 4/4 Error: Unimplemented()/HPI sysMonitorExit is
                        broken on linux-amd64

        I will add this entry to 6626677.

Comments

SUGGESTED FIX The simple fix would be to ensure that NEED_DL_LOCK is not defined and to rebuild the library such that the sysMonitor calls are not made.

17-04-2008

EVALUATION src/solaris/hpi/src/linker_md.c contains the following: /* * glibc-2.0 libdl is not MT safe. If you are building with any glibc, * chances are you might want to run the generated bits against glibc-2.0 * libdl.so, so always use locking for any version of glibc. */ #ifdef __GLIBC__ #define NEED_DL_LOCK #endif When NEED_DL_LOCK is defined the sysLoadLibrary, sysUnloadLibrary and sysFindLibraryEntry methods all take the form: #ifdef NEED_DL_LOCK sysMonitorEnter(sysThreadSelf(), &_dl_lock); result = dlopen(name, RTLD_NOW); sysMonitorExit(sysThreadSelf(), &_dl_lock); #else result = dlopen(name, RTLD_LAZY); #endif The bug in the sysMonitor code is that on Linux sysThreadSelf() is always zero, hence looking at the monitor entry code: int sysMonitorEnter(sys_thread_t *self, sys_mon_t *mid) { int err; sysAssert(mid != SYS_MID_NULL); err = mutex_trylock(&mid->mutex); if (err == 0) { /* no one owns it */ mid->monitor_owner = self; mid->entry_count = 1; return SYS_OK; } else if (err == EBUSY) { /* it's already locked */ if (mid->monitor_owner == self) { mid->entry_count++; return SYS_OK; } else { when the monitor is owned, an attempt to acquire it by another thread will actually "succeed" because it will see owner == self == 0 and mistakenly think this is a recursive entry by the owner thread. The increment of entry_count is not protected from other entries, or from concurrent exits (as all threads think they own this monitor). Consequently entry_count can have lost updates and eventually/occasionally sysMonitorExit will find the entry_count is zero when it should be > 0 and so the assert fails. This is why the failure is intermittent, it's just a race condition. But it means that *any* program executing on a debug build on linux can crash because of this. Given there there is no locking actually being applied it must be the case that the dl* functions are actually thread-safe on all platforms we care about. Consequently the simple fix would be to ensure that NEED_DL_LOCK is not defined and to rebuild the library such that the sysMonitor calls are not made. Investigating the comment "glibc-2.0 libdl is not MT safe" it is correct. However it was fixed in glibc 2.1 back in September 1997. See the cvs log: http://sources.redhat.com/cgi-bin/cvsweb.cgi/libc/elf/dl-open.c?rev=1.18&content-type=text/x-cvsweb-markup&cvsroot=glibc So NEED_DL_LOCK is not needed and we should remove the initial lines: /* * glibc-2.0 libdl is not MT safe. If you are building with any glibc, * chances are you might want to run the generated bits against glibc-2.0 * libdl.so, so always use locking for any version of glibc. */ #ifdef __GLIBC__ #define NEED_DL_LOCK #endif as they are obsolete. The dl* functions on Solaris are all marked MT-safe as well. Note: the dl_open calls use different arguments depending on whether NEED_DL_LOCK is defined: RTLD_NOW vs. RTLD_LAZY. It should be noted that before NEED_DL_LOCK existed RTLD_LAZY was used for a native-threads systems and RTLD_NOW was only used for green-threads (which also needed the locking). When NEED_DL_LOCK was added it utilised the green-threads code. Using RTLD_LAZY should be perfectly fine, but it is possible that there would be some impact of changing this (hopefully a performance improvement!). If that is a concern (and there is no time to investigate it) then the use of RTLD_NOW could be maintained explicitly for linux code.

17-04-2008

WORK AROUND In your JDK change lib/<arch>/native_threads/libhpi_g.so to be a link to lib/<arch>/native_threads/libhpi.so No debug version of the library is used so no assertions can fail. Both product and debug builds will execute the same (broken) HPI code.

12-02-2008