United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-7012505 BreakpointWithFullGC.sh fails with Internal Error (src/share/vm/oops/methodOop.cpp:220)
JDK-7012505 : BreakpointWithFullGC.sh fails with Internal Error (src/share/vm/oops/methodOop.cpp:220)

Details
Type:
Bug
Submit Date:
2011-01-14
Status:
Closed
Updated Date:
2012-02-01
Project Name:
JDK
Resolved Date:
2011-03-08
Component:
hotspot
OS:
generic
Sub-Component:
jvmti
CPU:
generic
Priority:
P2
Resolution:
Fixed
Affected Versions:
hs20
Fixed Versions:
hs20 (b07)

Related Reports
Backport:
Backport:
Relates:

Sub Tasks

Description
The following JT_JDK/com/sun/jdi test started failing in the 2011.01.13
RT_Baseline nightly:

    com/sun/jdi/BreakpointWithFullGC.sh

Here is a snippet of the hs_err file from a Solaris AMD64 Server
VM -Xmixed config:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (/tmp/jprt/P3/B/223720.dcubed/source/src/share/vm/oops/methodOop.cpp:220), pid=7331, tid=11
#  assert((is_native() && bci == 0) || (!is_native() && 0<= bci && bci<code_size())) failed: illegal bci
#
# JRE version: 7.0-b124
# Java VM: Java HotSpot(TM) 64-Bit Server VM (20.0-b06-internal-201101122237.dcubed.7009828_for_hsx_20-fastdebug mixed mode solaris-amd64 compressed oops)
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x0000000000eae000):  VMThread [stack: 0xfffffd7ff1040000,0xfffffd7ff1140000] [id=11]

Stack: [0xfffffd7ff1040000,0xfffffd7ff1140000],  sp=0xfffffd7ff113e8d0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0x250ad0a]  JVM_handle_solaris_signal+0x5b0d82;;  void VMError::report(outputStream*)+0x82e
V  [libjvm.so+0x250be39]  JVM_handle_solaris_signal+0x5b1eb1;;  void VMError::report_and_die()+0x4ed
V  [libjvm.so+0xe34a5f]  void report_vm_error(const char*,int,const char*,const char*)+0x55f;;  void report_vm_error(const char*,int,const char*,const char*)+0x55f
V  [libjvm.so+0x1da5a1d]  JVM_FindSignal+0x6abc65;;  unsigned char*methodOopDesc::bcp_from(int)const+0xa5
V  [libjvm.so+0x189c35b]  JVM_FindSignal+0x1a25a3;;  unsigned char*JvmtiBreakpoint::getCacheValue()+0x13
V  [libjvm.so+0x1897b26]  JVM_FindSignal+0x19dd6e;;  void JvmtiCurrentBreakpoints::oops_do(OopClosure*)+0x96
V  [libjvm.so+0x187c181]  JVM_FindSignal+0x1823c9;;  void JvmtiExport::oops_do(OopClosure*)+0x19
V  [libjvm.so+0x206a0e6]  JVM_handle_solaris_signal+0x11015e;;  void PSMarkSweep::mark_sweep_phase3()+0x10e
V  [libjvm.so+0x2067b99]  JVM_handle_solaris_signal+0x10dc11;;  void PSMarkSweep::invoke_no_policy(bool)+0x63d
V  [libjvm.so+0x20cf3c7]  JVM_handle_solaris_signal+0x17543f;;  void PSScavenge::invoke()+0x253
V  [libjvm.so+0x1fabe18]  JVM_handle_solaris_signal+0x51e90;;  HeapWord*ParallelScavengeHeap::failed_mem_allocate(unsigned long,bool)+0x170
V  [libjvm.so+0x250e1f3]  JVM_handle_solaris_signal+0x5b426b;;  void VM_ParallelGCFailedAllocation::doit()+0xdb
V  [libjvm.so+0x2549369]  JVM_handle_solaris_signal+0x5ef3e1;;  void VM_Operation::evaluate()+0xf9
V  [libjvm.so+0x2547616]  JVM_handle_solaris_signal+0x5ed68e;;  void VMThread::loop()+0x852
V  [libjvm.so+0x254673a]  JVM_handle_solaris_signal+0x5ec7b2;;  void VMThread::run()+0xb6
V  [libjvm.so+0x1f3e072]  JVM_FindSignal+0x8442ba;;  java_start+0x6a6
C  [libc.so.1+0xdbfbb]  _thr_slot_offset+0x31b;;  _thr_setup+0x5b
C  [libc.so.1+0xdc1e0]  _thr_slot_offset+0x540;;  _lwp_start+0x0

VM_Operation (0xfffffd7ffc1cd550): ParallelGCFailedAllocation, mode: safepoint, requested by thread 0x000000000045e000

Here is a link to one of the failing config:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-13/RT_Baseline/javase/solaris-amd64/server/compd/solaris-amd64_javase_server_compd_JT_JDK_com_sun_jdi/analysis.html

So far the above assertion failure has been seen on Linux AMD64 Server-64
VM -Xcomp, Solaris AMD64 Server-64 VM -Xcomp, Linux AMD64 Server-64
VM -Xmixed, and Solaris AMD64 Server-64 VM -Xmixed.


There is a SIGSEGV form of the failure mode. Here is a snippet from
the hs_err file from a Solaris SPARC Client VM -Xmixed:

#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0xfe6bb448, pid=15438, tid=3
#
# JRE version: 7.0-b124
# Java VM: Java HotSpot(TM) Client VM (20.0-b06-internal-201101122237.dcubed.7009828_for_hsx_20-fastdebug compiled mode solaris-sparc )
# Problematic frame:
# V  [libjvm.so+0xabb448]  unsigned char*methodOopDesc::bcp_from(int)const+0x90
#
# If you would like to submit a bug report, please visit:
#   http://java.sun.com/webapps/bugreport/crash.jsp
#

---------------  T H R E A D  ---------------

Current thread (0x00142400):  VMThread [stack: 0xfb100000,0xfb180000] [id=3]

siginfo:si_signo=SIGSEGV: si_errno=0, si_code=1 (SEGV_MAPERR), si_addr=0x00000026;; 

Registers:
 G1=0xfe580c74 G2=0xf72182b8 G3=0xf72182b8 G4=0xf5000000
 G5=0x00000000 G6=0x00000000 G7=0xfd860a00 Y=0x00000000
 O0=0x0018b500 O1=0xfeeb3400 O2=0x00000000 O3=0x00000000
 O4=0x00000004 O5=0xfec8b925 O6=0xfb17f1d8 O7=0xfe576184
 L0=0x00000002 L1=0x00000003 L2=0xfeec7394 L3=0x0044ec30
 L4=0xfe587454 L5=0xfeec7394 L6=0x00000000 L7=0x00000004
 I0=0xf72182b8 I1=0x0000002b I2=0xfb17f248 I3=0xf72182c0
 I4=0xfe587518 I5=0x00000000 I6=0xfb17f258 I7=0xfe580c74
 PC=0xfe6bb448 nPC=0xfe6bb44c


Top of Stack: (sp=0xfb17f1d8)
0xfb17f1d8:   00000002 00000003 feec7394 0044ec30
0xfb17f1e8:   fe587454 feec7394 00000000 00000004
0xfb17f1f8:   f72182b8 0000002b fb17f248 f72182c0
0xfb17f208:   fe587518 00000000 fb17f258 fe580c74
0xfb17f218:   00000000 00000000 00000000 0044ec34
0xfb17f228:   00000001 00000000 00000000 00000000
0xfb17f238:   00000000 f703c488 f72182b8 f72182b8
0xfb17f248:   f7218be0 00000000 f7218be0 f7218be0 

Instructions: (pc=0xfe6bb448)
0xfe6bb428:   40 01 eb e2 90 10 00 1a 80 a6 40 1c 26 48 00 17
0xfe6bb438:   25 3f ba cd 10 80 00 0d 1f 3f b3 60 ec 07 bf f4
0xfe6bb448:   f8 15 a0 26 80 a6 40 1c 16 40 00 08 1f 3f b3 60
0xfe6bb458:   ee 06 e0 00 ee 27 bf f0 e6 07 bf f0 10 80 00 1d 
;; 00000000fe6bb438 25 3f ba cd 10 80 00 0d 1f 3f b3 60 ec 07 bf f4
;; ---------------
;; 00000000fe6bb448 f8 15 a0 26             lduh  [ %l6 + 0x26 ], %i4
;; 00000000fe6bb44c 80 a6 40 1c             cmp  %i1, %i4
;; 00000000fe6bb450 16 40 00 08             bge,pn   %icc, 0x00000000fe6bb470
;; 00000000fe6bb454 1f 3f b3 60             sethi  %hi(0xfecd8000), %o7
;; 00000000fe6bb458 ee 06 e0 00             ld  [ %i3 ], %l7
;; 00000000fe6bb45c ee 27 bf f0             st  %l7, [ %fp + -16 ]
;; 00000000fe6bb460 e6 07 bf f0             ld  [ %fp + -16 ], %l3
;; 00000000fe6bb464 10 80 00 1d             b  0x00000000fe6bb4d8
;; 
Register to memory mapping:

G1=0xfe580c74: JVM_FindSignal+0x149cf8 in /export/local/common/jdk/baseline/solaris-sparc/jre/lib/sparc/client/libjvm.so at 0xfdc00000
G2=
[error occurred during error reporting (printing register info), id 0xe0000000]

Stack: [0xfb100000,0xfb180000],  sp=0xfb17f1d8,  free space=508k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xabb448]  unsigned char*methodOopDesc::bcp_from(int)const+0x90;;  unsigned char*methodOopDesc::bcp_from(int)const+0x90
V  [libjvm.so+0x980c7c]  void GrowableCache::oops_do(OopClosure*)+0xb0;;  void GrowableCache::oops_do(OopClosure*)+0xb0
V  [libjvm.so+0x97618c]  void JvmtiExport::oops_do(OopClosure*)+0xc;;  void JvmtiExport::oops_do(OopClosure*)+0xc
V  [libjvm.so+0xc1b2e4]  void SharedHeap::process_strong_roots(bool,bool,SharedHeap::ScanningOption,OopClosure*,CodeBlobClosure*,OopsInGenClosure*)+0x204;;  void SharedHeap::process_strong_roots(bool,bool,SharedHeap::ScanningOption,OopClosure*,CodeBlobClosure*,OopsInGenClosure*)+0x204
V  [libjvm.so+0x5ad5d4]  void GenCollectedHeap::gen_process_strong_roots(int,bool,bool,bool,SharedHeap::ScanningOption,OopsInGenClosure*,bool,OopsInGenClosure*)+0x3c;;  void GenCollectedHeap::gen_process_strong_roots(int,bool,bool,bool,SharedHeap::ScanningOption,OopsInGenClosure*,bool,OopsInGenClosure*)+0x3c
V  [libjvm.so+0x5b3000]  void GenMarkSweep::mark_sweep_phase3(int)+0x22c;;  void GenMarkSweep::mark_sweep_phase3(int)+0x22c
V  [libjvm.so+0x5b1c18]  void GenMarkSweep::invoke_at_safepoint(int,ReferenceProcessor*,bool)+0x260;;  void GenMarkSweep::invoke_at_safepoint(int,ReferenceProcessor*,bool)+0x260
V  [libjvm.so+0x5c5bbc]  void OneContigSpaceCardGeneration::collect(bool,bool,unsigned,bool)+0x60;;  void OneContigSpaceCardGeneration::collect(bool,bool,unsigned,bool)+0x60
V  [libjvm.so+0x5acbf8]  void GenCollectedHeap::do_collection(bool,bool,unsigned,bool,int)+0xc48;;  void GenCollectedHeap::do_collection(bool,bool,unsigned,bool,int)+0xc48
V  [libjvm.so+0x42cd48]  HeapWord*GenCollectorPolicy::satisfy_failed_allocation(unsigned,bool)+0x2d4;;  HeapWord*GenCollectorPolicy::satisfy_failed_allocation(unsigned,bool)+0x2d4
V  [libjvm.so+0xde1424]  void VM_GenCollectForAllocation::doit()+0xe0;;  void VM_GenCollectForAllocation::doit()+0xe0
V  [libjvm.so+0xe165c0]  void VM_Operation::evaluate()+0x178;;  void VM_Operation::evaluate()+0x178
V  [libjvm.so+0xe14908]  void VMThread::loop()+0x668;;  void VMThread::loop()+0x668
V  [libjvm.so+0xe13b24]  void VMThread::run()+0x10c;;  void VMThread::run()+0x10c
V  [libjvm.so+0xb408a0]  java_start+0x188;;  java_start+0x188

VM_Operation (0xfd8ff648): GenCollectForAllocation, mode: safepoint, requested by thread 0x0005cc00

Here is a link to one of the failing configs:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-13/RT_Baseline/javase/solaris-sparc/client/compd/solaris-sparc_javase_client_compd_JT_JDK_com_sun_jdi/analysis.html

The SIGSEGV failure has been seen in the Solaris SPARC configs.
The 2011.01.13 RT_Baseline nightly had one instance of the
following test fail:

    com/sun/jdi/BreakpointTest.java

Here is a snippet of the crashing thread's stack from the hs_err file:

Stack: [0xffffffff73300000,0xffffffff73400000],  sp=0xffffffff733fe9e0,  free space=1018k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xe82558]  unsigned char*methodOopDesc::bcp_from(int)const+0xb0
V  [libjvm.so+0xc95600]  void GrowableCache::oops_do(OopClosure*)+0xb0
V  [libjvm.so+0xc89cac]  void JvmtiExport::oops_do(OopClosure*)+0xc
V  [libjvm.so+0x109ae68]  void SharedHeap::process_strong_roots(bool,bool,SharedHeap::ScanningOption,OopClosure*,CodeBlobClosure*,OopsInGenClosure*)+0x228
V  [libjvm.so+0x8492e0]  void GenCollectedHeap::gen_process_strong_roots(int,bool,bool,bool,SharedHeap::ScanningOption,OopsInGenClosure*,bool,OopsInGenClosure*)+0x50
V  [libjvm.so+0x850978]  void GenMarkSweep::mark_sweep_phase3(int)+0x288
V  [libjvm.so+0x84f28c]  void GenMarkSweep::invoke_at_safepoint(int,ReferenceProcessor*,bool)+0x2ec
V  [libjvm.so+0x86a970]  void OneContigSpaceCardGeneration::collect(bool,bool,unsigned long,bool)+0x58
V  [libjvm.so+0x8488b4]  void GenCollectedHeap::do_collection(bool,bool,unsigned long,bool,int)+0xdac
V  [libjvm.so+0x61bd5c]  HeapWord*GenCollectorPolicy::satisfy_failed_allocation(unsigned long,bool)+0x344
V  [libjvm.so+0x12c37cc]  void VM_GenCollectForAllocation::doit()+0x10c
V  [libjvm.so+0x12fb8ac]  void VM_Operation::evaluate()+0x184
V  [libjvm.so+0x12f9e18]  void VMThread::loop()+0x940
V  [libjvm.so+0x12f8d74]  void VMThread::run()+0x114
V  [libjvm.so+0xf3aa00]  java_start+0x270

This failing stack is very similar to the stack in
the BreakpointWithFullGC.sh SIGSEGV failures.


Here is the URL for the failing config:

http://sqeweb.sfbay/net/sqenfs-2/export2/results/vm/gtee/JDK7/NIGHTLY/VM/2011-01-13/RT_Baseline/javase/solaris-sparcv9/server/compd/solaris-sparcv9_javase_server_compd_JT_JDK_com_sun_jdi/analysis.html

Note that in this config both BreakpointWithFullGC.sh and
BreakpointTest.java failed.

                                    

Comments
EVALUATION

The fix for the following bug:

    6458402 4/3 3 jvmti tests fail with CMS and +ExplicitGCInvokesConcurrent

modified the breakpoint cache logic. Don't know if this fix
caused this issue or uncovered this issue, but it's a good
place to start.
                                     
2011-01-16
EVALUATION

I'm able to reproduce an assertion failure with this test reliably on solaris-x86 fastdebug with -XX:+CheckUnhandledOops -Xcomp.  To reproduce, copy BreakpointWithFullGC.sh and ShellScaffold.sh to a local directory and edit ShellScaffold to pass the args (edit the line that starts with "thecmd = $jdk/bin/$java ..."

Then set env var TESTJAVA to $JAVA_HOME and you can just run the script with sh.

We get an unhandled oop (EAX is 0xfffffff1) at: 

V  [libjvm.so+0x86f944]  ciObject*ciObjectFactory::get(oop)+0x11bc
V  [libjvm.so+0x7f9f60]  ciKlass*ciEnv::get_klass_by_name_impl(ciKlass*,ciSymbol
*,bool)+0x1c9c
V  [libjvm.so+0x87b69e]  ciSignature::ciSignature(ciKlass*,ciSymbol*)+0x846
V  [libjvm.so+0x829173]  ciMethod::ciMethod(methodHandle)+0x1c2b
V  [libjvm.so+0x872cc5]  ciObject*ciObjectFactory::create_new_object(oop)+0xeb5
V  [libjvm.so+0x8702ec]  ciObject*ciObjectFactory::get(oop)+0x1b64
V  [libjvm.so+0x7fff4b]  ciMethod*ciEnv::get_method_by_index_impl(constantPoolHa
ndle,int,Bytecodes::Code,ciInstanceKlass*)+0x6f7
V  [libjvm.so+0x8016c1]  ciMethod*ciEnv::get_method_by_index(constantPoolHandle,
int,Bytecodes::Code,ciInstanceKlass*)+0x31d
V  [libjvm.so+0x883839]  ciMethod*ciBytecodeStream::get_method(bool&)+0x6a5
V  [libjvm.so+0x898dcd]  void ciTypeFlow::StateVector::do_invoke(ciBytecodeStrea
m*,bool)+0x2d
V  [libjvm.so+0x89ca52]  bool ciTypeFlow::StateVector::apply_one_bytecode(ciByte
codeStream*)+0x8ea
V  [libjvm.so+0x8a7c5d]  void ciTypeFlow::flow_block(ciTypeFlow::Block*,ciTypeFl
ow::StateVector*,ciTypeFlow::JsrSet*)+0x55d
V  [libjvm.so+0x8aa644]  void ciTypeFlow::df_flow_types(ciTypeFlow::Block*,bool,
ciTypeFlow::StateVector*,ciTypeFlow::JsrSet*)+0x898
V  [libjvm.so+0x8aaf2f]  void ciTypeFlow::flow_types()+0x477
V  [libjvm.so+0x8ac197]  void ciTypeFlow::do_flow()+0xc3
V  [libjvm.so+0x82ed57]  ciTypeFlow*ciMethod::get_flow_analysis()+0x11b
V  [libjvm.so+0x791cfc]  CallGenerator*CallGenerator::for_inline(ciMethod*,float
)+0x4c
V  [libjvm.so+0x9987a6]  Compile::Compile(ciEnv*,C2Compiler*,ciMethod*,int,bool,
bool)+0x108a
V  [libjvm.so+0x78b874]  void C2Compiler::compile_method(ciEnv*,ciMethod*,int)+0
x16c
V  [libjvm.so+0x9b9f13]  void CompileBroker::invoke_compiler_on_method(CompileTa
sk*)+0x1827
V  [libjvm.so+0x9b795b]  void CompileBroker::compiler_thread_loop()+0xd27
V  [libjvm.so+0x1b7ec82]  void compiler_thread_entry(JavaThread*,Thread*)+0x2e
V  [libjvm.so+0x1b71099]  void JavaThread::thread_main_inner()+0x179
V  [libjvm.so+0x1b70d49]  void JavaThread::run()+0x619
V  [libjvm.so+0x17d4263]  java_start+0x6fb
C  [libc.so.1+0xa73c7]  _thr_setup+0x4e
C  [libc.so.1+0xa76b0]  __moddi3+0x60

This does not reproduce with jvmg, unfortunately.
                                     
2011-01-18
EVALUATION

Naked oop found in ciEnv::get_klass_by_name_impl(), the variable 'klassOop found_klass'.
However, this is not the cause of the test failure.  With this handlized correctly, the test passes when run with -XX:+CheckUnhandledOops, but unfortunately continues to fail when run without the flag.
                                     
2011-01-18
EVALUATION

http://hg.openjdk.java.net/jdk7/hotspot-rt/hotspot/rev/9afee0b9fc1d
                                     
2011-01-20
EVALUATION

http://hg.openjdk.java.net/jdk7/hotspot/hotspot/rev/9afee0b9fc1d
                                     
2011-01-20



Hardware and Software, Engineered to Work Together