Bug ID: JDK-8217287 Error reporting of bug JDK-8216314 cause other errors and misleading hs

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 13

Priority: P4
Status: Closed
Resolution: Duplicate

Submitted: 2019-01-16
Updated: 2019-01-23
Resolved: 2019-01-23

While investigation crash with following hs_err I dinf that real issue is bug JDK-8216314. However there was another error during error reporting and another error in other thread.
hs_err:

# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGILL (0x4) at pc=0x0000000118ad8f11, pid=11620, tid=16643
#
# JRE version: Java(TM) SE Runtime Environment (13.0) (fastdebug build 13-internal+0-jdk13-jdk.108)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (fastdebug 13-internal+0-jdk13-jdk.108, compiled mode, g1 gc, bsd-amd64)
# Problematic frame:
# J 15051 c2 java.lang.invoke.LambdaForm$MH.linkToTargetMethod(JJLjava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object; java.base@13-internal (15 bytes) @ 0x0000000118ad8f11 [0x0000000118ad90c0+0xfffffffffffffe51]
..

---------------  T H R E A D  ---------------

Current thread (0x00007ff65100d000):  JavaThread "Attach Listener" daemon [_thread_in_vm, id=16643, stack(0x00007000077a8000,0x00007000078a8000)]

Stack: [0x00007000077a8000,0x00007000078a8000],  sp=0x00007000078a78a8,  free space=1022k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
J 15051 c2 java.lang.invoke.LambdaForm$MH.linkToTargetMethod(JJLjava/lang/Object;ILjava/lang/Object;)Ljava/lang/Object; java.base@13-internal (15 bytes) @ 0x0000000118ad8f11 [0x0000000118ad90c0+0xfffffffffffffe51]

[error occurred during error reporting (printing native stack), id 0xb, SIGSEGV (0xb) at pc=0x000000010dadf73e]


siginfo: si_signo: 4 (SIGILL), si_code: 1 (ILL_ILLOPC), si_addr: 0x0000000118ad8f11

Register to memory mapping:

RAX=0x0000000118a9d000 points into unknown readable memory: 00 00 00 00 00 00 00 80
RBX=0x00007000078a7938 is pointing into the stack for thread: 0x00007ff65100d000
RCX=0x000000010e579320: _ZTV13TypeNarrowOop+0x30 in /scratch/mesos/jib-master/install/jdk13-jdk.108/macosx-x64-debug.jdk/jdk-13/fastdebug/lib/server/libjvm.dylib at 0x000000010d600000
RDX=0x0 is NULL
RSP=0x00007000078a78a8 is pointing into the stack for thread: 0x00007ff65100d000
RBP=0x00007000078a7a30 is pointing into the stack for thread: 0x00007ff65100d000
RSI=0x000000010e2a69a8: __cxx_global_var_init+0x26c22 in /scratch/mesos/jib-master/install/jdk13-jdk.108/macosx-x64-debug.jdk/jdk-13/fastdebug/lib/server/libjvm.dylib at 0x000000010d600000
RDI=0x0000000118a95710 points into unknown readable memory: 00 d0 a9 18 01 00 00 00
R8 =0x00007000078a6d68 is pointing into the stack for thread: 0x00007ff65100d000
R9 =0x0000000000000010 is an unknown value
R10=0x00007000078a7880 is pointing into the stack for thread: 0x00007ff65100d000
R11=0x00007ff542c960a9 is an unknown value
R12=0x00007ff64f80f0d0 points into unknown readable memory: 98 68 56 0e 01 00 00 00
R13=0x0000000000000700 is an unknown value
R14=0x00000000000018ca is an unknown value
R15=0x0000000118a95710 points into unknown readable memory: 00 d0 a9 18 01 00 00 00


Registers:
RAX=0x0000000118a9d000, RBX=0x00007000078a7938, RCX=0x000000010e579320, RDX=0x0000000000000000
RSP=0x00007000078a78a8, RBP=0x00007000078a7a30, RSI=0x000000010e2a69a8, RDI=0x0000000118a95710
R8 =0x00007000078a6d68, R9 =0x0000000000000010, R10=0x00007000078a7880, R11=0x00007ff542c960a9
R12=0x00007ff64f80f0d0, R13=0x0000000000000700, R14=0x00000000000018ca, R15=0x0000000118a95710
RIP=0x0000000118ad8f11, EFLAGS=0x0000000000010202, ERR=0x0000000000000000
  TRAPNO=0x0000000000000006

Stack from core:

thread #14, stop reason = signal SIGSTOP
    frame #0: 0x00007fff6dc10b6e libsystem_kernel.dylib`__pthread_kill + 10
    frame #1: 0x00007fff6dddb080 libsystem_pthread.dylib`pthread_kill + 333
    frame #2: 0x00007fff6db6c1ae libsystem_c.dylib`abort + 127
    frame #3: 0x000000010dff5c47 libjvm.dylib`os::abort(dump_core=true, siginfo=<unavailable>, context=<unavailable>) at os_bsd.cpp:1062 [opt]
    frame #4: 0x000000010e21c377 libjvm.dylib`VMError::report_and_die(id=<unavailable>, message=<unavailable>, detail_fmt=<unavailable>, detail_args=<unavailable>, thread=<unavailable>, pc="H\x8b\a<FF>P \x0f\xb6<C0>A\x89G\x18H\x83\U00000088[A^A_]\U00000095H\x89<E5><B8>
^C", siginfo=0x00007000078a5ac0, context=0x00007000078a5b28, filename=0x0000000000000000, lineno=0, size=0) at vmError.cpp:1551 [opt]
    frame #5: 0x000000010e21b88a libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>, detail_fmt=<unavailable>) at vmError.cpp:1269 [opt]
    frame #6: 0x000000010e21c3c5 libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>) at vmError.cpp:1275 [opt]
    frame #7: 0x000000010e21ccf0 libjvm.dylib`crash_handler(sig=11, info=0x00007000078a5ac0, ucVoid=0x00007000078a5b28) at vmError_posix.cpp:140 [opt]
    frame #8: 0x00007fff6ddcef5a libsystem_platform.dylib`_sigtramp + 26
    frame #9: 0x000000010dadf73f libjvm.dylib`frame::frame(this=0x00007000078a5c80, sp=<unavailable>, unextended_sp=<unavailable>, fp=<unavailable>, pc=" <DF>X\x0e\x01") at frame_x86.inline.hpp:82 [opt]
    frame #10: 0x000000010daded61 libjvm.dylib`frame::sender_for_compiled_frame(RegisterMap*) const [inlined] frame::frame(this=<unavailable>, sp=<unavailable>, unextended_sp=<unavailable>, fp=<unavailable>, pc=<unavailable>) at frame_x86.inline.hpp:66 [opt]
    frame #11: 0x000000010daded50 libjvm.dylib`frame::sender_for_compiled_frame(this=0x00007000078a6f50, map=0x00007000078a5cd0) const at frame_x86.cpp:482 [opt]
    frame #12: 0x000000010dade47b libjvm.dylib`frame::sender(this=0x00007000078a6f50, map=0x00007000078a5cd0) const at frame_x86.cpp:498 [opt]
    frame #13: 0x000000010e218c8f libjvm.dylib`VMError::print_native_stack(st=<unavailable>, fr=frame @ 0x00007000078a6f50, t=0x00007ff65100d000, buf="libjvm.dylib+0x9f67d7", buf_size=2000) at vmError.cpp:252 [opt]
    frame #14: 0x000000010e219dd3 libjvm.dylib`VMError::report(st=0x000000010e67c988, _verbose=true) at vmError.cpp:705 [opt]
    frame #15: 0x000000010e21beab libjvm.dylib`VMError::report_and_die(id=<unavailable>, message=<unavailable>, detail_fmt=<unavailable>, detail_args=0x00007000078a71e0, thread=<unavailable>, pc="\aW\x0e\x01", siginfo=0x00007000078a7788, context=0x00007000078a77f0, filename=0x0000000000000000, lineno=0, size=0) at vmError.cpp:1466 [opt]
    frame #16: 0x000000010e21b88a libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>, detail_fmt=<unavailable>) at vmError.cpp:1269 [opt]
    frame #17: 0x000000010e21c3c5 libjvm.dylib`VMError::report_and_die(thread=<unavailable>, sig=<unavailable>, pc=<unavailable>, siginfo=<unavailable>, context=<unavailable>) at vmError.cpp:1275 [opt]
    frame #18: 0x000000010dffa144 libjvm.dylib`::JVM_handle_bsd_signal(sig=4, info=<unavailable>, ucVoid=<unavailable>, abort_if_unrecognized=1) at os_bsd_x86.cpp:781 [opt]
    frame #19: 0x000000010dff794a libjvm.dylib`signalHandler(sig=4, info=0x00007000078a7788, uc=0x00007000078a77f0) at os_bsd.cpp:2658 [opt]
    frame #20: 0x00007fff6ddcef5a libsystem_platform.dylib`_sigtramp + 26
    frame #21: 0x0000000118ad8f12
    frame #22: 0x000000010d96d81f libjvm.dylib`CodeCache::print_names(out=0x00000000000018ca) at codeCache.cpp:1745 [opt]
    frame #23: 0x000000010d9b8e25 libjvm.dylib`CompileBroker::print_heapinfo(out=0x00007000078a7c68, function=<unavailable>, granularity="4096") at compileBroker.cpp:2740 [opt]
    frame #24: 0x000000010daaa51b libjvm.dylib`DCmd::parse_and_execute(source=<unavailable>, out=0x00007000078a7c68, cmdline=<unavailable>, delim=' ', __the_thread__=0x00007ff65100d000) at diagnosticFramework.cpp:409 [opt]
    frame #25: 0x000000010d7faa8b libjvm.dylib`jcmd(op=<unavailable>, out=0x00007000078a7c68) at attachListener.cpp:204 [opt]
    frame #26: 0x000000010d7fa439 libjvm.dylib`attach_listener_thread_entry(thread=<unavailable>, __the_thread__=<unavailable>) at attachListener.cpp:391 [opt]
    frame #27: 0x000000010e193500 libjvm.dylib`JavaThread::thread_main_inner(this=0x00007ff65100d000) at thread.cpp:1868 [opt]
    frame #28: 0x000000010e192fd4 libjvm.dylib`JavaThread::run(this=0x00007ff65100d000) at thread.cpp:1851 [opt]
    frame #29: 0x000000010e18f71f libjvm.dylib`Thread::call_run(this=0x00007ff65100d000) at thread.cpp:400 [opt]
    frame #30: 0x000000010dff5344 libjvm.dylib`thread_native_entry(thread=0x00007ff65100d000) at os_bsd.cpp:692 [opt]
    frame #31: 0x00007fff6ddd8661 libsystem_pthread.dylib`_pthread_body + 340
    frame #32: 0x00007fff6ddd850d libsystem_pthread.dylib`_pthread_start + 377
    frame #33: 0x00007fff6ddd7bf9 libsystem_pthread.dylib`thread_start + 13
  thread #15, stop reason = signal SIGSTOP
    frame #0: 0x00007fff6dc10d8a libsystem_kernel.dylib`__semwait_signal + 10
    frame #1: 0x00007fff6db8b724 libsystem_c.dylib`nanosleep + 199
    frame #2: 0x00007fff6db8b586 libsystem_c.dylib`sleep + 41
    frame #3: 0x000000010dff7390 libjvm.dylib`os::infinite_sleep() at os_bsd.cpp:2231 [opt]
    frame #4: 0x000000010e21bab9 libjvm.dylib`VMError::report_and_die(id=<unavailable>, message=<unavailable>, detail_fmt=<unavailable>, detail_args=0x00007000079a8d80, thread=0x00007ff65100e800, pc=0x0000000000000000, siginfo=0x0000000000000000, context=0x0000000000000000, filename="/scratch/mesos/slaves/07fc96ef-bf4d-487f-b22f-a84e49f5f44a-S21444/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/47a2dfd6-c3c0-4679-ad75-c8b6294b030e/runs/c328beaa-d4dd-4ed2-bf5c-8d1b35ae95f9/workspace/open/src/hotspot/share/memory/heap.cpp", lineno=71, size=0) at ostream.hpp:0 [opt]
    frame #5: 0x000000010e21c4f1 libjvm.dylib`VMError::report_and_die(thread=<unavailable>, context=0x0000000000000000, filename=<unavailable>, lineno=<unavailable>, message=<unavailable>, detail_fmt=<unavailable>, detail_args=0x00007000079a8d80) at vmError.cpp:1294 [opt]
    frame #6: 0x000000010da3037d libjvm.dylib`report_vm_error(file="/scratch/mesos/slaves/07fc96ef-bf4d-487f-b22f-a84e49f5f44a-S21444/frameworks/1735e8a2-a1db-478c-8104-60c8b0af87dd-0196/executors/47a2dfd6-c3c0-4679-ad75-c8b6294b030e/runs/c328beaa-d4dd-4ed2-bf5c-8d1b35ae95f9/workspace/open/src/hotspot/share/memory/heap.cpp", line=71, error_msg="assert(beg < end && end <= _number_of_committed_segments) failed", detail_fmt="interval end   out of bounds") at debug.cpp:230 [opt]
    frame #7: 0x000000010dbcb91d libjvm.dylib`CodeHeap::mark_segmap_as_used(this=0x00007ff64f80f0d0, beg=203241, end=9223372036854978961) at heap.cpp:71 [opt]
    frame #8: 0x000000010dbcce85 libjvm.dylib`CodeHeap::merge_right(this=0x00007ff64f80f0d0, a=<unavailable>) at heap.cpp:412 [opt]
    frame #9: 0x000000010dbccd89 libjvm.dylib`CodeHeap::insert_after(this=0x00007ff64f80f0d0, a=0x0000000118a6e780, b=0x0000000118a9a480) at heap.cpp:398 [opt]
    frame #10: 0x000000010dbcc97e libjvm.dylib`CodeHeap::deallocate(this=0x00007ff64f80f0d0, p=0x0000000118a9a490) at heap.cpp:251 [opt]
    frame #11: 0x000000010d9696bc libjvm.dylib`CodeCache::free(cb=0x0000000118a9a490) at codeCache.cpp:570 [opt]
    frame #12: 0x000000010d962968 libjvm.dylib`BufferBlob::free(blob=0x0000000118a9a490) at codeBlob.cpp:269 [opt]
    frame #13: 0x000000010d964776 libjvm.dylib`CodeBuffer::free_blob(this=0x00007000079aa6f8) at codeBuffer.cpp:221 [opt]
    frame #14: 0x000000010d904e9c libjvm.dylib`ciEnv::register_method(this=<unavailable>, target=<unavailable>, entry_bci=-1, offsets=0x00007000079aa650, orig_pc_offset=128, code_buffer=<unavailable>, frame_words=20, oop_map_set=0x00007ff65b7af6c0, handler_table=0x00007000079aa8d8, inc_table=0x00007000079aa8f8, compiler=0x00007ff64fa6bcd0, has_unsafe_access=<unavailable>, has_wide_vectors=<unavailable>, rtm_state=NoRTM) at ciEnv.cpp:1040 [opt]
    frame #15: 0x000000010d9a31ac libjvm.dylib`Compile::Compile(this=0x00007000079a9d98, ci_env=0x0000700000000080, compiler=0x00007ff64fa6bcd0, target=0x0000000000000000, osr_bci=20, subsume_loads=<unavailable>, do_escape_analysis=<unavailable>, eliminate_boxing=<unavailable>, directive=0x00007ff64f8f5a60) at compile.cpp:938 [opt]
    frame #16: 0x000000010d8ced3d libjvm.dylib`C2Compiler::compile_method(this=0x00007ff64fa6bcd0, env=0x00007000079aab30, target=0x00007ff65a1cfa40, entry_bci=-1, directive=0x00007ff64f8f5a60) at c2compiler.cpp:109 [opt]
    frame #17: 0x000000010d9b65eb libjvm.dylib`CompileBroker::invoke_compiler_on_method(task=0x00007ff6565cd6c0) at compileBroker.cpp:2120 [opt]
    frame #18: 0x000000010d9b5e72 libjvm.dylib`CompileBroker::compiler_thread_loop() at compileBroker.cpp:1807 [opt]
    frame #19: 0x000000010e193500 libjvm.dylib`JavaThread::thread_main_inner(this=0x00007ff65100e800) at thread.cpp:1868 [opt]
    frame #20: 0x000000010e192fd4 libjvm.dylib`JavaThread::run(this=0x00007ff65100e800) at thread.cpp:1851 [opt]
    frame #21: 0x000000010e18f71f libjvm.dylib`Thread::call_run(this=0x00007ff65100e800) at thread.cpp:400 [opt]
    frame #22: 0x000000010dff5344 libjvm.dylib`thread_native_entry(thread=0x00007ff65100e800) at os_bsd.cpp:692 [opt]
    frame #23: 0x00007fff6ddd8661 libsystem_pthread.dylib`_pthread_body + 340
    frame #24: 0x00007fff6ddd850d libsystem_pthread.dylib`_pthread_start + 377
    frame #25: 0x00007fff6ddd7bf9 libsystem_pthread.dylib`thread_start + 13

Thanks Lutz. I've verified that your fix is not part of that build: http://hg.openjdk.java.net/jdk/jdk/file/642346a11059/src/hotspot/share/code/codeHeapState.cpp
23-01-2019
Well, what should I say? If JDK-8216314 is not included in the tested VM (fastdebug build 13-internal+0-jdk13-jdk.108), Tobias' analysis is correct. Otherwise, the basic assumptions (holding the CodeCache_lock protects against concurrent modifications) that led to the fix have to be questioned.
23-01-2019
Closing this as duplicate of JDK-8216314 for now. Please re-open if it shows up again.
23-01-2019
So according to the stack traces from the core file, thread #14 crashed with SIGILL because CodeCache::print_names called into 0x118ad8f11 which is random/dead/overwritten code cache space (this is due to JDK-8216314). Error reporting in this thread then crashed as well, because the stack is screwed up (see frame #9). That's expected. Now thread #15 also crashed in CodeHeap::mark_segmap_as_used() because we hit the assert(beg < end && end <= _number_of_committed_segments). We've never seen this assert before but since we are executing some random code in thread #14, in theory anything could happen. I would propose to close this as duplicate of JDK-8216314 if it does not show up after the fix went in. [~lucy], any comments here?
23-01-2019
[~lmesnik], I am truly sorry, but I don't understand what you were trying to say. could you please explain it clearly? I did though understand what JVM is already an inconsistent state before this crash happens. can it be what the crash described in JDK-8216314 is the cause of this "another" crash? or have you observed this crash on the builds w/ the fix for 8216314?
23-01-2019