JDK-8213014 : Crash in CompileBroker::make_thread due to OOM
  • Type: Bug
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 11,12
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2018-10-26
  • Updated: 2019-09-12
  • Resolved: 2018-11-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 12
11.0.6-oracleFixed 12 b19Fixed
Related Reports
Relates :  
JDK-8198756 - Lazy allocation of compiler threads
Description
JDK12 build generated JVM crash
==
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  SIGSEGV (0xb) at pc=0x00002ac051c274c0, pid=130424, tid=130429
#
# JRE version: Java(TM) SE Runtime Environment (12.0) (build 12-internal+0-2018-10-26-0613010.fmatte.jdk)
# Java VM: Java HotSpot(TM) 64-Bit Server VM (12-internal+0-2018-10-26-0613010.fmatte.jdk, mixed mode, tiered, compressed oops, serial gc, linux-amd64)
# Problematic frame:
# V  [libjvm.so+0x63f4c0]  CompileBroker::make_thread(_jobject*, CompileQueue*, AbstractCompiler*, bool, Thread*)+0x240
#
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" (or dumping to /scratch/fairoz/JDK/experimental/8208372/JDK12/jdk/open/make/core.130424)
==
Comments
Fix Request (11u) This patch fixes the corner case in compiler memory management and keeps codebases in sync (I see 11.0.6-oracle). Patch applies cleanly to 11u, passes tier1, tier2.
10-09-2019
ILW = Crash while creating compiler/sweeper threads (instead of graceful bailout), on systems with low memory / process limit, -XX:-UseDynamicNumberOfCompilerThreads = MLM = P4
30-10-2018
@David: Right, the compiler_thread argument is not needed. I've removed it and updated the webrev. Triggering a build failure by setting a low ulimit does work for me but the VM handles it gracefully by bailing out through vm_exit_during_initialization. The difficulty is to make it fail at the creation of the sweeper thread which passes comp == NULL and will therefore trigger the crash. @Fairoz: Okay, thanks for trying to verify! I can reproduce the failure by modifying the VM to always fail in make_thread when the sweeper thread is created. I'll send the webrev for review. Please file a new bug if this problem shows up again after the fix is in. http://cr.openjdk.java.net/~thartmann/8213014/webrev.00/
30-10-2018
If comp != NULL -> compiler_thread == true and comp == NULL -> compiler_thread == false then the bool compiler_thread seems to be superfluous? FWIW I could not trigger a build failure even by reducing ulimit-u down to 1024. But of course it will depend on what else is running, by the same user, concurrently with the build.
30-10-2018
Sure Tobias, i will do that, the machine on which i got this issue is located in Denver, it is still down. I will try once it is up. Thanks,
29-10-2018
This code was heavily modified by JDK-8198756. I think the problem is that we are dereferencing 'comp' which can be NULL if we are creating the sweeper thread: // First release lock before aborting VM. if (thread == NULL || thread->osthread() == NULL) { if (UseDynamicNumberOfCompilerThreads && comp->num_compiler_threads() > 0) { [...] This patch should fix the problem(s): http://cr.openjdk.java.net/~thartmann/8213014/webrev.00/ Fairoz, could you please verify?
29-10-2018
We shouldn't be failing the java.lang.Thread creation, but at the same time we do not check whether it does fail: jobject thread_handle = JNIHandles::make_global(create_thread_oop(name_buffer, THREAD)); there's no following null check on the thread_handle before it gets passed into make_thread and is used with the new created Compiler/Sweeper thread.
28-10-2018
CompilerThread creation already follows the expected pattern of checking the thread and osThread for NULL: // At this point it may be possible that no osthread was created for the // JavaThread due to lack of memory. We would have to throw an exception // in that case. However, since this must work and we do not allow // exceptions anyway, check and abort if this fails. But first release the // lock. if (thread != NULL && thread->osthread() != NULL) { so where exactly are we hitting the SEGV?
28-10-2018
The problem was likely caused by too low a process limit - I see it was 4096 in the hs-err log - which may be low during a build with many processes and JVMs running.
28-10-2018
There is a lot of space in Java heap (from hs_err file): def new generation total 9792K, used 348K [0x00000000e0000000, 0x00000000e0aa0000, 0x00000000eaaa0000) We should not fail JavaThread object allocation. It seems we need to check result of os::create_thread() in JavaThread() (compiler thread is subclass of JavaThread): http://hg.openjdk.java.net/jdk/jdk/file/91a57277c419/src/hotspot/share/runtime/thread.cpp#l1689 And exit gracefully like next: http://hg.openjdk.java.net/jdk/jdk/file/91a57277c419/src/hotspot/share/runtime/thread.cpp#l3779
26-10-2018
I could see another error after that * For target support_jmods_java.management.rmi.jmod: [0.131s][warning][os,thread] Failed to start thread - pthread_create failed (EAGAIN) for attributes: stacksize: 1024k, guardsize: 0k, detached. Error occurred during initialization of VM java.lang.OutOfMemoryError: unable to create native thread: possibly out of memory or process/resource limits reached * All command lines available in /scratch/fairoz/JDK/experimental/8208372/JDK12/jdk/build/linux-x64/make-support/failure-logs. === End of repeated output === OOM due to "unable to create native thread"? does this can result JVM to crash?
26-10-2018