JDK-8175342 : assert(InstanceKlass::cast(k)->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation
  • Type: Bug
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 9,10
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • Submitted: 2017-02-22
  • Updated: 2017-08-29
  • Resolved: 2017-03-17
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10
10 b21Fixed
Related Reports
Duplicate :  
JDK-8175287 - tools/launcher/TooSmallStackSize.java fails with StackOverflowError on linux-arm64
Relates :  
JDK-8176768 - hotspot ignores PTHREAD_STACK_MIN when creating new threads
Relates :  
JDK-8176797 - [TESTBUG] tools/launcher/Settings.java -Xss size is too small
Relates :  
JDK-8169373 - Work around linux NPTL stack guard error
Relates :  
JDK-8140520 - segfault on solaris-amd64 with "-XX:VMThreadStackSize=1" option
Relates :  
JDK-8175287 - tools/launcher/TooSmallStackSize.java fails with StackOverflowError on linux-arm64
Description
I saw this just once, on arm, during testing of a clean JDK10 repo:

#  Internal Error (/scratch/opt/jprt/T/P1/215921.cplummer/s/hotspot/src/share/vm/utilities/exceptions.cpp:230), pid=72834, tid=72835
#  assert(InstanceKlass::cast(k)->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation

---------------  T H R E A D  ---------------

Current thread (0x0000ffffa0019800):  JavaThread "Unknown thread" [_thread_in_vm, id=72835, stack(0x0000ffffa6590000,0x0000ffffa65c0000)]

Stack: [0x0000ffffa6590000,0x0000ffffa65c0000],  sp=0x0000ffffa65bdbe0,  free space=182k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xff26e4]  VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x140;;  VMError::report_and_die(int, char const*, char const*, std::__va_list, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x140
V  [libjvm.so+0xff32ec]  VMError::report_and_die(Thread*, char const*, int, char const*, char const*, std::__va_list)+0x54;;  VMError::report_and_die(Thread*, char const*, int, char const*, char const*, std::__va_list)+0x54
V  [libjvm.so+0x6a4e08]  report_vm_error(char const*, int, char const*, char const*, ...)+0xe0;;  report_vm_error(char const*, int, char const*, char const*, ...)+0xe0
V  [libjvm.so+0x751858]  Exceptions::throw_stack_overflow_exception(Thread*, char const*, int, methodHandle const&)+0x344;;  Exceptions::throw_stack_overflow_exception(Thread*, char const*, int, methodHandle const&)+0x344
V  [libjvm.so+0x947cdc]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x2c4;;  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x2c4
V  [libjvm.so+0xd31bd8]  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*), JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x20;;  os::os_exception_wrapper(void (*)(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*), JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x20
V  [libjvm.so+0x903868]  InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0x19c;;  InstanceKlass::call_class_initializer_impl(instanceKlassHandle, Thread*)+0x19c
V  [libjvm.so+0x903a4c]  InstanceKlass::call_class_initializer(Thread*)+0x90;;  InstanceKlass::call_class_initializer(Thread*)+0x90
V  [libjvm.so+0x903dc8]  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x360;;  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x360
V  [libjvm.so+0x90420c]  InstanceKlass::initialize(Thread*)+0xec;;  InstanceKlass::initialize(Thread*)+0xec
V  [libjvm.so+0x903bb0]  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x148;;  InstanceKlass::initialize_impl(instanceKlassHandle, Thread*)+0x148
V  [libjvm.so+0x90420c]  InstanceKlass::initialize(Thread*)+0xec;;  InstanceKlass::initialize(Thread*)+0xec
V  [libjvm.so+0xf5dab4]  initialize_class(Symbol*, Thread*)+0x44;;  initialize_class(Symbol*, Thread*)+0x44
V  [libjvm.so+0xf691dc]  Threads::initialize_java_lang_classes(JavaThread*, Thread*)+0x80;;  Threads::initialize_java_lang_classes(JavaThread*, Thread*)+0x80
V  [libjvm.so+0xf6b794]  Threads::create_vm(JavaVMInitArgs*, bool*)+0x428;;  Threads::create_vm(JavaVMInitArgs*, bool*)+0x428
V  [libjvm.so+0xa0ed98]  JNI_CreateJavaVM+0xb8;;  JNI_CreateJavaVM+0xb8
C  [libjli.so+0x6b04]  JavaMain+0x80;;  JavaMain+0x80
C  [libpthread.so.0+0x7e2c]  start_thread+0xb0
C  [libc.so.6+0xc8c40]  clone+0x70
Comments
Sigh, found a 3rd bug, but at least this time it's a test bug. Only shows up after this fix for this CR is in place. Created JDK-8176797.
15-03-2017
Yes - the same as what happens when we create a JavaThread in the VM.
15-03-2017
So you are saying rather than having the launcher add the default guard page size to the stack size, it should instead call pthread_attr_setguardsize() to set the guard size to 0. I'll give this a try.
15-03-2017
Chris: yes as I was explaining we add the guard size to the requested stack so that when glibc steals it back we have what we originally wanted. Looking at JDK-8169373 only non-JavaThreads have the fix applied to them as only they should be having a glibc guard page set. So it seems to me the correct fix here is for the launcher to also not request a guard page as the thread it creates will be a JavaThread.
15-03-2017
I removed the comment regarding the PTHREAD_STACK_MIN fix and moved it to JDK-8176768 where it belongs. The fix for this CR seems to be resolved on unix systems by adding _SC_PAGESIZE to the stack size specified in ContinueInNewThread0().
15-03-2017
I've added 9 to affects version. Need to decide what the priority should be. Impact is High. Work around is Low. Not sure about likelihood. On thunder devices, if you try a really small stack size it says to use at least 256k. If you then use 256k you will see the problem. So in this respect is seems somewhat likely, although at the moment we've only seen this on xgene systems, so maybe not that likely. With medium likelihood it's a p2, but low likelihood it's a p4. Leaving as a p3 for now.
14-03-2017
It looks like we already account for the guard_size in os::create_thread(): // Calculate stack size if it's not specified by caller. size_t stack_size = os::Posix::get_initial_stack_size(thr_type, req_stack_size); // In the Linux NPTL pthread implementation the guard size mechanism // is not implemented properly. The posix standard requires adding // the size of the guard pages to the stack size, instead Linux // takes the space out of 'stacksize'. Thus we adapt the requested // stack_size by the size of the guard pages to mimick proper // behaviour. stack_size = align_size_up(stack_size + os::Linux::default_guard_size(thr_type), vm_page_size()); pthread_attr_setstacksize(&attr, stack_size); os::Linux::default_guard_size() returns the page size. However, this issue is turning up with the thread created in ContinueInNewThread0(), which does not account for guard size. So I'm now thinking the best fix is to just add the guard size to the stack size used by ContinueInNewThread0() just like we are doing in os::create_thread().
14-03-2017
@Calvin: The sizes of are various zones/guards and the actual stack usage is different on each platform, so something like this only turning up on solaris-sparc would not be surprising.
14-03-2017
@David: I looked a bit more at pthread_attr_getguardsize(). It says: "The default guard size is the same as the system page size." The changes were added as part of JDK-8169373 (which I think you are familiar with) which says: " In the Linux NPTL pthread implementation the guard size mechanism is not implemented properly. The posix standard requires to add the size of the guard pages to the stack size, instead Linux takes the space out of 'stacksize'." Given the default guard size being the page size, I think we should just always add the page size to _java_thread_min_stack_allowed. It's simple and should always get the job done. Worse case is sometime it's overkill, but that's better than sometimes not adding guard size when we should be.
14-03-2017
@Calvin: same assert but for a different root cause. When I see this on aarch64, it only happens because -Xss is set to too small of a stack size (it is set to a small size that should be disallowed). Your case is not setting -Xss. It looks like you are hitting infinite recursion, so eventually there is an attempt to throw SOE for that reason, and in this case the assert is not because _java_thread_min_stack_allowed is too small, but because one of the guard zones has failed us. So the assert is misleading. It might be best to create a new CR for it.
14-03-2017
I created JDK-8176768 for the xgene 8mb stack size issue.
14-03-2017
I think I know why xgene is having an issue. The stack size is set according to -Xss by ContinueInNewThread0 in java_md_solinux.c: int ContinueInNewThread0(int (JNICALL *continuation)(void *), jlong stack_size, void * args) { ... if (stack_size > 0) { pthread_attr_setstacksize(&attr, stack_size); } ... if (pthread_create(&tid, &attr, (void *(*)(void*))continuation, (void*)args) == 0) { ... On xgene pthread_attr_setstacksize() is returning EINVAL. If I then call pthread_attr_getstacksize (not shown), it still returns 8mb, so this is the size the new thread (the one that replaces the primordial thread) is created with. The man page says EINVAL can happens if the stack size is less than PTHREAD_STACK_MIN, which it looks like it is in this case. PTHREAD_STACK_MIN is 128k for aarch64 and we are trying to set the stack size to 72k, so this looks like a separate bug to be fixed. Now as to the cause of this bug on Thunder, *after* the thread is created using the -Xss size, current_stack_region() is called to figure out how much usable stack we have. if (pthread_attr_getstack(&attr, (void **)bottom, size) != 0) { fatal("Cannot locate current stack attributes!"); } // Work around NPTL stack guard error. size_t guard_size = 0; rslt = pthread_attr_getguardsize(&attr, &guard_size); if (rslt != 0) { fatal("pthread_attr_getguardsize failed with error = %d", rslt); } *bottom += guard_size; *size -= guard_size; On thunder guard_size is 64k, so this *subtracts* from the 256k size that pthread_attr_getstack() returns. It does not add to it. This is a calculation done *after* we already have a thread with a set stack size, so this is memory we are losing from that stack, and it hasn't been accounted for when we calculated _java_thread_min_stack_allowed. So as far as hotspot is concerned, we have a 192k of stack to use, from which we still need to subtract the red, yellow, and shadow zones, which leaves us with nothing.
14-03-2017
Sorry Chris I'm confused. The glibc guard page bug handling is supposed to expand the requested stack by the amount of the guard page so that when glibc incorrectly places it in our allocated stack we still have the amount of stack we requested. So if we ask for a stack of 32*K + 192K (for r/y/s pages) + 64K for glibc guard, we should get an actual stack of 320KB (5 OS pages), which when we remove the guards leaves us with one OS page of 64KB which is > the 32KB minimum we said we needed. With regard to 8MB stack usage, again that should only be for the primordial process thread. So even if a Ubuntu variant perhaps this OS has been configured differently? I wonder if there is a way to tell?
14-03-2017
xgene is running Ubuntu: xgene host: Linux <hostname> 3.13.0-62-generic #102-Ubuntu SMP Tue Aug 11 15:39:31 UTC 2015 aarch64 aarch64 aarch64 GNU/Linux thunder host: Linux <hostname> 3.18.0-02413-ga5b6b29 #1 SMP Fri Jul 10 09:21:47 PDT 2015 aarch64 aarch64 aarch64 GNU/Linux Page size and actual stack usage are pretty much independent. Having a bigger page size should not result in increased stack usage. The initial value of _java_thread_min_stack_allowed is suppose to (I believe) just take into account the actual stack usage by the JVM (not counting any os or clib usage). The sizes of various zones/guards are added to it at runtime. Then the whole things is rounded up to the page size. Since everything added to it already has been rounded up to the page size, this is essentially the same as rounding up the initial _java_thread_min_stack_allow to the page size. Where page size does come into play is with various zone/guard pages. These have to be multiples of the page size. Although their DEFAULT values are specified in terms of the number of pages, this was always intended to be 4k pages. When we started dealing with 64k pages, we decided the best solution was to still treat the size as being specified in 4k pages, but round up to the actual page size. There's nothing functionally wrong with this, and once again makes sense to do since the that amount of space we need in each of these zones is not actually a function of the page size. It's a function of actual stack usage. But since they need to be specified in increments of page size, we have to round them all up.
14-03-2017
I know there has been a lot of discussion in this area in the past but I thought by now we would have no problem handling 64K pages! size_t os::Posix::_java_thread_min_stack_allowed = (32 DEBUG_ONLY(+ 4)) * K; Surely this needs to be calculated as minimum_number_of_pages * page-size ? Again something I'm sure has been discussed a number of times in the past. The 8MB stack should only be for the primordial process thread. Is xgene running some special Linux version that doesn't implement pthreads properly ??
14-03-2017
The problem is only on Thunder devices, where this assert occurs every time you pass in -Xss256, which is the minimum allowed stack size. Thunder uses 64k pages. On xgene, which uses 4k pages, -Xss72 is the smallest allowed, and it passes fine every time. However, the page size is not really the issue. The cause seems to be that _java_thread_min_stack_allowed not accounting for how we determine the bottom and size of the stack in os::current_stack_region(). First, our _java_thread_min_stack_allowed calculation: size_t os::Posix::_java_thread_min_stack_allowed = (32 DEBUG_ONLY(+ 4)) * K; #define DEFAULT_STACK_YELLOW_PAGES (2) #define DEFAULT_STACK_RED_PAGES (1) #define DEFAULT_STACK_SHADOW_PAGES (5 DEBUG_ONLY(+1)) The 3 yellow, red, and shadow zones sizes are given in 4k page sizes, but we round each one up to the os page size, so they each end up being 64k on Thunder devices. _java_thread_min_stack_allowed starts at 36k, but we add to it each of the 3 zones, and then round up to a 64k boundary again, so you end up with 256k as the minimum allowed stack size. This is the size our thread actually gets set to, but then when determining the bottom of the stack and its size (and therefore how much usable stack is available), we use the following code in current_stack_region(): // Work around NPTL stack guard error. size_t guard_size = 0; rslt = pthread_attr_getguardsize(&attr, &guard_size); if (rslt != 0) { fatal("pthread_attr_getguardsize failed with error = %d", rslt); } *bottom += guard_size; *size -= guard_size; On the Thunder devices, guard_size is always 64k. So although the thread stack is indeed 256k, this reduces what we think we can use to 192k. But then from that we subtract the red, yellow, and shadow zones sizes. Since the size of the 3 zones is 192k, we effectively have no usable stack. Thus the first time we call os::stack_shadow_pages_available(), it returns false because we are already well into the shadow zone. Then we try to throw a SOE, and this triggers the assert. One possible fix is to always add guard_size() to _java_thread_min_stack_allowed. However, I've noticed guard_size() is not always the same. On my desktop x86 with 4k pages and on xgene with 4k pages, for most threads it is 0k, but sometimes it is 4k (this is within the same running java app, not different apps). So maybe we just always add the page size to _java_thread_min_stack_allowed, but what happens if guard_size is actually more than one page? Also, since the guard_size does not seem to be the same for every thread, what happens if the main thread we use to compute _java_thread_min_stack_allowed has guard_size == 0, but other threads have a guard_size == 64k? BTW, I noticed a different issue on xgene. It looks like our pthread_attr_setstacksize() calls are being ignored, because the thread always ends up being 8mb (which is the default thread stack size according to ulimit). For this reason we aren't effectively testing if _java_thread_min_stack_allowed is being calculated properly.
14-03-2017
[~dcubed] - The fix for JDK-8173474 only applies to open aarch64 port, not the oracle contributed arm64 port, so it's not relevant that it's not in the repo I am testing. Likely JDK-8170655 introduced the problem.
07-03-2017
[~dcubed] - I was using JDK10/hs.
22-02-2017
The fix for JDK-8173474 is present in JDK10/JDK10: http://hg.openjdk.java.net/jdk10/jdk10/hotspot/rev/f3b3d77a1751 However, it is not present in JDK10/hs: http://hg.openjdk.java.net/jdk10/hs/hotspot/rev/f3b3d77a1751 [~cjplummer] - I don't know if your "JDK10" is based on JDK10/JDK10 or on JDK10/hs.
22-02-2017
Copied from JDK-8175287: There were a couple of recent fixes that tweaked these sizes on ARM64: JDK-8170655: [posix] Fix minimum stack size computations $ hg diff -r 548cb3b7b713 -r c7a256349729 src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp diff -r 548cb3b7b713 -r c7a256349729 src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp --- a/src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp Tue Nov 08 16:30:36 2016 +0100 +++ b/src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp Mon Dec 19 15:48:57 2016 -0800 @@ -473,8 +473,10 @@ //////////////////////////////////////////////////////////////////////////////// // thread stack -size_t os::Posix::_compiler_thread_min_stack_allowed = 64 * K; -size_t os::Posix::_java_thread_min_stack_allowed = 64 * K; +// Minimum usable stack sizes required to get to user code. Space for +// HotSpot guard pages is added later. +size_t os::Posix::_compiler_thread_min_stack_allowed = 32 * K; +size_t os::Posix::_java_thread_min_stack_allowed = 32 * K; size_t os::Posix::_vm_internal_thread_min_stack_allowed = 64 * K; // return default stack size for thr_type JDK-8173474: AArch64: fix reported -Xss minimum $ hg diff -r 2503995d745e -r f3b3d77a1751 src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp diff -r 2503995d745e -r f3b3d77a1751 src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp --- a/src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp Thu Jan 12 16:27:40 2017 +0000 +++ b/src/os_cpu/linux_aarch64/vm/os_linux_aarch64.cpp Fri Jan 27 07:56:13 2017 -0500 <snip> @@ -534,9 +475,9 @@ // Minimum usable stack sizes required to get to user code. Space for // HotSpot guard pages is added later. -size_t os::Posix::_compiler_thread_min_stack_allowed = 32 * K; -size_t os::Posix::_java_thread_min_stack_allowed = 32 * K; -size_t os::Posix::_vm_internal_thread_min_stack_allowed = 64 * K; +size_t os::Posix::_compiler_thread_min_stack_allowed = 72 * K; +size_t os::Posix::_java_thread_min_stack_allowed = 72 * K; +size_t os::Posix::_vm_internal_thread_min_stack_allowed = 72 * K; // return default stack size for thr_type size_t os::Posix::default_stack_size(os::ThreadType thr_type) { Update: Since the most recent of these fixes, JDK-8173474, is listed as fixed in JDK9-B155 and this test still fails there. It is likely that these sizes need to be tweaked for product builds.
22-02-2017
JDK-8175287 seems related.
22-02-2017
RULE "tools/launcher/Settings.java" Crash Internal Error ...exceptions.cpp...assert(InstanceKlass::cast(k)->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation RULE "tools/launcher/TooSmallStackSize.java" Crash Internal Error ...exceptions.cpp...assert(InstanceKlass::cast(k)->is_initialized()) failed: need to increase java_thread_min_stack_allowed calculation http://aurora.us.oracle.com/functional/faces/RunDetails.xhtml?names=1950401.rbt-chris.plummer-hs6.clean-20170216-2242-34104-441
22-02-2017
Probably related to changes for JDK-8140520.
22-02-2017