Bug ID: JDK-8217765 Internal Error (javaCalls.cpp:61) guarantee(thread->can_call

JDK-8217765 : Internal Error (javaCalls.cpp:61) guarantee(thread->can_call_java()) failed

Type: Bug
Component: hotspot
Sub-Component: runtime
Affected Version: 11.0.2,12,13

Priority: P2
Status: Closed
Resolution: Fixed

Submitted: 2019-01-24
Updated: 2021-10-28
Resolved: 2019-02-21

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 11	JDK 12	JDK 13
11.0.4-oracleFixed	12.0.2Fixed	13 b10Fixed

Related Reports

Duplicate :	JDK-8276101 - guarantee(thread->can_call_java()) failed: cannot make java calls from the nativ
Relates :	JDK-8261395 - C1 crash "cannot make java calls from the native compiler"
Relates :	JDK-8260509 - JNI-originated "guarantee(thread->can_call_java()) failed: cannot make java calls from the native compiler" crash

Description

A DESCRIPTION OF THE PROBLEM :
#
# A fatal error has been detected by the Java Runtime Environment:
#
#  Internal Error (javaCalls.cpp:61), pid=3356, tid=3370
#  guarantee(thread->can_call_java()) failed: cannot make java calls from the native compiler
#
# JRE version: OpenJDK Runtime Environment (11.0.2+9) (build 11.0.2+9)
# Java VM: OpenJDK 64-Bit Server VM (11.0.2+9, mixed mode, tiered, compressed oops, parallel gc, linux-amd64)
# Core dump will be written. Default location: Core dumps may be processed with "/usr/libexec/abrt-hook-ccpp %s %c %p %u %g %t e" (or dumping to /usr/share/tomcat/core.3356)
#
# If you would like to submit a bug report, please visit:
#   http://bugreport.java.com/bugreport/crash.jsp
#

---------------  S U M M A R Y ------------

Command Line: -Xms4500m -Xmx6000m -Xss228k -XX:NewRatio=4 -XX:+PrintGC -XX:+UseParallelGC -XX:-UseGCOverheadLimit -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -DGAT_RUN_JOBS= -Dsun.net.client.defaultReadTimeout=60000 -Dsun.net.client.defaultConnectTimeout=30000 -Dhttp.keepAlive=false -DCLOUD_TYPE=gce -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start

Host: Intel(R) Xeon(R) CPU @ 2.30GHz, 2 cores, 7G, CentOS release 6.10 (Final)
Time: Thu Jan 24 02:46:32 2019 UTC elapsed time: 430 seconds (0d 0h 7m 10s)

---------------  T H R E A D  ---------------

Current thread (0x00007f262c0b6800):  JavaThread "C1 CompilerThread0" daemon [_thread_in_vm, id=3370, stack(0x00007f261420c000,0x00007f261430d000)]


Current CompileTask:
C1: 430334 14336       2       com.baker.gat.docs.Doc$DocResourceComparator::compareDetails (522 bytes)

Stack: [0x00007f261420c000,0x00007f261430d000],  sp=0x00007f26143094d0,  free space=1013k
Native frames: (J=compiled Java code, A=aot compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xe27304]  VMError::report_and_die(int, char const*, char const*, __va_list_tag*, Thread*, unsigned char*, void*, void*, char const*, int, unsigned long)+0x314
V  [libjvm.so+0xe27f2f]  VMError::report_and_die(Thread*, void*, char const*, int, char const*, char const*, __va_list_tag*)+0x2f
V  [libjvm.so+0x699736]  report_vm_error(char const*, int, char const*, char const*, ...)+0x106
V  [libjvm.so+0x8822d6]  JavaCallWrapper::JavaCallWrapper(methodHandle const&, Handle, JavaValue*, Thread*)+0x286
V  [libjvm.so+0x884604]  JavaCalls::call_helper(JavaValue*, methodHandle const&, JavaCallArguments*, Thread*)+0x1d4
V  [libjvm.so+0x8833ad]  JavaCalls::call_special(JavaValue*, Handle, Klass*, Symbol*, Symbol*, Handle, Handle, Thread*)+0x1bd
V  [libjvm.so+0xd857e6]  SystemDictionary::validate_protection_domain(InstanceKlass*, Handle, Handle, Thread*)+0x226
V  [libjvm.so+0xd87634]  SystemDictionary::resolve_instance_class_or_null(Symbol*, Handle, Handle, Thread*)+0xa84
V  [libjvm.so+0xd89dc2]  SystemDictionary::resolve_or_fail(Symbol*, Handle, Handle, bool, Thread*)+0x52
V  [libjvm.so+0x68debd]  ConstantPool::klass_at_impl(constantPoolHandle const&, int, bool, Thread*)+0xbd
V  [libjvm.so+0x8608c0]  ConstantPool::klass_at(int, Thread*)+0x60
V  [libjvm.so+0x85af72]  InstanceKlass::has_nest_member(InstanceKlass*, Thread*) const+0x122
V  [libjvm.so+0x85b3a6]  InstanceKlass::nest_host(Symbol*, Thread*) [clone .part.112]+0x316
V  [libjvm.so+0x85b665]  InstanceKlass::nest_host(Symbol*, Thread*)+0x95
V  [libjvm.so+0x85b6d6]  InstanceKlass::has_nestmate_access_to(InstanceKlass*, Thread*)+0x66
V  [libjvm.so+0xca5e8d]  Reflection::verify_member_access(Klass const*, Klass const*, Klass const*, AccessFlags, bool, bool, Thread*)+0x27d
V  [libjvm.so+0xa63ddd]  LinkResolver::check_method_accessability(Klass*, Klass*, Klass*, methodHandle const&, Thread*)+0x5d
V  [libjvm.so+0xa64604]  LinkResolver::resolve_method(LinkInfo const&, Bytecodes::Code, Thread*)+0x144
V  [libjvm.so+0xa6a376]  LinkResolver::resolve_special_call_or_null(LinkInfo const&)+0xb6
V  [libjvm.so+0x577bd3]  ciEnv::lookup_method(ciInstanceKlass*, ciKlass*, Symbol*, Symbol*, Bytecodes::Code, constantTag)+0x163
V  [libjvm.so+0x577cf0]  ciEnv::get_method_by_index_impl(constantPoolHandle const&, int, Bytecodes::Code, ciInstanceKlass*)+0x110
V  [libjvm.so+0x5784a5]  ciEnv::get_method_by_index(constantPoolHandle const&, int, Bytecodes::Code, ciInstanceKlass*)+0x3a5
V  [libjvm.so+0x5a722a]  ciBytecodeStream::get_method(bool&, ciSignature**)+0x15a
V  [libjvm.so+0x4b9e86]  GraphBuilder::invoke(Bytecodes::Code)+0x36
V  [libjvm.so+0x4bc1ef]  GraphBuilder::iterate_bytecodes_for_block(int)+0x110f
V  [libjvm.so+0x4bdf0c]  GraphBuilder::iterate_all_blocks(bool)+0x8c
V  [libjvm.so+0x4be797]  GraphBuilder::try_inline_full(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x767
V  [libjvm.so+0x4bee60]  GraphBuilder::try_inline(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x1d0
V  [libjvm.so+0x4ba102]  GraphBuilder::invoke(Bytecodes::Code)+0x2b2
V  [libjvm.so+0x4bc1ef]  GraphBuilder::iterate_bytecodes_for_block(int)+0x110f
V  [libjvm.so+0x4bdf0c]  GraphBuilder::iterate_all_blocks(bool)+0x8c
V  [libjvm.so+0x4be797]  GraphBuilder::try_inline_full(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x767
V  [libjvm.so+0x4bee60]  GraphBuilder::try_inline(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x1d0
V  [libjvm.so+0x4ba102]  GraphBuilder::invoke(Bytecodes::Code)+0x2b2
V  [libjvm.so+0x4bc1ef]  GraphBuilder::iterate_bytecodes_for_block(int)+0x110f
V  [libjvm.so+0x4bdf0c]  GraphBuilder::iterate_all_blocks(bool)+0x8c
V  [libjvm.so+0x4be797]  GraphBuilder::try_inline_full(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x767
V  [libjvm.so+0x4bee60]  GraphBuilder::try_inline(ciMethod*, bool, bool, Bytecodes::Code, Instruction*)+0x1d0
V  [libjvm.so+0x4ba102]  GraphBuilder::invoke(Bytecodes::Code)+0x2b2
V  [libjvm.so+0x4bc1ef]  GraphBuilder::iterate_bytecodes_for_block(int)+0x110f
V  [libjvm.so+0x4bdffa]  GraphBuilder::iterate_all_blocks(bool)+0x17a
V  [libjvm.so+0x4bf847]  GraphBuilder::GraphBuilder(Compilation*, IRScope*)+0x2d7
V  [libjvm.so+0x4c3b6a]  IRScope::IRScope(Compilation*, IRScope*, int, ciMethod*, int, bool)+0x1ea
V  [libjvm.so+0x4c4796]  IR::IR(Compilation*, ciMethod*, int)+0xa6
V  [libjvm.so+0x4a5566]  Compilation::build_hir() [clone .part.93]+0x166
V  [libjvm.so+0x4a5db3]  Compilation::compile_java_method()+0x3a3
V  [libjvm.so+0x4a5f4f]  Compilation::compile_method()+0x12f
V  [libjvm.so+0x4a63af]  Compilation::Compilation(AbstractCompiler*, ciEnv*, ciMethod*, int, BufferBlob*, DirectiveSet*)+0x22f
V  [libjvm.so+0x4a6c8e]  Compiler::compile_method(ciEnv*, ciMethod*, int, DirectiveSet*)+0xae
V  [libjvm.so+0x645b4c]  CompileBroker::invoke_compiler_on_method(CompileTask*)+0x4fc
V  [libjvm.so+0x6476c8]  CompileBroker::compiler_thread_loop()+0x598
V  [libjvm.so+0xdc2b0d]  JavaThread::thread_main_inner()+0x21d
V  [libjvm.so+0xdc2eb7]  JavaThread::run()+0x377
V  [libjvm.so+0xc076a0]  thread_native_entry(Thread*)+0xf0


---------------  P R O C E S S  ---------------

Threads class SMR info:
_java_thread_list=0x00007f25fc052780, length=66, elements={
0x00007f262c010000, 0x00007f262c09b800, 0x00007f262c09f800, 0x00007f262c0b2000,
0x00007f262c0b4800, 0x00007f262c0b6800, 0x00007f262c0b8800, 0x00007f262c147000,
0x00007f262c15b000, 0x00007f25d0763000, 0x00007f25d0c3f000, 0x00007f25d0c40800,
0x00007f25d0efe800, 0x00007f25d041f800, 0x00007f25d0593000, 0x00007f25ec019800,
0x00007f25ec011000, 0x00007f25ec00f800, 0x00007f25ec00e000, 0x00007f25d0629800,
0x00007f25d03e5800, 0x00007f25d05f6800, 0x00007f25d0bb1000, 0x00007f25d1153000,
0x00007f25d1154800, 0x00007f25d115c800, 0x00007f25d115e800, 0x00007f25d1160800,
0x00007f25d1162800, 0x00007f25d1165000, 0x00007f25d1167000, 0x00007f25d1169000,
0x00007f25d116b000, 0x00007f25d116d000, 0x00007f25d1172000, 0x00007f25d1173000,
0x00007f262c3cf800, 0x00007f262c37f000, 0x00007f262c384000, 0x00007f262c385800,
0x00007f262c1e7000, 0x00007f262c212000, 0x00007f262c213800, 0x00007f262c3c9000,
0x00007f262c3ca000, 0x00007f262c3cc000, 0x00007f262c21e800, 0x00007f262c220000,
0x00007f262c222000, 0x00007f262c224800, 0x00007f262c216800, 0x00007f262c218800,
0x00007f262c21a800, 0x00007f262c21c800, 0x00007f262c0e0000, 0x00007f262c0e2800,
0x00007f262c0e4800, 0x00007f262c0e6800, 0x00007f262c11a800, 0x00007f262c11c800,
0x00007f262c11e800, 0x00007f25ec2ed800, 0x00007f25c416c800, 0x00007f25d40a5800,
0x00007f25d40a7000, 0x00007f25d40a8000
}

Java Threads: ( => current thread )
  0x00007f262c010000 JavaThread "main" [_thread_in_native, id=3360, stack(0x00007f2633ada000,0x00007f2633b13000)]
  0x00007f262c09b800 JavaThread "Reference Handler" daemon [_thread_blocked, id=3366, stack(0x00007f2604447000,0x00007f2604480000)]
  0x00007f262c09f800 JavaThread "Finalizer" daemon [_thread_blocked, id=3367, stack(0x00007f2630092000,0x00007f26300cb000)]
  0x00007f262c0b2000 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=3368, stack(0x00007f2630059000,0x00007f2630092000)]
  0x00007f262c0b4800 JavaThread "C2 CompilerThread0" daemon [_thread_in_native, id=3369, stack(0x00007f261430d000,0x00007f261440e000)]
=>0x00007f262c0b6800 JavaThread "C1 CompilerThread0" daemon [_thread_in_vm, id=3370, stack(0x00007f261420c000,0x00007f261430d000)]
  0x00007f262c0b8800 JavaThread "Sweeper thread" daemon [_thread_blocked, id=3371, stack(0x00007f26141d3000,0x00007f261420c000)]
  0x00007f262c147000 JavaThread "Service Thread" daemon [_thread_blocked, id=3372, stack(0x00007f2630020000,0x00007f2630059000)]
  0x00007f262c15b000 JavaThread "Common-Cleaner" daemon [_thread_blocked, id=3374, stack(0x00007f2614099000,0x00007f26140d2000)]
  0x00007f25d0763000 JavaThread "Abandoned connection cleanup thread" daemon [_thread_blocked, id=3378, stack(0x00007f25f0aa5000,0x00007f25f0ade000)]
  0x00007f25d0c3f000 JavaThread "BakerScheduler_Worker-1" [_thread_blocked, id=3379, stack(0x00007f25f075c000,0x00007f25f0795000)]
  0x00007f25d0c40800 JavaThread "BakerScheduler_QuartzSchedulerThread" [_thread_blocked, id=3380, stack(0x00007f25f0723000,0x00007f25f075c000)]
  0x00007f25d0efe800 JavaThread "java-sdk-http-connection-reaper" daemon [_thread_blocked, id=3385, stack(0x00007f25f06ea000,0x00007f25f0723000)]
  0x00007f25d041f800 JavaThread "pool-1-thread-1" [_thread_blocked, id=3389, stack(0x00007f25f004d000,0x00007f25f0086000)]
  0x00007f25d0593000 JavaThread "aggregated-log-reporter-1-thread-1" daemon [_thread_blocked, id=3390, stack(0x00007f25f0014000,0x00007f25f004d000)]
  0x00007f25ec019800 JavaThread "Timer-0" daemon [_thread_blocked, id=3391, stack(0x00007f25ca7c1000,0x00007f25ca7fa000)]
  0x00007f25ec011000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#0" daemon [_thread_blocked, id=3392, stack(0x00007f25ca788000,0x00007f25ca7c1000)]
  0x00007f25ec00f800 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#1" daemon [_thread_blocked, id=3393, stack(0x00007f25ca74f000,0x00007f25ca788000)]
  0x00007f25ec00e000 JavaThread "com.mchange.v2.async.ThreadPoolAsynchronousRunner$PoolThread-#2" daemon [_thread_blocked, id=3394, stack(0x00007f25ca716000,0x00007f25ca74f000)]
  0x00007f25d0629800 JavaThread "MaintenanceTimer-1-thread-1" daemon [_thread_blocked, id=3395, stack(0x00007f25ca4dd000,0x00007f25ca516000)]
  0x00007f25d03e5800 JavaThread "cluster-ClusterId{value='5c49256c0e24bf0d1c4d174b', description='null'}-gat-gsuite:27017" daemon [_thread_blocked, id=3396, stack(0x00007f25ca4a4000,0x00007f25ca4dd000)]
  0x00007f25d05f6800 JavaThread "CleanCursors-2-thread-1" daemon [_thread_blocked, id=3397, stack(0x00007f25ca46b000,0x00007f25ca4a4000)]
  0x00007f25d0bb1000 JavaThread "GAT-TASKS-REFRESH-3-thread-1" daemon [_thread_blocked, id=3400, stack(0x00007f25ca432000,0x00007f25ca46b000)]
  0x00007f25d1153000 JavaThread "MongoQuartzScheduler_Worker-1" [_thread_blocked, id=3401, stack(0x00007f25c9ff9000,0x00007f25ca032000)]
  0x00007f25d1154800 JavaThread "MongoQuartzScheduler_Worker-2" [_thread_blocked, id=3402, stack(0x00007f25c9fc0000,0x00007f25c9ff9000)]
  0x00007f25d115c800 JavaThread "MongoQuartzScheduler_Worker-3" [_thread_blocked, id=3403, stack(0x00007f25c9f87000,0x00007f25c9fc0000)]
  0x00007f25d115e800 JavaThread "MongoQuartzScheduler_Worker-4" [_thread_blocked, id=3404, stack(0x00007f25c9f4e000,0x00007f25c9f87000)]
  0x00007f25d1160800 JavaThread "MongoQuartzScheduler_Worker-5" [_thread_blocked, id=3405, stack(0x00007f25c9f15000,0x00007f25c9f4e000)]
  0x00007f25d1162800 JavaThread "MongoQuartzScheduler_Worker-6" [_thread_blocked, id=3406, stack(0x00007f25c9edc000,0x00007f25c9f15000)]
  0x00007f25d1165000 JavaThread "MongoQuartzScheduler_Worker-7" [_thread_blocked, id=3407, stack(0x00007f25c9ea3000,0x00007f25c9edc000)]
  0x00007f25d1167000 JavaThread "MongoQuartzScheduler_Worker-8" [_thread_blocked, id=3408, stack(0x00007f25c9e6a000,0x00007f25c9ea3000)]
  0x00007f25d1169000 JavaThread "MongoQuartzScheduler_Worker-9" [_thread_blocked, id=3409, stack(0x00007f25c9e31000,0x00007f25c9e6a000)]
  0x00007f25d116b000 JavaThread "MongoQuartzScheduler_Worker-10" [_thread_blocked, id=3410, stack(0x00007f25c9df8000,0x00007f25c9e31000)]
  0x00007f25d116d000 JavaThread "MongoQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=3411, stack(0x00007f25c9dbf000,0x00007f25c9df8000)]
  0x00007f25d1172000 JavaThread "cluster-ClusterId{value='5c49256d0e24bf0d1c4d174d', description='null'}-gat-gsuite:27017" daemon [_thread_blocked, id=3412, stack(0x00007f25c9d86000,0x00007f25c9dbf000)]
  0x00007f25d1173000 JavaThread "CleanCursors-4-thread-1" daemon [_thread_blocked, id=3413, stack(0x00007f25c9d4d000,0x00007f25c9d86000)]
  0x00007f262c3cf800 JavaThread "ContainerBackgroundProcessor[StandardEngine[Catalina]]" daemon [_thread_blocked, id=3427, stack(0x00007f25c9d14000,0x00007f25c9d4d000)]
  0x00007f262c37f000 JavaThread "http-bio-8080-exec-1" daemon [_thread_blocked, id=3428, stack(0x00007f25c92db000,0x00007f25c9314000)]
  0x00007f262c384000 JavaThread "http-bio-8080-exec-2" daemon [_thread_blocked, id=3429, stack(0x00007f25c92a2000,0x00007f25c92db000)]
  0x00007f262c385800 JavaThread "http-bio-8080-exec-3" daemon [_thread_blocked, id=3430, stack(0x00007f25c9269000,0x00007f25c92a2000)]
  0x00007f262c1e7000 JavaThread "http-bio-8080-exec-4" daemon [_thread_blocked, id=3431, stack(0x00007f25c9230000,0x00007f25c9269000)]
  0x00007f262c212000 JavaThread "http-bio-8080-exec-5" daemon [_thread_blocked, id=3432, stack(0x00007f25c91f7000,0x00007f25c9230000)]
  0x00007f262c213800 JavaThread "http-bio-8080-exec-6" daemon [_thread_blocked, id=3433, stack(0x00007f25c91be000,0x00007f25c91f7000)]
  0x00007f262c3c9000 JavaThread "http-bio-8080-exec-7" daemon [_thread_blocked, id=3434, stack(0x00007f25c9185000,0x00007f25c91be000)]
  0x00007f262c3ca000 JavaThread "http-bio-8080-exec-8" daemon [_thread_blocked, id=3435, stack(0x00007f25c914c000,0x00007f25c9185000)]
  0x00007f262c3cc000 JavaThread "http-bio-8080-exec-9" daemon [_thread_blocked, id=3436, stack(0x00007f25c9113000,0x00007f25c914c000)]
  0x00007f262c21e800 JavaThread "http-bio-8080-exec-10" daemon [_thread_blocked, id=3437, stack(0x00007f25c90da000,0x00007f25c9113000)]
  0x00007f262c220000 JavaThread "http-bio-8080-Acceptor-0" daemon [_thread_in_native, id=3438, stack(0x00007f25c90a1000,0x00007f25c90da000)]
  0x00007f262c222000 JavaThread "http-bio-8080-AsyncTimeout" daemon [_thread_blocked, id=3439, stack(0x00007f25c9068000,0x00007f25c90a1000)]
  0x00007f262c224800 JavaThread "ajp-bio-8009-exec-1" daemon [_thread_blocked, id=3440, stack(0x00007f25c902f000,0x00007f25c9068000)]
  0x00007f262c216800 JavaThread "ajp-bio-8009-exec-2" daemon [_thread_blocked, id=3441, stack(0x00007f25c8ff6000,0x00007f25c902f000)]
  0x00007f262c218800 JavaThread "ajp-bio-8009-exec-3" daemon [_thread_blocked, id=3442, stack(0x00007f25c8fbd000,0x00007f25c8ff6000)]
  0x00007f262c21a800 JavaThread "ajp-bio-8009-exec-4" daemon [_thread_blocked, id=3443, stack(0x00007f25c8f84000,0x00007f25c8fbd000)]
  0x00007f262c21c800 JavaThread "ajp-bio-8009-exec-5" daemon [_thread_blocked, id=3444, stack(0x00007f25c8f4b000,0x00007f25c8f84000)]
  0x00007f262c0e0000 JavaThread "ajp-bio-8009-exec-6" daemon [_thread_blocked, id=3445, stack(0x00007f25c8f12000,0x00007f25c8f4b000)]
  0x00007f262c0e2800 JavaThread "ajp-bio-8009-exec-7" daemon [_thread_blocked, id=3446, stack(0x00007f25c8ed9000,0x00007f25c8f12000)]
  0x00007f262c0e4800 JavaThread "ajp-bio-8009-exec-8" daemon [_thread_blocked, id=3447, stack(0x00007f25c8ea0000,0x00007f25c8ed9000)]
  0x00007f262c0e6800 JavaThread "ajp-bio-8009-exec-9" daemon [_thread_blocked, id=3448, stack(0x00007f25c8e67000,0x00007f25c8ea0000)]
  0x00007f262c11a800 JavaThread "ajp-bio-8009-exec-10" daemon [_thread_blocked, id=3449, stack(0x00007f25c8e2e000,0x00007f25c8e67000)]
  0x00007f262c11c800 JavaThread "ajp-bio-8009-Acceptor-0" daemon [_thread_in_native, id=3450, stack(0x00007f25c8df5000,0x00007f25c8e2e000)]
  0x00007f262c11e800 JavaThread "ajp-bio-8009-AsyncTimeout" daemon [_thread_blocked, id=3451, stack(0x00007f25c8dbc000,0x00007f25c8df5000)]
  0x00007f25ec2ed800 JavaThread "runScan webmaster@hh-cs.org" daemon [_thread_blocked, id=3721, stack(0x00007f261402f000,0x00007f2614068000)]
  0x00007f25c416c800 JavaThread "webmaster@hh-cs.org" daemon [_thread_blocked, id=3732, stack(0x00007f25f0086000,0x00007f25f00bf000)]
  0x00007f25d40a5800 JavaThread "MaintenanceTimer-5-thread-1" daemon [_thread_blocked, id=3760, stack(0x00007f25c874a000,0x00007f25c8783000)]
  0x00007f25d40a7000 JavaThread "cluster-ClusterId{value='5c4926c60e24bf0d1c4d174e', description='null'}-mongo-yoda-s1-m3:27017" daemon [_thread_blocked, id=3761, stack(0x00007f25c8783000,0x00007f25c87bc000)]
  0x00007f25d40a8000 JavaThread "CleanCursors-6-thread-1" daemon [_thread_blocked, id=3762, stack(0x00007f25f1324000,0x00007f25f135d000)]

Other Threads:
  0x00007f262c093000 VMThread "VM Thread" [stack: 0x00007f261440f000,0x00007f261450f000] [id=3365]
  0x00007f262c149800 WatcherThread [stack: 0x00007f26140d3000,0x00007f26141d3000] [id=3373]
  0x00007f262c037000 GCTaskThread "ParGC Thread#0" [stack: 0x00007f2631863000,0x00007f2631963000] [id=3364]
  0x00007f2600055000 GCTaskThread "ParGC Thread#1" [stack: 0x00007f25f0996000,0x00007f25f0a96000] [id=3377]

Threads with active compile tasks:
C2 CompilerThread014232       4       org.bson.BsonBinaryReader::readBsonType (349 bytes)
C1 CompilerThread014336       2       com.baker.gat.docs.Doc$DocResourceComparator::compareDetails (522 bytes)

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread: None

Heap address: 0x0000000689000000, size: 6000 MB, Compressed Oops mode: Zero based, Oop shift amount: 3
Narrow klass base: 0x0000000800000000, Narrow klass shift: 0
Compressed class space size: 1073741824 Address: 0x0000000800000000

Heap:
 PSYoungGen      total 1032192K, used 365214K [0x00000007b5000000, 0x00000007ff980000, 0x0000000800000000)
  eden space 835584K, 37% used [0x00000007b5000000,0x00000007c80277e8,0x00000007e8000000)
  from space 196608K, 27% used [0x00000007e8000000,0x00000007eb4801b0,0x00000007f4000000)
  to   space 188928K, 0% used [0x00000007f4100000,0x00000007f4100000,0x00000007ff980000)
 ParOldGen       total 3686400K, used 370619K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 10% used [0x0000000689000000,0x000000069f9eef68,0x000000076a000000)
 Metaspace       used 93145K, capacity 95450K, committed 95576K, reserved 1134592K
  class space    used 10249K, capacity 11069K, committed 11136K, reserved 1048576K

Card table byte_map: [0x00007f26322c3000,0x00007f2632e7c000] _byte_map_base: 0x00007f262ee7b000

Marking Bits: (ParMarkBitMap*) 0x00007f2634f75ac0
 Begin Bits: [0x00007f2604480000, 0x00007f260a240000)
 End Bits:   [0x00007f260a240000, 0x00007f2610000000)

Polling page: 0x00007f26351b7000

Metaspace:

Usage:
  Non-class:     82.40 MB capacity,    80.95 MB ( 98%) used,     1.20 MB (  1%) free+waste,   257.25 KB ( <1%) overhead. 
      Class:     10.81 MB capacity,    10.01 MB ( 93%) used,   705.92 KB (  6%) free+waste,   113.69 KB (  1%) overhead. 
       Both:     93.21 MB capacity,    90.96 MB ( 98%) used,     1.89 MB (  2%) free+waste,   370.94 KB ( <1%) overhead. 

Virtual space:
  Non-class space:       84.00 MB reserved,      82.46 MB ( 98%) committed 
      Class space:        1.00 GB reserved,      10.88 MB (  1%) committed 
             Both:        1.08 GB reserved,      93.34 MB (  8%) committed 

Chunk freelists:
   Non-Class:  4.38 KB
       Class:  384 bytes
        Both:  4.75 KB

CodeHeap 'non-profiled nmethods': size=120036Kb used=9274Kb max_used=9274Kb free=110761Kb
 bounds [0x00007f261c71f000, 0x00007f261d02f000, 0x00007f2623c58000]
CodeHeap 'profiled nmethods': size=120032Kb used=29529Kb max_used=29529Kb free=90502Kb
 bounds [0x00007f26151e7000, 0x00007f2616ed7000, 0x00007f261c71f000]
CodeHeap 'non-nmethods': size=5692Kb used=1327Kb max_used=1381Kb free=4364Kb
 bounds [0x00007f2614c58000, 0x00007f2614ec8000, 0x00007f26151e7000]
 total_blobs=18768 nmethods=12601 adapters=659
 compilation: enabled
              stopped_count=0, restarted_count=0
 full_count=0

Compilation events (10 events):
Event: 430.313 Thread 0x00007f262c0b6800 nmethod 14331 0x00007f2616ed1790 code [0x00007f2616ed1940, 0x00007f2616ed1ac8]
Event: 430.313 Thread 0x00007f262c0b6800 14332       2       java.lang.Long::compareTo (12 bytes)
Event: 430.313 Thread 0x00007f262c0b6800 nmethod 14332 0x00007f2616ed1b90 code [0x00007f2616ed1d40, 0x00007f2616ed1ef0]
Event: 430.313 Thread 0x00007f262c0b6800 14333       2       com.baker.gat.filter.BaseComparator::compare (56 bytes)
Event: 430.313 Thread 0x00007f262c0b6800 nmethod 14333 0x00007f2616ed1f90 code [0x00007f2616ed2160, 0x00007f2616ed2458]
Event: 430.313 Thread 0x00007f262c0b6800 14334       2       com.baker.gat.docs.Doc$DocResourceComparator::compareDetails (13 bytes)
Event: 430.314 Thread 0x00007f262c0b6800 nmethod 14334 0x00007f2616ed2590 code [0x00007f2616ed2760, 0x00007f2616ed2948]
Event: 430.314 Thread 0x00007f262c0b6800 14335       2       com.google.gdata.data.DateTime::compareTo (82 bytes)
Event: 430.314 Thread 0x00007f262c0b6800 nmethod 14335 0x00007f2616ed2a10 code [0x00007f2616ed2c00, 0x00007f2616ed3218]
Event: 430.314 Thread 0x00007f262c0b6800 14336       2       com.baker.gat.docs.Doc$DocResourceComparator::compareDetails (522 bytes)

GC Heap History (10 events):
Event: 364.148 GC heap before
{Heap before GC invocations=10 (full 3):
 PSYoungGen      total 691200K, used 691171K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 635904K, 100% used [0x00000007b5000000,0x00000007dbd00000,0x00000007dbd00000)
  from space 55296K, 99% used [0x00000007e5080000,0x00000007e8678d88,0x00000007e8680000)
  to   space 151040K, 0% used [0x00000007dbd00000,0x00000007dbd00000,0x00000007e5080000)
 ParOldGen       total 3686400K, used 138744K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 3% used [0x0000000689000000,0x000000069177e0a8,0x000000076a000000)
 Metaspace       used 86945K, capacity 88743K, committed 88920K, reserved 1128448K
  class space    used 9621K, capacity 10268K, committed 10368K, reserved 1048576K
}
Event: 364.385 GC heap after
{Heap after GC invocations=10 (full 3):
 PSYoungGen      total 786944K, used 106055K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 635904K, 0% used [0x00000007b5000000,0x00000007b5000000,0x00000007dbd00000)
  from space 151040K, 70% used [0x00000007dbd00000,0x00000007e2491d30,0x00000007e5080000)
  to   space 163840K, 0% used [0x00000007f6000000,0x00000007f6000000,0x0000000800000000)
 ParOldGen       total 3686400K, used 138744K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 3% used [0x0000000689000000,0x000000069177e0a8,0x000000076a000000)
 Metaspace       used 86945K, capacity 88743K, committed 88920K, reserved 1128448K
  class space    used 9621K, capacity 10268K, committed 10368K, reserved 1048576K
}
Event: 390.802 GC heap before
{Heap before GC invocations=11 (full 3):
 PSYoungGen      total 786944K, used 741959K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 635904K, 100% used [0x00000007b5000000,0x00000007dbd00000,0x00000007dbd00000)
  from space 151040K, 70% used [0x00000007dbd00000,0x00000007e2491d30,0x00000007e5080000)
  to   space 163840K, 0% used [0x00000007f6000000,0x00000007f6000000,0x0000000800000000)
 ParOldGen       total 3686400K, used 138744K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 3% used [0x0000000689000000,0x000000069177e0a8,0x000000076a000000)
 Metaspace       used 92288K, capacity 94509K, committed 94808K, reserved 1132544K
  class space    used 10211K, capacity 11021K, committed 11136K, reserved 1048576K
}
Event: 391.026 GC heap after
{Heap after GC invocations=11 (full 3):
 PSYoungGen      total 1057792K, used 96750K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 893952K, 0% used [0x00000007b5000000,0x00000007b5000000,0x00000007eb900000)
  from space 163840K, 59% used [0x00000007f6000000,0x00000007fbe7b9f0,0x0000000800000000)
  to   space 167424K, 0% used [0x00000007eb900000,0x00000007eb900000,0x00000007f5c80000)
 ParOldGen       total 3686400K, used 169134K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 4% used [0x0000000689000000,0x000000069352b998,0x000000076a000000)
 Metaspace       used 92288K, capacity 94509K, committed 94808K, reserved 1132544K
  class space    used 10211K, capacity 11021K, committed 11136K, reserved 1048576K
}
Event: 412.890 GC heap before
{Heap before GC invocations=12 (full 3):
 PSYoungGen      total 1057792K, used 990702K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 893952K, 100% used [0x00000007b5000000,0x00000007eb900000,0x00000007eb900000)
  from space 163840K, 59% used [0x00000007f6000000,0x00000007fbe7b9f0,0x0000000800000000)
  to   space 167424K, 0% used [0x00000007eb900000,0x00000007eb900000,0x00000007f5c80000)
 ParOldGen       total 3686400K, used 169134K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 4% used [0x0000000689000000,0x000000069352b998,0x000000076a000000)
 Metaspace       used 92676K, capacity 94921K, committed 95064K, reserved 1132544K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}
Event: 413.217 GC heap after
{Heap after GC invocations=12 (full 3):
 PSYoungGen      total 1036800K, used 142739K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 893952K, 0% used [0x00000007b5000000,0x00000007b5000000,0x00000007eb900000)
  from space 142848K, 99% used [0x00000007eb900000,0x00000007f4464da0,0x00000007f4480000)
  to   space 187904K, 0% used [0x00000007f4880000,0x00000007f4880000,0x0000000800000000)
 ParOldGen       total 3686400K, used 184801K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 5% used [0x0000000689000000,0x00000006944785f0,0x000000076a000000)
 Metaspace       used 92676K, capacity 94921K, committed 95064K, reserved 1132544K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}
Event: 419.984 GC heap before
{Heap before GC invocations=13 (full 3):
 PSYoungGen      total 1036800K, used 1036691K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 893952K, 100% used [0x00000007b5000000,0x00000007eb900000,0x00000007eb900000)
  from space 142848K, 99% used [0x00000007eb900000,0x00000007f4464da0,0x00000007f4480000)
  to   space 187904K, 0% used [0x00000007f4880000,0x00000007f4880000,0x0000000800000000)
 ParOldGen       total 3686400K, used 184801K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 5% used [0x0000000689000000,0x00000006944785f0,0x000000076a000000)
 Metaspace       used 92704K, capacity 94923K, committed 95064K, reserved 1132544K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}
Event: 420.288 GC heap after
{Heap after GC invocations=13 (full 3):
 PSYoungGen      total 1023488K, used 127824K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 835584K, 0% used [0x00000007b5000000,0x00000007b5000000,0x00000007e8000000)
  from space 187904K, 68% used [0x00000007f4880000,0x00000007fc5543c0,0x0000000800000000)
  to   space 196608K, 0% used [0x00000007e8000000,0x00000007e8000000,0x00000007f4000000)
 ParOldGen       total 3686400K, used 248728K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 6% used [0x0000000689000000,0x00000006982e60e8,0x000000076a000000)
 Metaspace       used 92704K, capacity 94923K, committed 95064K, reserved 1132544K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}
Event: 426.937 GC heap before
{Heap before GC invocations=14 (full 3):
 PSYoungGen      total 1023488K, used 963408K [0x00000007b5000000, 0x0000000800000000, 0x0000000800000000)
  eden space 835584K, 100% used [0x00000007b5000000,0x00000007e8000000,0x00000007e8000000)
  from space 187904K, 68% used [0x00000007f4880000,0x00000007fc5543c0,0x0000000800000000)
  to   space 196608K, 0% used [0x00000007e8000000,0x00000007e8000000,0x00000007f4000000)
 ParOldGen       total 3686400K, used 248728K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 6% used [0x0000000689000000,0x00000006982e60e8,0x000000076a000000)
 Metaspace       used 92722K, capacity 94987K, committed 95320K, reserved 1134592K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}
Event: 427.217 GC heap after
{Heap after GC invocations=14 (full 3):
 PSYoungGen      total 1032192K, used 53760K [0x00000007b5000000, 0x00000007ff980000, 0x0000000800000000)
  eden space 835584K, 0% used [0x00000007b5000000,0x00000007b5000000,0x00000007e8000000)
  from space 196608K, 27% used [0x00000007e8000000,0x00000007eb4801b0,0x00000007f4000000)
  to   space 188928K, 0% used [0x00000007f4100000,0x00000007f4100000,0x00000007ff980000)
 ParOldGen       total 3686400K, used 370619K [0x0000000689000000, 0x000000076a000000, 0x00000007b5000000)
  object space 3686400K, 10% used [0x0000000689000000,0x000000069f9eef68,0x000000076a000000)
 Metaspace       used 92722K, capacity 94987K, committed 95320K, reserved 1134592K
  class space    used 10213K, capacity 11026K, committed 11136K, reserved 1048576K
}

Deoptimization events (10 events):
Event: 419.830 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cfae9fc method=org.eclipse.persistence.internal.queries.ContainerPolicy.collectObjectForNewCollection(Ljava/util/Map;Ljava/util/Map;Ljava/lang/Object;Lorg/eclipse/persistence/internal/sess
Event: 419.830 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cfcbe00 method=org.eclipse.persistence.internal.sessions.CollectionChangeRecord.hasChanges()Z @ 4 c2
Event: 419.830 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cf40854 method=org.eclipse.persistence.mappings.CollectionMapping.compareForChange(Ljava/lang/Object;Ljava/lang/Object;Lorg/eclipse/persistence/internal/sessions/ObjectChangeSet;Lorg/eclip
Event: 419.901 Thread 0x00007f25c416c800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f261cfcf05c method=org.eclipse.persistence.internal.sessions.AbstractSession.prepareDatabaseQuery(Lorg/eclipse/persistence/queries/DatabaseQuery;)Lorg/eclipse/persistence/queries/DatabaseQ
Event: 421.469 Thread 0x00007f25c416c800 Uncommon trap: reason=class_check action=maybe_recompile pc=0x00007f261cfcf05c method=org.eclipse.persistence.internal.sessions.AbstractSession.prepareDatabaseQuery(Lorg/eclipse/persistence/queries/DatabaseQuery;)Lorg/eclipse/persistence/queries/DatabaseQ
Event: 425.320 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cb5704c method=org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(Lorg/eclipse/persistence/queries/DatabaseQuery;Lorg/eclipse/persistence/internal/sessions/AbstractReco
Event: 428.861 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cfbb960 method=org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(Lorg/eclipse/persistence/queries/DatabaseQuery;Lorg/eclipse/persistence/internal/sessions/AbstractReco
Event: 428.864 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cfba29c method=org.eclipse.persistence.internal.sessions.AbstractSession.executeQuery(Lorg/eclipse/persistence/queries/DatabaseQuery;Lorg/eclipse/persistence/internal/sessions/AbstractReco
Event: 429.620 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_if action=reinterpret pc=0x00007f261cdbc0b0 method=org.bson.io.OutputBuffer.writeCharacters(Ljava/lang/String;Z)I @ 68 c2
Event: 429.998 Thread 0x00007f25c416c800 Uncommon trap: reason=unstable_fused_if action=reinterpret pc=0x00007f261ced8fd4 method=java.net.URLDecoder.decode(Ljava/lang/String;Ljava/nio/charset/Charset;)Ljava/lang/String; @ 59 c2

Classes redefined (0 events):
No events

Internal exceptions (10 events):
Event: 371.499 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007beaea0b8}: java.lang.invoke.DirectMethodHandle$Holder.invokeSpecialIFC(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;)V> (0x00000007beaea0b8) thrown at [/scratch/opt/mach5/meso
Event: 374.234 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c12777c0}: java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(Ljava/lang/Object;Ljava/lang/Object;IIIIIILjava/lang/Object;)Ljava/lang/Object;> (0x00000007c12777c0) thrown at [/scratch/opt/mach5/me
Event: 374.235 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c127ac58}: java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(Ljava/lang/Object;IIIIIILjava/lang/Object;)Ljava/lang/Object;> (0x00000007c127ac58) thrown at [/scratch/opt/mach5/mesos/work_dir/slav
Event: 374.236 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c12a5078}: java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(Ljava/lang/Object;Ljava/lang/Object;IIILjava/lang/Object;)Ljava/lang/Object;> (0x00000007c12a5078) thrown at [/scratch/opt/mach5/mesos
Event: 374.296 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c143db08}: java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(Ljava/lang/Object;Ljava/lang/Object;IILjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava
Event: 374.301 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c14ac1c0}: java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(Ljava/lang/Object;Ljava/lang/Object;IILjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/
Event: 374.301 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c14b05f0}: java.lang.invoke.DelegatingMethodHandle$Holder.reinvoke_L(Ljava/lang/Object;IILjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava
Event: 374.308 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c153ac38}: java.lang.invoke.DirectMethodHandle$Holder.invokeStatic(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;IILjava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/l
Event: 374.318 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c15bdc30}: java.lang.invoke.DirectMethodHandle$Holder.invokeSpecial(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/la
Event: 374.318 Thread 0x00007f25c416c800 Exception <a 'java/lang/NoSuchMethodError'{0x00000007c15c1d00}: java.lang.invoke.Invokers$Holder.linkToTargetMethod(Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;Ljava/lang/Object;IILjava/lang/

Events (10 events):
Event: 430.167 loading class com/baker/gat/filter/BaseComparator$1
Event: 430.167 loading class com/baker/gat/filter/BaseComparator$1 done
Event: 430.169 loading class com/baker/gat/filter/GatFilter
Event: 430.169 loading class com/baker/gat/filter/GatFilter done
Event: 430.170 loading class com/baker/gat/acls/SharedResourceFilter$1
Event: 430.170 loading class com/baker/gat/acls/SharedResourceFilter$1 done
Event: 430.176 Thread 0x00007f25c416c800 DEOPT PACKING pc=0x00007f2615bf3a6d sp=0x00007f25f00bd380
Event: 430.176 Thread 0x00007f25c416c800 DEOPT UNPACKING pc=0x00007f2614c8013a sp=0x00007f25f00bc840 mode 0
Event: 430.310 loading class com/baker/gat/docs/Doc$1
Event: 430.310 loading class com/baker/gat/docs/Doc$1 done


Dynamic libraries:
00400000-00401000 r-xp 00000000 08:01 920383                             /var/gat/java/jdk-11.0.2/bin/java
00600000-00601000 r--p 00000000 08:01 920383                             /var/gat/java/jdk-11.0.2/bin/java
00601000-00602000 rw-p 00001000 08:01 920383                             /var/gat/java/jdk-11.0.2/bin/java
00665000-006cb000 rw-p 00000000 00:00 0                                  [heap]
689000000-76a000000 rw-p 00000000 00:00 0 
76a000000-7b5000000 ---p 00000000 00:00 0 
7b5000000-7ff980000 rw-p 00000000 00:00 0 
7ff980000-800000000 ---p 00000000 00:00 0 
800000000-800ae0000 rw-p 00000000 00:00 0 
800ae0000-840000000 ---p 00000000 00:00 0 
364e600000-364e620000 r-xp 00000000 08:01 524598                         /lib64/ld-2.12.so
364e820000-364e821000 r--p 00020000 08:01 524598                         /lib64/ld-2.12.so
364e821000-364e822000 rw-p 00021000 08:01 524598                         /lib64/ld-2.12.so
364e822000-364e823000 rw-p 00000000 00:00 0 
364ea00000-364eb8b000 r-xp 00000000 08:01 524611                         /lib64/libc-2.12.so
364eb8b000-364ed8a000 ---p 0018b000 08:01 524611                         /lib64/libc-2.12.so
364ed8a000-364ed8e000 r--p 0018a000 08:01 524611                         /lib64/libc-2.12.so
364ed8e000-364ed90000 rw-p 0018e000 08:01 524611                         /lib64/libc-2.12.so
364ed90000-364ed94000 rw-p 00000000 00:00 0 
364ee00000-364ee17000 r-xp 00000000 08:01 524616                         /lib64/libpthread-2.12.so
364ee17000-364f017000 ---p 00017000 08:01 524616                         /lib64/libpthread-2.12.so
364f017000-364f018000 r--p 00017000 08:01 524616                         /lib64/libpthread-2.12.so
364f018000-364f019000 rw-p 00018000 08:01 524616                         /lib64/libpthread-2.12.so
364f019000-364f01d000 rw-p 00000000 00:00 0 
364f200000-364f202000 r-xp 00000000 08:01 525673                         /lib64/libdl-2.12.so
364f202000-364f402000 ---p 00002000 08:01 525673                         /lib64/libdl-2.12.so
364f402000-364f403000 r--p 00002000 08:01 525673                         /lib64/libdl-2.12.so
364f403000-364f404000 rw-p 00003000 08:01 525673                         /lib64/libdl-2.12.so
364f600000-364f607000 r-xp 00000000 08:01 526056                         /lib64/librt-2.12.so
364f607000-364f806000 ---p 00007000 08:01 526056                         /lib64/librt-2.12.so
364f806000-364f807000 r--p 00006000 08:01 526056                         /lib64/librt-2.12.so
364f807000-364f808000 rw-p 00007000 08:01 526056                         /lib64/librt-2.12.so
364fa00000-364fa15000 r-xp 00000000 08:01 524976                         /lib64/libz.so.1.2.3
364fa15000-364fc14000 ---p 00015000 08:01 524976                         /lib64/libz.so.1.2.3
364fc14000-364fc15000 r--p 00014000 08:01 524976                         /lib64/libz.so.1.2.3
364fc15000-364fc16000 rw-p 00015000 08:01 524976                         /lib64/libz.so.1.2.3
364fe00000-364fe83000 r-xp 00000000 08:01 526060                         /lib64/libm-2.12.so
364fe83000-3650082000 ---p 00083000 08:01 526060                         /lib64/libm-2.12.so
3650082000-3650083000 r--p 00082000 08:01 526060                         /lib64/libm-2.12.so
3650083000-3650084000 rw-p 00083000 08:01 526060                         /lib64/libm-2.12.so
3653200000-3653216000 r-xp 00000000 08:01 526080                         /lib64/libresolv-2.12.so
3653216000-3653416000 ---p 00016000 08:01 526080                         /lib64/libresolv-2.12.so
3653416000-3653417000 r--p 00016000 08:01 526080                         /lib64/libresolv-2.12.so
3653417000-3653418000 rw-p 00017000 08:01 526080                         /lib64/libresolv-2.12.so
3653418000-365341a000 rw-p 00000000 00:00 0 
7f25c3636000-7f25c36b6000 rw-p 00000000 00:00 0 
7f25c36b6000-7f25c3836000 ---p 00000000 00:00 0 
7f25c3836000-7f25c3a36000 rw-p 00000000 00:00 0 
7f25c3a36000-7f25c3a3a000 ---p 00000000 00:00 0 
7f25c3a3a000-7f25c3a6f000 rw-p 00000000 00:00 0 
7f25c3a6f000-7f25c3a73000 ---p 00000000 00:00 0 
7f25c3a73000-7f25c3aa8000 rw-p 00000000 00:00 0 
7f25c3aa8000-7f25c3aac000 ---p 00000000 00:00 0 
7f25c3aac000-7f25c3ae1000 rw-p 00000000 00:00 0 
7f25c3ae1000-7f25c3ae5000 ---p 00000000 00:00 0 
7f25c3ae5000-7f25c3b1a000 rw-p 00000000 00:00 0 
7f25c3b1a000-7f25c3b1e000 ---p 00000000 00:00 0 
7f25c3b1e000-7f25c3b53000 rw-p 00000000 00:00 0 
7f25c3b53000-7f25c3b57000 ---p 00000000 00:00 0 
7f25c3b57000-7f25c3b8c000 rw-p 00000000 00:00 0 
7f25c3b8c000-7f25c3b90000 ---p 00000000 00:00 0 
7f25c3b90000-7f25c3bc5000 rw-p 00000000 00:00 0 
7f25c3bc5000-7f25c3bc9000 ---p 00000000 00:00 0 
7f25c3bc9000-7f25c3bfe000 rw-p 00000000 00:00 0 
7f25c3bfe000-7f25c3c02000 ---p 00000000 00:00 0 
7f25c3c02000-7f25c3c37000 rw-p 00000000 00:00 0 
7f25c3c37000-7f25c3c3b000 ---p 00000000 00:00 0 
7f25c3c3b000-7f25c3c70000 rw-p 00000000 00:00 0 
7f25c3c70000-7f25c3c74000 ---p 00000000 00:00 0 
7f25c3c74000-7f25c3ca9000 rw-p 00000000 00:00 0 
7f25c3ca9000-7f25c3cad000 ---p 00000000 00:00 0 
7f25c3cad000-7f25c3ce2000 rw-p 00000000 00:00 0 
7f25c3ce2000-7f25c3ce6000 ---p 00000000 00:00 0 
7f25c3ce6000-7f25c3d1b000 rw-p 00000000 00:00 0 
7f25c3d1b000-7f25c3d1f000 ---p 00000000 00:00 0 
7f25c3d1f000-7f25c3d54000 rw-p 00000000 00:00 0 
7f25c3d54000-7f25c3d58000 ---p 00000000 00:00 0 
7f25c3d58000-7f25c3d8d000 rw-p 00000000 00:00 0 
7f25c3d8d000-7f25c3d91000 ---p 00000000 00:00 0 
7f25c3d91000-7f25c3dc6000 rw-p 00000000 00:00 0 
7f25c3dc6000-7f25c3dca000 ---p 00000000 00:00 0 
7f25c3dca000-7f25c3dff000 rw-p 00000000 00:00 0 
7f25c3dff000-7f25c3e03000 ---p 00000000 00:00 0 
7f25c3e03000-7f25c3e38000 rw-p 00000000 00:00 0 
7f25c3e38000-7f25c3e3c000 ---p 00000000 00:00 0 
7f25c3e3c000-7f25c3e71000 rw-p 00000000 00:00 0 
7f25c3e71000-7f25c3e75000 ---p 00000000 00:00 0 
7f25c3e75000-7f25c3eaa000 rw-p 00000000 00:00 0 
7f25c3eaa000-7f25c3eae000 ---p 00000000 00:00 0 
7f25c3eae000-7f25c3ee3000 rw-p 00000000 00:00 0 
7f25c3ee3000-7f25c3ee7000 ---p 00000000 00:00 0 
7f25c3ee7000-7f25c3f1c000 rw-p 00000000 00:00 0 
7f25c3f1c000-7f25c3f20000 ---p 00000000 00:00 0 
7f25c3f20000-7f25c3f55000 rw-p 00000000 00:00 0 
7f25c3f55000-7f25c3f59000 ---p 00000000 00:00 0 
7f25c3f59000-7f25c3f8e000 rw-p 00000000 00:00 0 
7f25c3f8e000-7f25c3f92000 ---p 00000000 00:00 0 
7f25c3f92000-7f25c3fc7000 rw-p 00000000 00:00 0 
7f25c3fc7000-7f25c3fcb000 ---p 00000000 00:00 0 
7f25c3fcb000-7f25c4000000 rw-p 00000000 00:00 0 
7f25c4000000-7f25c41ad000 rw-p 00000000 00:00 0 
7f25c41ad000-7f25c8000000 ---p 00000000 00:00 0 
7f25c802d000-7f25c8031000 ---p 00000000 00:00 0 
7f25c8031000-7f25c8066000 rw-p 00000000 00:00 0 
7f25c8066000-7f25c806a000 ---p 00000000 00:00 0 
7f25c806a000-7f25c829f000 rw-p 00000000 00:00 0 
7f25c829f000-7f25c849f000 rw-p 00000000 00:00 0 
7f25c849f000-7f25c869f000 rw-p 00000000 00:00 0 
7f25c869f000-7f25c86a3000 ---p 00000000 00:00 0 
7f25c86a3000-7f25c86d8000 rw-p 00000000 00:00 0 
7f25c86d8000-7f25c86dc000 ---p 00000000 00:00 0 
7f25c86dc000-7f25c8711000 rw-p 00000000 00:00 0 
7f25c8711000-7f25c8715000 ---p 00000000 00:00 0 
7f25c8715000-7f25c874a000 rw-p 00000000 00:00 0 
7f25c874a000-7f25c874e000 ---p 00000000 00:00 0 
7f25c874e000-7f25c8783000 rw-p 00000000 00:00 0 
7f25c8783000-7f25c8787000 ---p 00000000 00:00 0 
7f25c8787000-7f25c89bc000 rw-p 00000000 00:00 0 
7f25c89bc000-7f25c8bbc000 rw-p 00000000 00:00 0 
7f25c8bbc000-7f25c8dbc000 rw-p 00000000 00:00 0 
7f25c8dbc000-7f25c8dc0000 ---p 00000000 00:00 0 
7f25c8dc0000-7f25c8df5000 rw-p 00000000 00:00 0 
7f25c8df5000-7f25c8df9000 ---p 00000000 00:00 0 
7f25c8df9000-7f25c8e2e000 rw-p 00000000 00:00 0 
7f25c8e2e000-7f25c8e32000 ---p 00000000 00:00 0 
7f25c8e32000-7f25c8e67000 rw-p 00000000 00:00 0 
7f25c8e67000-7f25c8e6b000 ---p 00000000 00:00 0 
7f25c8e6b000-7f25c8ea0000 rw-p 00000000 00:00 0 
7f25c8ea0000-7f25c8ea4000 ---p 00000000 00:00 0 
7f25c8ea4000-7f25c8ed9000 rw-p 00000000 00:00 0 
7f25c8ed9000-7f25c8edd000 ---p 00000000 00:00 0 
7f25c8edd000-7f25c8f12000 rw-p 00000000 00:00 0 
7f25c8f12000-7f25c8f16000 ---p 00000000 00:00 0 
7f25c8f16000-7f25c8f4b000 rw-p 00000000 00:00 0 
7f25c8f4b000-7f25c8f4f000 ---p 00000000 00:00 0 
7f25c8f4f000-7f25c8f84000 rw-p 00000000 00:00 0 
7f25c8f84000-7f25c8f88000 ---p 00000000 00:00 0 
7f25c8f88000-7f25c8fbd000 rw-p 00000000 00:00 0 
7f25c8fbd000-7f25c8fc1000 ---p 00000000 00:00 0 
7f25c8fc1000-7f25c8ff6000 rw-p 00000000 00:00 0 
7f25c8ff6000-7f25c8ffa000 ---p 00000000 00:00 0 
7f25c8ffa000-7f25c902f000 rw-p 00000000 00:00 0 
7f25c902f000-7f25c9033000 ---p 00000000 00:00 0 
7f25c9033000-7f25c9068000 rw-p 00000000 00:00 0 
7f25c9068000-7f25c906c000 ---p 00000000 00:00 0 
7f25c906c000-7f25c90a1000 rw-p 00000000 00:00 0 
7f25c90a1000-7f25c90a5000 ---p 00000000 00:00 0 
7f25c90a5000-7f25c90da000 rw-p 00000000 00:00 0 
7f25c90da000-7f25c90de000 ---p 00000000 00:00 0 
7f25c90de000-7f25c9113000 rw-p 00000000 00:00 0 
7f25c9113000-7f25c9117000 ---p 00000000 00:00 0 
7f25c9117000-7f25c914c000 rw-p 00000000 00:00 0 
7f25c914c000-7f25c9150000 ---p 00000000 00:00 0 
7f25c9150000-7f25c9185000 rw-p 00000000 00:00 0 
7f25c9185000-7f25c9189000 ---p 00000000 00:00 0 
7f25c9189000-7f25c91be000 rw-p 00000000 00:00 0 
7f25c91be000-7f25c91c2000 ---p 00000000 00:00 0 
7f25c91c2000-7f25c91f7000 rw-p 00000000 00:00 0 
7f25c91f7000-7f25c91fb000 ---p 00000000 00:00 0 
7f25c91fb000-7f25c9230000 rw-p 00000000 00:00 0 
7f25c9230000-7f25c9234000 ---p 00000000 00:00 0 
7f25c9234000-7f25c9269000 rw-p 00000000 00:00 0 
7f25c9269000-7f25c926d000 ---p 00000000 00:00 0 
7f25c926d000-7f25c92a2000 rw-p 00000000 00:00 0 
7f25c92a2000-7f25c92a6000 ---p 00000000 00:00 0 
7f25c92a6000-7f25c92db000 rw-p 00000000 00:00 0 
7f25c92db000-7f25c92df000 ---p 00000000 00:00 0 
7f25c92df000-7f25c9514000 rw-p 00000000 00:00 0 
7f25c9514000-7f25c9714000 rw-p 00000000 00:00 0 
7f25c9714000-7f25c9914000 rw-p 00000000 00:00 0 
7f25c9914000-7f25c9b14000 rw-p 00000000 00:00 0 
7f25c9b14000-7f25c9d14000 rw-p 00000000 00:00 0 
7f25c9d14000-7f25c9d18000 ---p 00000000 00:00 0 
7f25c9d18000-7f25c9d4d000 rw-p 00000000 00:00 0 
7f25c9d4d000-7f25c9d51000 ---p 00000000 00:00 0 
7f25c9d51000-7f25c9d86000 rw-p 00000000 00:00 0 
7f25c9d86000-7f25c9d8a000 ---p 00000000 00:00 0 
7f25c9d8a000-7f25c9dbf000 rw-p 00000000 00:00 0 
7f25c9dbf000-7f25c9dc3000 ---p 00000000 00:00 0 
7f25c9dc3000-7f25c9df8000 rw-p 00000000 00:00 0 
7f25c9df8000-7f25c9dfc000 ---p 00000000 00:00 0 
7f25c9dfc000-7f25c9e31000 rw-p 00000000 00:00 0 
7f25c9e31000-7f25c9e35000 ---p 00000000 00:00 0 
7f25c9e35000-7f25c9e6a000 rw-p 00000000 00:00 0 
7f25c9e6a000-7f25c9e6e000 ---p 00000000 00:00 0 
7f25c9e6e000-7f25c9ea3000 rw-p 00000000 00:00 0 
7f25c9ea3000-7f25c9ea7000 ---p 00000000 00:00 0 
7f25c9ea7000-7f25c9edc000 rw-p 00000000 00:00 0 
7f25c9edc000-7f25c9ee0000 ---p 00000000 00:00 0 
7f25c9ee0000-7f25c9f15000 rw-p 00000000 00:00 0 
7f25c9f15000-7f25c9f19000 ---p 00000000 00:00 0 
7f25c9f19000-7f25c9f4e000 rw-p 00000000 00:00 0 
7f25c9f4e000-7f25c9f52000 ---p 00000000 00:00 0 
7f25c9f52000-7f25c9f87000 rw-p 00000000 00:00 0 
7f25c9f87000-7f25c9f8b000 ---p 00000000 00:00 0 
7f25c9f8b000-7f25c9fc0000 rw-p 00000000 00:00 0 
7f25c9fc0000-7f25c9fc4000 ---p 00000000 00:00 0 
7f25c9fc4000-7f25c9ff9000 rw-p 00000000 00:00 0 
7f25c9ff9000-7f25c9ffd000 ---p 00000000 00:00 0 
7f25c9ffd000-7f25ca232000 rw-p 00000000 00:00 0 
7f25ca232000-7f25ca432000 rw-p 00000000 00:00 0 
7f25ca432000-7f25ca436000 ---p 00000000 00:00 0 
7f25ca436000-7f25ca46b000 rw-p 00000000 00:00 0 
7f25ca46b000-7f25ca46f000 ---p 00000000 00:00 0 
7f25ca46f000-7f25ca4a4000 rw-p 00000000 00:00 0 
7f25ca4a4000-7f25ca4a8000 ---p 00000000 00:00 0 
7f25ca4a8000-7f25ca4dd000 rw-p 00000000 00:00 0 
7f25ca4dd000-7f25ca4e1000 ---p 00000000 00:00 0 
7f25ca4e1000-7f25ca716000 rw-p 00000000 00:00 0 
7f25ca716000-7f25ca71a000 ---p 00000000 00:00 0 
7f25ca71a000-7f25ca74f000 rw-p 00000000 00:00 0 
7f25ca74f000-7f25ca753000 ---p 00000000 00:00 0 
7f25ca753000-7f25ca788000 rw-p 00000000 00:00 0 
7f25ca788000-7f25ca78c000 ---p 00000000 00:00 0 
7f25ca78c000-7f25ca7c1000 rw-p 00000000 00:00 0 
7f25ca7c1000-7f25ca7c5000 ---p 00000000 00:00 0 
7f25ca7c5000-7f25ca9fa000 rw-p 00000000 00:00 0 
7f25ca9fa000-7f25cabfa000 rw-p 00000000 00:00 0 
7f25cabfa000-7f25cadfa000 rw-p 00000000 00:00 0 
7f25cadfa000-7f25caffa000 rw-p 00000000 00:00 0 
7f25caffa000-7f25cb1fa000 rw-p 00000000 00:00 0 
7f25cb1fa000-7f25cb3fa000 rw-p 00000000 00:00 0 
7f25cb3fa000-7f25cb5fa000 rw-p 00000000 00:00 0 
7f25cb5fa000-7f25cb5ff000 r-xp 00000000 08:01 524342                     /lib64/libnss_dns-2.12.so
7f25cb5ff000-7f25cb7fe000 ---p 00005000 08:01 524342                     /lib64/libnss_dns-2.12.so
7f25cb7fe000-7f25cb7ff000 r--p 00004000 08:01 524342                     /lib64/libnss_dns-2.12.so
7f25cb7ff000-7f25cb800000 rw-p 00005000 08:01 524342                     /lib64/libnss_dns-2.12.so
7f25cb800000-7f25cba00000 rw-p 00000000 00:00 0 
7f25cba00000-7f25cbc00000 rw-p 00000000 00:00 0 
7f25cbc00000-7f25cbe00000 rw-p 00000000 00:00 0 
7f25cbe00000-7f25cbff6000 rw-p 00000000 00:00 0 
7f25cbff6000-7f25cc000000 ---p 00000000 00:00 0 
7f25cc000000-7f25cc021000 rw-p 00000000 00:00 0 
7f25cc021000-7f25d0000000 ---p 00000000 00:00 0 
7f25d0000000-7f25d134a000 rw-p 00000000 00:00 0 
7f25d134a000-7f25d4000000 ---p 00000000 00:00 0 
7f25d4000000-7f25d4621000 rw-p 00000000 00:00 0 
7f25d4621000-7f25d8000000 ---p 00000000 00:00 0 
7f25d8000000-7f25d8021000 rw-p 00000000 00:00 0 
7f25d8021000-7f25dc000000 ---p 00000000 00:00 0 
7f25dc000000-7f25dc0f1000 rw-p 00000000 00:00 0 
7f25dc0f1000-7f25e0000000 ---p 00000000 00:00 0 
7f25e0000000-7f25e004b000 rw-p 00000000 00:00 0 
7f25e004b000-7f25e4000000 ---p 00000000 00:00 0 
7f25e4000000-7f25e663a000 rw-p 00000000 00:00 0 
7f25e663a000-7f25e8000000 ---p 00000000 00:00 0 
7f25e8000000-7f25e908f000 rw-p 00000000 00:00 0 
7f25e908f000-7f25ec000000 ---p 00000000 00:00 0 
7f25ec000000-7f25ec332000 rw-p 00000000 00:00 0 
7f25ec332000-7f25f0000000 ---p 00000000 00:00 0 
7f25f0014000-7f25f0018000 ---p 00000000 00:00 0 
7f25f0018000-7f25f004d000 rw-p 00000000 00:00 0 
7f25f004d000-7f25f0051000 ---p 00000000 00:00 0 
7f25f0051000-7f25f0086000 rw-p 00000000 00:00 0 
7f25f0086000-7f25f008a000 ---p 00000000 00:00 0 
7f25f008a000-7f25f02bf000 rw-p 00000000 00:00 0 
7f25f02bf000-7f25f04bf000 rw-p 00000000 00:00 0 
7f25f04bf000-7f25f06bf000 rw-p 00000000 00:00 0 
7f25f06ea000-7f25f06ee000 ---p 00000000 00:00 0 
7f25f06ee000-7f25f0723000 rw-p 00000000 00:00 0 
7f25f0723000-7f25f0727000 ---p 00000000 00:00 0 
7f25f0727000-7f25f075c000 rw-p 00000000 00:00 0 
7f25f075c000-7f25f0760000 ---p 00000000 00:00 0 
7f25f0760000-7f25f0995000 rw-p 00000000 00:00 0 
7f25f0995000-7f25f0996000 ---p 00000000 00:00 0 
7f25f0996000-7f25f0a96000 rw-p 00000000 00:00 0 
7f25f0aa5000-7f25f0aa9000 ---p 00000000 00:00 0 
7f25f0aa9000-7f25f0cde000 rw-p 00000000 00:00 0 
7f25f0cde000-7f25f0d1e000 r-xp 00000000 08:01 1050094                    /var/gat/java/jdk-11.0.2/lib/libsunec.so
7f25f0d1e000-7f25f0f1d000 ---p 00040000 08:01 1050094                    /var/gat/java/jdk-11.0.2/lib/libsunec.so
7f25f0f1d000-7f25f0f22000 r--p 0003f000 08:01 1050094                    /var/gat/java/jdk-11.0.2/lib/libsunec.so
7f25f0f22000-7f25f0f24000 rw-p 00044000 08:01 1050094                    /var/gat/java/jdk-11.0.2/lib/libsunec.so
7f25f0f24000-7f25f1124000 rw-p 00000000 00:00 0 
7f25f1124000-7f25f1324000 rw-p 00000000 00:00 0 
7f25f1324000-7f25f1328000 ---p 00000000 00:00 0 
7f25f1328000-7f25f135d000 rw-p 00000000 00:00 0 
7f25f135d000-7f25f135f000 r-xp 00000000 08:01 1050068                    /var/gat/java/jdk-11.0.2/lib/libextnet.so
7f25f135f000-7f25f155e000 ---p 00002000 08:01 1050068                    /var/gat/java/jdk-11.0.2/lib/libextnet.so
7f25f155e000-7f25f155f000 r--p 00001000 08:01 1050068                    /var/gat/java/jdk-11.0.2/lib/libextnet.so
7f25f155f000-7f25f1560000 rw-p 00002000 08:01 1050068                    /var/gat/java/jdk-11.0.2/lib/libextnet.so
7f25f1560000-7f25f1760000 rw-p 00000000 00:00 0 
7f25f1760000-7f25f1960000 rw-p 00000000 00:00 0 
7f25f1960000-7f25f1965000 r-xp 00000000 08:01 1050085                    /var/gat/java/jdk-11.0.2/lib/libmanagement_ext.so
7f25f1965000-7f25f1b64000 ---p 00005000 08:01 1050085                    /var/gat/java/jdk-11.0.2/lib/libmanagement_ext.so
7f25f1b64000-7f25f1b65000 r--p 00004000 08:01 1050085                    /var/gat/java/jdk-11.0.2/lib/libmanagement_ext.so
7f25f1b65000-7f25f1b66000 rw-p 00005000 08:01 1050085                    /var/gat/java/jdk-11.0.2/lib/libmanagement_ext.so
7f25f1b66000-7f25f1b6a000 r-xp 00000000 08:01 1050083                    /var/gat/java/jdk-11.0.2/lib/libmanagement.so
7f25f1b6a000-7f25f1d69000 ---p 00004000 08:01 1050083                    /var/gat/java/jdk-11.0.2/lib/libmanagement.so
7f25f1d69000-7f25f1d6a000 r--p 00003000 08:01 1050083                    /var/gat/java/jdk-11.0.2/lib/libmanagement.so
7f25f1d6a000-7f25f1d6b000 rw-p 00004000 08:01 1050083                    /var/gat/java/jdk-11.0.2/lib/libmanagement.so
7f25f1d6b000-7f25f1f6b000 rw-p 00000000 00:00 0 
7f25f1f6b000-7f25f216b000 rw-p 00000000 00:00 0 
7f25f216b000-7f25f8000000 r--p 00000000 08:01 18534                      /usr/lib/locale/locale-archive
7f25f8000000-7f25f8021000 rw-p 00000000 00:00 0 
7f25f8021000-7f25fc000000 ---p 00000000 00:00 0 
7f25fc000000-7f25fc062000 rw-p 00000000 00:00 0 
7f25fc062000-7f2600000000 ---p 00000000 00:00 0 
7f2600000000-7f2600294000 rw-p 00000000 00:00 0 
7f2600294000-7f2604000000 ---p 00000000 00:00 0 
7f260401e000-7f2604034000 r-xp 00000000 08:01 1050087                    /var/gat/java/jdk-11.0.2/lib/libnet.so
7f2604034000-7f2604234000 ---p 00016000 08:01 1050087                    /var/gat/java/jdk-11.0.2/lib/libnet.so
7f2604234000-7f2604235000 r--p 00016000 08:01 1050087                    /var/gat/java/jdk-11.0.2/lib/libnet.so
7f2604235000-7f2604236000 rw-p 00017000 08:01 1050087                    /var/gat/java/jdk-11.0.2/lib/libnet.so
7f2604236000-7f2604246000 r-xp 00000000 08:01 1050088                    /var/gat/java/jdk-11.0.2/lib/libnio.so
7f2604246000-7f2604445000 ---p 00010000 08:01 1050088                    /var/gat/java/jdk-11.0.2/lib/libnio.so
7f2604445000-7f2604446000 r--p 0000f000 08:01 1050088                    /var/gat/java/jdk-11.0.2/lib/libnio.so
7f2604446000-7f2604447000 rw-p 00010000 08:01 1050088                    /var/gat/java/jdk-11.0.2/lib/libnio.so
7f2604447000-7f260444b000 ---p 00000000 00:00 0 
7f260444b000-7f2610000000 rw-p 00000000 00:00 0 
7f2610000000-7f2610030000 rw-p 00000000 00:00 0 
7f2610030000-7f2614000000 ---p 00000000 00:00 0 
7f261402f000-7f2614033000 ---p 00000000 00:00 0 
7f2614033000-7f2614099000 rw-p 00000000 00:00 0 
7f2614099000-7f261409d000 ---p 00000000 00:00 0 
7f261409d000-7f26140d2000 rw-p 00000000 00:00 0 
7f26140d2000-7f26140d3000 ---p 00000000 00:00 0 
7f26140d3000-7f26141d3000 rw-p 00000000 00:00 0 
7f26141d3000-7f26141d7000 ---p 00000000 00:00 0 
7f26141d7000-7f261420c000 rw-p 00000000 00:00 0 
7f261420c000-7f2614210000 ---p 00000000 00:00 0 
7f2614210000-7f261430d000 rw-p 00000000 00:00 0 
7f261430d000-7f2614311000 ---p 00000000 00:00 0 
7f2614311000-7f261440e000 rw-p 00000000 00:00 0 
7f261440e000-7f261440f000 ---p 00000000 00:00 0 
7f261440f000-7f2614c58000 rw-p 00000000 00:00 0 
7f2614c58000-7f2614ec8000 rwxp 00000000 00:00 0 
7f2614ec8000-7f26151e7000 ---p 00000000 00:00 0 
7f26151e7000-7f2616ed7000 rwxp 00000000 00:00 0 
7f2616ed7000-7f261c71f000 ---p 00000000 00:00 0 
7f261c71f000-7f261d03f000 rwxp 00000000 00:00 0 
7f261d03f000-7f2623c58000 ---p 00000000 00:00 0 
7f2623c58000-7f262c000000 r--s 00000000 08:01 1050098                    /var/gat/java/jdk-11.0.2/lib/modules
7f262c000000-7f262c4e2000 rw-p 00000000 00:00 0 
7f262c4e2000-7f2630000000 ---p 00000000 00:00 0 
7f2630020000-7f2630024000 ---p 00000000 00:00 0 
7f2630024000-7f2630059000 rw-p 00000000 00:00 0 
7f2630059000-7f263005d000 ---p 00000000 00:00 0 
7f263005d000-7f2630092000 rw-p 00000000 00:00 0 
7f2630092000-7f2630096000 ---p 00000000 00:00 0 
7f2630096000-7f2630bf2000 rw-p 00000000 00:00 0 
7f2630bf2000-7f2631862000 rw-p 00000000 00:00 0 
7f2631862000-7f2631863000 ---p 00000000 00:00 0 
7f2631863000-7f263206b000 rw-p 00000000 00:00 0 
7f263206b000-7f26322c3000 ---p 00000000 00:00 0 
7f26322c3000-7f26329cb000 rw-p 00000000 00:00 0 
7f26329cb000-7f2632c23000 ---p 00000000 00:00 0 
7f2632c23000-7f2632e78000 rw-p 00000000 00:00 0 
7f2632e78000-7f2632e7b000 ---p 00000000 00:00 0 
7f2632e7b000-7f2632e8f000 rw-p 00000000 00:00 0 
7f2632e8f000-7f2632f67000 ---p 00000000 00:00 0 
7f2632f67000-7f2632fa1000 rw-p 00000000 00:00 0 
7f2632fa1000-7f2633052000 ---p 00000000 00:00 0 
7f2633052000-7f2633057000 rw-p 00000000 00:00 0 
7f2633057000-7f263305e000 ---p 00000000 00:00 0 
7f263305e000-7f2633079000 r-xp 00000000 08:01 1050079                    /var/gat/java/jdk-11.0.2/lib/libjimage.so
7f2633079000-7f2633278000 ---p 0001b000 08:01 1050079                    /var/gat/java/jdk-11.0.2/lib/libjimage.so
7f2633278000-7f263327a000 r--p 0001a000 08:01 1050079                    /var/gat/java/jdk-11.0.2/lib/libjimage.so
7f263327a000-7f263327b000 rw-p 0001c000 08:01 1050079                    /var/gat/java/jdk-11.0.2/lib/libjimage.so
7f263327b000-7f2633282000 r-xp 00000000 08:01 1050097                    /var/gat/java/jdk-11.0.2/lib/libzip.so
7f2633282000-7f2633481000 ---p 00007000 08:01 1050097                    /var/gat/java/jdk-11.0.2/lib/libzip.so
7f2633481000-7f2633482000 r--p 00006000 08:01 1050097                    /var/gat/java/jdk-11.0.2/lib/libzip.so
7f2633482000-7f2633483000 rw-p 00007000 08:01 1050097                    /var/gat/java/jdk-11.0.2/lib/libzip.so
7f2633483000-7f263348b000 rw-s 00000000 08:01 655407                     /tmp/hsperfdata_tomcat/3356
7f263348b000-7f2633498000 r-xp 00000000 08:01 525852                     /lib64/libnss_files-2.12.so
7f2633498000-7f2633697000 ---p 0000d000 08:01 525852                     /lib64/libnss_files-2.12.so
7f2633697000-7f2633698000 r--p 0000c000 08:01 525852                     /lib64/libnss_files-2.12.so
7f2633698000-7f2633699000 rw-p 0000d000 08:01 525852                     /lib64/libnss_files-2.12.so
7f2633699000-7f26336a0000 r--s 00000000 08:01 5378                       /usr/lib64/gconv/gconv-modules.cache
7f26336a0000-7f26336c9000 r-xp 00000000 08:01 1050075                    /var/gat/java/jdk-11.0.2/lib/libjava.so
7f26336c9000-7f26338c8000 ---p 00029000 08:01 1050075                    /var/gat/java/jdk-11.0.2/lib/libjava.so
7f26338c8000-7f26338c9000 r--p 00028000 08:01 1050075                    /var/gat/java/jdk-11.0.2/lib/libjava.so
7f26338c9000-7f26338cb000 rw-p 00029000 08:01 1050075                    /var/gat/java/jdk-11.0.2/lib/libjava.so
7f26338cb000-7f26338d8000 r-xp 00000000 08:01 1050096                    /var/gat/java/jdk-11.0.2/lib/libverify.so
7f26338d8000-7f2633ad7000 ---p 0000d000 08:01 1050096                    /var/gat/java/jdk-11.0.2/lib/libverify.so
7f2633ad7000-7f2633ad9000 r--p 0000c000 08:01 1050096                    /var/gat/java/jdk-11.0.2/lib/libverify.so
7f2633ad9000-7f2633ada000 rw-p 0000e000 08:01 1050096                    /var/gat/java/jdk-11.0.2/lib/libverify.so
7f2633ada000-7f2633ade000 ---p 00000000 00:00 0 
7f2633ade000-7f2633b13000 rw-p 00000000 00:00 0 
7f2633b13000-7f2634c50000 r-xp 00000000 08:01 1050109                    /var/gat/java/jdk-11.0.2/lib/server/libjvm.so
7f2634c50000-7f2634e4f000 ---p 0113d000 08:01 1050109                    /var/gat/java/jdk-11.0.2/lib/server/libjvm.so
7f2634e4f000-7f2634f0e000 r--p 0113c000 08:01 1050109                    /var/gat/java/jdk-11.0.2/lib/server/libjvm.so
7f2634f0e000-7f2634f47000 rw-p 011fb000 08:01 1050109                    /var/gat/java/jdk-11.0.2/lib/server/libjvm.so
7f2634f47000-7f2634fa1000 rw-p 00000000 00:00 0 
7f2634fa1000-7f2634fb1000 r-xp 00000000 08:01 1050060                    /var/gat/java/jdk-11.0.2/lib/jli/libjli.so
7f2634fb1000-7f26351b0000 ---p 00010000 08:01 1050060                    /var/gat/java/jdk-11.0.2/lib/jli/libjli.so
7f26351b0000-7f26351b1000 r--p 0000f000 08:01 1050060                    /var/gat/java/jdk-11.0.2/lib/jli/libjli.so
7f26351b1000-7f26351b2000 rw-p 00010000 08:01 1050060                    /var/gat/java/jdk-11.0.2/lib/jli/libjli.so
7f26351b2000-7f26351b3000 rw-p 00000000 00:00 0 
7f26351b6000-7f26351b7000 rw-p 00000000 00:00 0 
7f26351b7000-7f26351b8000 ---p 00000000 00:00 0 
7f26351b8000-7f26351b9000 r--p 00000000 00:00 0 
7f26351b9000-7f26351bb000 rw-p 00000000 00:00 0 
7fff0d4a6000-7fff0d4bb000 rw-p 00000000 00:00 0                          [stack]
7fff0d59c000-7fff0d59d000 r-xp 00000000 00:00 0                          [vdso]
ffffffffff600000-ffffffffff601000 r-xp 00000000 00:00 0                  [vsyscall]


VM Arguments:
jvm_args: -Xms4500m -Xmx6000m -Xss228k -XX:NewRatio=4 -XX:+PrintGC -XX:+UseParallelGC -XX:-UseGCOverheadLimit -Dhttps.protocols=TLSv1,TLSv1.1,TLSv1.2 -Dorg.apache.jasper.runtime.BodyContentImpl.LIMIT_BUFFER=true -DGAT_RUN_JOBS= -Dsun.net.client.defaultReadTimeout=60000 -Dsun.net.client.defaultConnectTimeout=30000 -Dhttp.keepAlive=false -DCLOUD_TYPE=gce -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager 
java_command: org.apache.catalina.startup.Bootstrap start
java_class_path (initial): :/usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/bin/build-classpath: error: JAVAVER_LIBDIR /usr/share/java-11.0.2 does not exist or is not a directory
Launcher Type: SUN_STANDARD

[Global flags]
     intx CICompilerCount                          = 2                                         {product} {ergonomic}
   size_t InitialHeapSize                          = 4718592000                                {product} {command line}
   size_t MaxHeapSize                              = 6291456000                                {product} {command line}
   size_t MaxNewSize                               = 1258291200                                {product} {ergonomic}
   size_t MinHeapDeltaBytes                        = 524288                                    {product} {ergonomic}
    uintx NewRatio                                 = 4                                         {product} {command line}
   size_t NewSize                                  = 943718400                                 {product} {ergonomic}
    uintx NonNMethodCodeHeapSize                   = 5824844                                {pd product} {ergonomic}
    uintx NonProfiledCodeHeapSize                  = 122916698                              {pd product} {ergonomic}
   size_t OldSize                                  = 3774873600                                {product} {ergonomic}
     bool PrintGC                                  = true                                      {product} {command line}
    uintx ProfiledCodeHeapSize                     = 122916698                              {pd product} {ergonomic}
    uintx ReservedCodeCacheSize                    = 251658240                              {pd product} {ergonomic}
     bool SegmentedCodeCache                       = true                                      {product} {ergonomic}
     intx ThreadStackSize                          = 228                                    {pd product} {command line}
     bool UseCompressedClassPointers               = true                                 {lp64_product} {ergonomic}
     bool UseCompressedOops                        = true                                 {lp64_product} {ergonomic}
     bool UseGCOverheadLimit                       = false                                     {product} {command line}
     bool UseParallelGC                            = true                                      {product} {command line}

Logging:
Log output configuration:
 #0: stdout all=warning,gc=info uptime,level,tags
 #1: stderr all=off uptime,level,tags

Environment Variables:
JAVA_HOME=/var/gat/java/jdk-11.0.2
PATH=/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/var/gat/java/jdk-11.0.2/bin:/usr/local/sbin:/usr/local/bin:/usr/bin:/usr/sbin:/sbin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin:/bin
SHELL=/bin/sh

Signal Handlers:
SIGSEGV: [libjvm.so+0xe28d40], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGBUS: [libjvm.so+0xe28d40], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGFPE: [libjvm.so+0xe28d40], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGPIPE: [libjvm.so+0xbfd500], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGXFSZ: [libjvm.so+0xbfd500], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGILL: [libjvm.so+0xe28d40], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGUSR2: [libjvm.so+0xbfd3a0], sa_mask[0]=00000000000000000000000000000000, sa_flags=SA_RESTART|SA_SIGINFO
SIGHUP: [libjvm.so+0xbfd700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGINT: SIG_IGN, sa_mask[0]=00000000000000000000000000000000, sa_flags=none
SIGTERM: [libjvm.so+0xbfd700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO
SIGQUIT: [libjvm.so+0xbfd700], sa_mask[0]=11111111011111111101111111111110, sa_flags=SA_RESTART|SA_SIGINFO


---------------  S Y S T E M  ---------------

OS:CentOS release 6.10 (Final)
uname:Linux 2.6.32-754.2.1.el6.x86_64 #1 SMP Fri Jul 13 12:50:12 UTC 2018 x86_64
libc:glibc 2.12 NPTL 2.12 
rlimit: STACK 10240k, CORE 0k, NPROC 1024, NOFILE 500000, AS infinity, DATA infinity, FSIZE infinity
load average:0.40 0.20 0.11

/proc/meminfo:
MemTotal:        7675280 kB
MemFree:         4839028 kB
Buffers:           23984 kB
Cached:           758868 kB
SwapCached:            0 kB
Active:          2280344 kB
Inactive:         345300 kB
Active(anon):    1842968 kB
Inactive(anon):      228 kB
Active(file):     437376 kB
Inactive(file):   345072 kB
Unevictable:           0 kB
Mlocked:               0 kB
SwapTotal:       5242876 kB
SwapFree:        5242876 kB
Dirty:                80 kB
Writeback:             0 kB
AnonPages:       1842804 kB
Mapped:            28584 kB
Shmem:               360 kB
Slab:             100196 kB
SReclaimable:      42464 kB
SUnreclaim:        57732 kB
KernelStack:        3344 kB
PageTables:        11028 kB
NFS_Unstable:          0 kB
Bounce:                0 kB
WritebackTmp:          0 kB
CommitLimit:     9080516 kB
Committed_AS:    5570476 kB
VmallocTotal:   34359738367 kB
VmallocUsed:       27104 kB
VmallocChunk:   34359699296 kB
HardwareCorrupted:     0 kB
AnonHugePages:   1562624 kB
HugePages_Total:       0
HugePages_Free:        0
HugePages_Rsvd:        0
HugePages_Surp:        0
Hugepagesize:       2048 kB
DirectMap4k:        6132 kB
DirectMap2M:     2615296 kB
DirectMap1G:     5242880 kB


/proc/sys/kernel/threads-max (system-wide limit on the number of threads):
58735


/proc/sys/vm/max_map_count (maximum number of memory map areas a process may have):
65530


/proc/sys/kernel/pid_max (system-wide limit on number of process identifiers):
32768




CPU:total 2 (initial active 2) (1 cores per cpu, 2 threads per core) family 6 model 63 stepping 0, cmov, cx8, fxsr, mmx, sse, sse2, sse3, ssse3, sse4.1, sse4.2, popcnt, avx, avx2, aes, clmul, erms, lzcnt, ht, tsc, tscinvbit, bmi1, bmi2, fma
CPU Model and flags from /proc/cpuinfo:
model name	: Intel(R) Xeon(R) CPU @ 2.30GHz
flags		: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cmov pat pse36 clflush mmx fxsr sse sse2 ss ht syscall nx pdpe1gb rdtscp lm constant_tsc rep_good xtopology nonstop_tsc unfair_spinlock eagerfpu pni pclmulqdq ssse3 fma cx16 pcid sse4_1 sse4_2 x2apic movbe popcnt aes xsave avx f16c rdrand hypervisor lahf_lm abm arat xsaveopt invpcid_single ssbd pti retpoline ibrs ibpb stibp fsgsbase bmi1 avx2 smep bmi2 erms invpcid arch_capabilities

Memory: 4k page, physical 7675280k(4839028k free), swap 5242876k(5242876k free)

vm_info: OpenJDK 64-Bit Server VM (11.0.2+9) for linux-amd64 JRE (11.0.2+9), built on Jan 17 2019 21:18:23 by "mach5one" with gcc 7.3.0

END.



FREQUENCY : always

Comments

Fix Request Backporting this patch to 11u keeps JVM from crashing on one of the code paths, like David says above. The patch does not apply cleanly to 11u, because surrounding code is a bit different. 11u RFR: https://mail.openjdk.java.net/pipermail/jdk-updates-dev/2019-March/000757.html
11-03-2019
Fix request: The backport to 12 applies cleanly and addresses a potential issue with unexpected code "in the wild". It is extremely low risk as it only affects one code path which would crash without the fix.
21-02-2019
Here's the patch for 12u http://cr.openjdk.java.net/~dholmes/8217765/webrev.12u/
21-02-2019
URL: http://hg.openjdk.java.net/jdk/jdk/rev/81a9748bc86c User: dholmes Date: 2019-02-21 00:42:02 +0000
21-02-2019
For the record the submitter was given a patched 11u binary to test and that resolved the problem. Fix will now be prepared for 13 and backported to 12u and 11u.
19-02-2019
Update from the submitter == After some days compiling and running with 11.0.2 the problem can not be reproduced so I consider this problem fixed. Thanks and sorry for not catching the old compiler version before. ==
18-02-2019
I can't see anything that definitely helps to pin down the reason why we end up in the code we do. The fact we fail when calling SystemDictionary::validate_protection_domain means that the class is loaded by that stage. That means it was either already loaded and the dictionary->find call rejected the protection_domain; or it was loaded incidentally or else concurrently while proceeding to the SystemDictionary::validate_protection_domain call. It seems more likely that it was in fact already loaded (with respect to the current classloader) but given the same_package check we have already passed I can see no way for the protection_domain to have been deemed invalid. Without a reproducer it will be near impossible to determine the exact failure scenario. I can put in additional checks on the klass_at paths but will be unable to validate them as I can't write a test case.
01-02-2019
No problem, David. The links I posted earlier point to the tip of jdk11u repo which is slightly ahead of 11.0.2-ga.
31-01-2019
Thanks for picking up on the slight code difference between 11 and 12+, though I don't think it is really material. I was thinking along the same lines as you that the problem may be related to the code executed before we actually try to load the class (which may already be loaded if it is the expected target class) and so lead us into Java code execution unexpectedly. It may also be that there has been a change in 11.0.2 compared to 11 GA. I will dig further. Many thanks again for the time you've put into this.
31-01-2019
Just found that 11 & 12/13 differ in InstanceKlass::has_nest_member(). Here's the version where the crash happened: http://hg.openjdk.java.net/jdk-updates/jdk11u/file/6cd57917d46d/src/hotspot/share/oops/instanceKlass.cpp#l164 // Check names first and if they match then check actual klass. This avoids // resolving anything unnecessarily. for (int i = 0; i < _nest_members->length(); i++) { int cp_index = _nest_members->at(i); Symbol* name = _constants->klass_name_at(cp_index); if (name == k->name()) { log_trace(class, nestmates)("- Found it at nest_members[%d] => cp[%d]", i, cp_index); // names match so check actual klass - this may trigger class loading if // it doesn't match (but that should be impossible) Klass* k2 = _constants->klass_at(cp_index, CHECK_false); if (k2 == k) { log_trace(class, nestmates)("- class is listed as a nest member"); return true; } else { // same name but different klass! log_trace(class, nestmates)(" - klass comparison failed!"); // can't have different classes for the same name, so we're done return false; } } } log_trace(class, nestmates)("- class is NOT a nest member!"); return false; } There's no CP tag check yet and the logic always goes through symbolic name comparison before checking resolved Klass. > The only scenario where I can see us reaching the impossible code is if we find the unresolved CP entry and attempt to resolve it, calling SystemDictionary::resolve_or_fail, and somehow the classloader context has changed, resulting in the existing class not being found and so we attempt to load through the changed classloader. But that itself should be impossible because the classloader associated with the CP of the NestHost class can't change. Agree. Same package check is enough to guarantee that symbolic comparison is as strong as comparing resolved classes (it's impossible to have 2 different classes with the same name in the same package). So, ConstantPool::klass_at() can be made optional and guarded by thread->can_call_java() check. Looking at SystemDictionary::resolve_or_fail(), I think the following check is crucial: ===== Klass SystemDictionary::resolve_instance_class_or_null(Symbol* name, Handle class_loader, Handle protection_domain, TRAPS) { ... class_loader = Handle(THREAD, java_lang_ClassLoader::non_reflection_class_loader(class_loader())); ClassLoaderData* loader_data = register_loader(class_loader); Dictionary* dictionary = loader_data->dictionary(); unsigned int d_hash = dictionary->compute_hash(name); // Do lookup to see if class already exist and the protection domain // has the right access // This call uses find which checks protection domain already matches // All subsequent calls use find_class, and set has_loaded_class so that // before we return a result we call out to java to check for valid protection domain // to allow returning the Klass* and add it to the pd_set if it is valid { Klass* probe = dictionary->find(d_hash, name, protection_domain); if (probe != NULL) return probe; } ... ===== If there's any chance "dictionary->find(d_hash, name, protection_domain)" can't find the loaded class (I suspect protection domain may cause some problems here), then execution will continue till protection domain checks (which triggered the upcall): ===== http://hg.openjdk.java.net/jdk-updates/jdk11u/file/6cd57917d46d/src/hotspot/share/classfile/systemDictionary.cpp#l882 // return if the protection domain in NULL if (protection_domain() == NULL) return k; // Check the protection domain has the right access if (dictionary->is_valid_protection_domain(d_hash, name, protection_domain)) { return k; } // Verify protection domain. If it fails an exception is thrown validate_protection_domain(k, class_loader, protection_domain, CHECK_NULL); return k; } ===== Unfortunately, I don't know enough about protection domains to reason about the checks happening in Dictionary::find()/is_valid_protection_domain(), but it seems if "dictionary->is_valid_protection_domain(d_hash, name, protection_domain)" fails, then "SystemDictionary::resolve_or_fail/resolve_or_null" is guaranteed to reach "validate_protection_domain" (irrespective of whether the class is already loaded or not).
31-01-2019
#1: I see 2 calls to ConstantPool::klass_at(). Are you 100% sure the crash happened in the latter one (!_constants->tag_at(cp_index).is_klass() and _constants->klass_name_at(cp_index) contains a Symbol*)? I am inferring that the crash happens at the latter one because the former will not result in class loading as is_klass() indicates the entry has already been resolved. #2: I assume it is possible to have duplicated entries (same class name) in NestHost attribute and they are resolved separately. Considering nest host & nest members are in the same package (checked in InstanceKlass::nest_host), it seems it should be enough to just check symbolic names, but _constants->klass_at(...) is forced in order to resolve the entry, right? If the other duplicated element is already resolved, I believe repeated lookup should hit `dictionary->find(d_hash, name, protection_domain)` check in `SystemDictionary::resolve_instance_class_or_null` and bypass repeated class loading & protection domain validation. Does it agree with your understanding of the code? I assume you mean NestMembers attribute not NestHost. Yes it can contain duplicate entries. The duplicate entry should normally not get resolved as we should always hit on the first entry. The duplicate entry would get resolved if we somehow were able to load a second class of the same name, claiming to be in the same nest (which is what appears to be happening in this case) and the first entry was already resolved to the correct class of this name. I don't think duplicate entries is really relevant to this case, at least you don't need a duplicate entry to hit the code in question. If the name is the same we expect the klass to resolve to the klass involved in the query, but we need to verify that in case there could be a strange/unexpected use of classloaders. That said in exploring the impossible scenario further I can't get past the same-package check if the target class is loaded by a different classloader. The only scenario where I can see us reaching the impossible code is if we find the unresolved CP entry and attempt to resolve it, calling SystemDictionary::resolve_or_fail, and somehow the classloader context has changed, resulting in the existing class not being found and so we attempt to load through the changed classloader. But that itself should be impossible because the classloader associated with the CP of the NestHost class can't change. I can "fix" this by guarding the klass_at call in the matching name case using the same "can execute Java" condition as we do in nest_host(). That will protect this code path. But it doesn't explain how we might hit this code path. The code path through SystemDictionary::resolve_or_fail/resolve_or_null was slightly modified by the nestmates project to push an assertion "can execute Java code" deeper into the logic to where it was certain we would actually have to load the class. That allowed us to resolve nestmember/nesthost entries in the cases where the class was in fact already loaded.
31-01-2019
Yes, my bad. ======= InstanceKlass* InstanceKlass::nest_host(Symbol* validationException, TRAPS) { ... // JVMS 5.4.4 indicates package check comes first if (is_same_class_package(k)) { // Now check actual membership. We can't be a member if our "host" is // not an instance class. if (k->is_instance_klass()) { nest_host_k = InstanceKlass::cast(k); bool is_member = nest_host_k->has_nest_member(this, CHECK_NULL); ======= bool InstanceKlass::has_nest_member(InstanceKlass* k, TRAPS) const { ... for (int i = 0; i < _nest_members->length(); i++) { int cp_index = _nest_members->at(i); if (_constants->tag_at(cp_index).is_klass()) { Klass* k2 = _constants->klass_at(cp_index, CHECK_false); ... } else { Symbol* name = _constants->klass_name_at(cp_index); if (name == k->name()) { // names match so check actual klass - this may trigger class loading if // it doesn't match (but that should be impossible) Klass* k2 = _constants->klass_at(cp_index, CHECK_false); ======= 2 questions: #1: I see 2 calls to ConstantPool::klass_at(). Are you 100% sure the crash happened in the latter one (!_constants->tag_at(cp_index).is_klass() and _constants->klass_name_at(cp_index) contains a Symbol*)? #2: I assume it is possible to have duplicated entries (same class name) in NestHost attribute and they are resolved separately. Considering nest host & nest members are in the same package (checked in InstanceKlass::nest_host), it seems it should be enough to just check symbolic names, but _constants->klass_at(...) is forced in order to resolve the entry, right? If the other duplicated element is already resolved, I believe repeated lookup should hit `dictionary->find(d_hash, name, protection_domain)` check in `SystemDictionary::resolve_instance_class_or_null` and bypass repeated class loading & protection domain validation. Does it agree with your understanding of the code? Overall, I find it very hard to reason about the logic in `SystemDictionary::resolve_or_fail()` et al w.r.t. execution in compiler thread. I'd prefer to see access check conservatively bails out if it needs to call into it.
30-01-2019
> V [libjvm.so+0x8608c0] ConstantPool::klass_at(int, Thread)+0x60 > V [libjvm.so+0x85af72] InstanceKlass::has_nest_member(InstanceKlass, Thread) const+0x122 The call is from has_nest_member not from nest_host. This is not the code you are referencing above. The problem is in has_nest_member where we hit an "impossible case: // names match so check actual klass - this may trigger class loading if // it doesn't match (but that should be impossible) Klass k2 = _constants->klass_at(cp_index, CHECK_false);
30-01-2019
Top of stack: V [libjvm.so+0x8833ad] JavaCalls::call_special(JavaValue, Handle, Klass, Symbol, Symbol, Handle, Handle, Thread)+0x1bd V [libjvm.so+0xd857e6] SystemDictionary::validate_protection_domain(InstanceKlass, Handle, Handle, Thread)+0x226 V [libjvm.so+0xd87634] SystemDictionary::resolve_instance_class_or_null(Symbol, Handle, Handle, Thread)+0xa84 V [libjvm.so+0xd89dc2] SystemDictionary::resolve_or_fail(Symbol, Handle, Handle, bool, Thread)+0x52 V [libjvm.so+0x68debd] ConstantPool::klass_at_impl(constantPoolHandle const&, int, bool, Thread)+0xbd V [libjvm.so+0x8608c0] ConstantPool::klass_at(int, Thread)+0x60 V [libjvm.so+0x85af72] InstanceKlass::has_nest_member(InstanceKlass, Thread) const+0x122 V [libjvm.so+0x85b3a6] InstanceKlass::nest_host(Symbol, Thread) [clone .part.112]+0x316 V [libjvm.so+0x85b665] InstanceKlass::nest_host(Symbol, Thread)+0x95 ==== InstanceKlass InstanceKlass::nest_host(Symbol* validationException, TRAPS) { ... if (!THREAD->can_call_java() && !_constants->tag_at(_nest_host_index).is_klass()) { ... Klass* k = _constants->klass_at(_nest_host_index, THREAD); === Klass* ConstantPool::klass_at_impl(const constantPoolHandle& this_cp, int which, ... Klass* klass = this_cp->resolved_klasses()->at(resolved_klass_index); if (klass != NULL) { return klass; } ... Klass* k = SystemDictionary::resolve_or_fail(name, loader, protection_domain, true, THREAD); ================== Offending upcall happens in `SystemDictionary::validate_protection_domain()` which is called from the very end of `SystemDictionary::resolve_instance_class_or_null()`. If class were actually loaded, I'd expect an upcall to happen much earlier. It seems more likely one of the checks under locks found the Klass, but after `dictionary->find(...)` call: InstanceKlass SystemDictionary::resolve_instance_class_or_null(Symbol* name, ... { InstanceKlass* probe = dictionary->find(d_hash, name, protection_domain); if (probe != NULL) return probe; } ... // return if the protection domain in NULL if (protection_domain() == NULL) return k; // Check the protection domain has the right access if (dictionary->is_valid_protection_domain(d_hash, name, protection_domain)) { return k; } // Verify protection domain. If it fails an exception is thrown validate_protection_domain(k, class_loader, protection_domain, CHECK_NULL); return k; } ================== Leaving a possibility of calling into `SystemDictionary::resolve_or_fail()` from compiler threads seems erroneous to me. I'd expect to see a stricter check in `InstanceKlass::nest_host(...)` to completely avoid `ConstantPool::klass_at(...)` call unless `_constants->resolved_klasses()->at(resolved_klass_index)` is already populated. From a compiler perspective, there's nothing wrong with performing a stricter check than necessary. It's treated as if the klass/method are not-yet-loaded (or already unloaded).
30-01-2019
There may be a race in that someone may concurrently resolve the same CP entry, but you won't see it switch from a Klass* back to a Symbol. So if you see a Klass then it is a resolved CP entry and no class loading will occur. If you see a Symbol* then it is unresolved and so class loading is likely to occur. We skip the attempt to resolve the CP entry if the current thread can't run Java and it appears the CP entry needs resolving.
30-01-2019
I'm curious why the following check is enough to reliably avoid resolution attempt in the context of compiler threads: http://hg.openjdk.java.net/jdk/jdk/file/tip/src/hotspot/share/oops/instanceKlass.cpp#l218 if (!THREAD->can_call_java() && !_constants->tag_at(_nest_host_index).is_klass()) { The following comment in ConstantPool::klass_at_impl makes me suspect there may be a race here: http://hg.openjdk.java.net/jdk/jdk/file/tip/src/hotspot/share/oops/constantPool.cpp#l453 // A resolved constantPool entry will contain a Klass, otherwise a Symbol. // It is not safe to rely on the tag bit's here, since we don't have a lock, and // the entry and tag is not updated atomicly.
30-01-2019
Can the submitter comment on how different classloaders may be used with the class shown in the code above? It's possible the application/library is using classloaders in a way that is not compatible with the way nestmates are defined in JDK 11. Specifically all nested types must be loaded by the same classloader as the enclosing type, to ensure they are all in the same package, if they are to rely on nestmate access to private members. Potential workaround: compile the problematic classes using -release 10 or -source/-target 10, so that the new nestmate logic is not used.
29-01-2019
Please see JDK-8189940 for discussion of the general issue of compiler threads needing to load nest-host classes and how we resolved it. It seems the current bug is hitting "impossible" code that involves another nest member not the nest-host. The target class we are checking is already loaded (from the perspective of the current class) as we are doing: klass_a->has_nestmate_access_to(klass_b) and the two classes have the same nest_host listed, but when we ask the nest_host to validate that the target class is a member, it has a member with the same name listed, but that member has not been loaded! This is AFAICS only theoretically possible if the target class has been loaded by a different classloader. But it would seem to imply a loader constraint violation to me and hence should be impossible. Though it's possible the constraint violation would only be detected if the current classloading attempt actually went ahead ...
29-01-2019
> We made a fix to the compiler to reject compilation forcing it back to the interpreter so that the access check could be performed. [~dholmes] can you point to the relevant place in the code, please? The only thing compilers can do in that vein is to reject any compilation/inlining requests for a method in a class without fully loaded nest-host. It may be too aggressive (in the worst case, nest-host loading may be delayed indefinitely). Compilers are robust against partially resolved/initialized code and properly communicating such information to compilers is the preferred way to approach the problem. (ciEnv uses unloaded_method() / unloaded_klass() to represent affected code.)
25-01-2019
You can't just not load the nest-host if you need to perform an access check! These access checks are only supposed to be needed in Java code processing threads! The whole thing falls apart if non-JavaThreads need to load nest-hosts! We made a fix to the compiler to reject compilation forcing it back to the interpreter so that the access check could be performed. It's not at all clear why this current situation is not handled, or if we missed this case, how we managed to miss it despite so much testing! That said the comments above seems to suggest something more bizarre is occurring with the compiled code.
25-01-2019
Additional information from submitter == The crash came after I tweaked a method to diagnose a very critical potential bug. (sorry for the lack of formatting) This method is called millions of times and the returned objects are kept in the heap for long time. public static DocAction create(Doc doc, String type, Account who, long when) { switch (type) { case "DocEdited": return new DocEdited(doc, who, when); case "DocVisited": return new DocVisited(doc, who, when); case "DocDownloaded": return new DocDownloaded(doc, who, when); case "DocPrinted": return new DocPrinted(doc, who, when); case "DocAclSharedOut": return new DocAclSharedOut(doc, who, when); case "DocAclChange": return new DocAclChange(doc, who, when); case "DocAclChange2": return new DocAclChange2(doc, who, when); case "DocAclPublicOn": return new DocAclPublicOn(doc, who, when); case "DocAclPublicOff": return new DocAclPublicOff(doc, who, when); } throw new IllegalArgumentException(type); } When compiled with Java 11 we started receiving errors: IllegalArgumentException: DocAclPublicOn at com.baker.gat.actions.DocUserActions$DocAction.create(DocUserActions.java:58) As those errors are impossible I added a type = type.trim(); before the switch and moved the DocAclPublicOn case to the first position. Then after deploying the modified version ALL of the instances running that code crashed with the reported one.
25-01-2019
V [libjvm.so+0x85b6d6] InstanceKlass::has_nestmate_access_to(InstanceKlass, Thread)+0x66 V [libjvm.so+0xca5e8d] Reflection::verify_member_access(Klass const, Klass const, Klass const, AccessFlags, bool, bool, Thread)+0x27d V [libjvm.so+0xa63ddd] LinkResolver::check_method_accessability(Klass, Klass, Klass, methodHandle const&, Thread)+0x5d V [libjvm.so+0xa64604] LinkResolver::resolve_method(LinkInfo const&, Bytecodes::Code, Thread)+0x144 V [libjvm.so+0xa6a376] LinkResolver::resolve_special_call_or_null(LinkInfo const&)+0xb6 V [libjvm.so+0x577bd3] ciEnv::lookup_method(ciInstanceKlass, ciKlass, Symbol, Symbol*, Bytecodes::Code, constantTag)+0x163 The nestmates code is supposed to guard the cases where we attempt to resolve an access check in the compiler thread, so that it won't try to load the call - which is not permitted and why we fail the guarantee.
25-01-2019
There is no reproducer. There is C1 crash in thirdparty code com.baker.gat.docs.Doc$DocResourceComparator::compareDetails (522 bytes) There is an attemp to make thread to call java from the native compiler resulted in assert failure.
25-01-2019
I wanted to call has_nestmate_access_to from the VMThread during redefinition, so I added this: http://cr.openjdk.java.net/~coleenp/add_delete/webrev/src/hotspot/share/oops/instanceKlass.cpp.udiff.html It would be nice if we had an has_nestmate_access_to_noresolve sort of function.
25-01-2019