JDK-8027456 : native heap OOM using ParallelGC
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 7u40
  • Priority: P3
  • Status: Resolved
  • Resolution: Cannot Reproduce
  • OS: solaris_10
  • Submitted: 2013-10-08
  • Updated: 2013-12-16
  • Resolved: 2013-12-16
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 7
7u60Resolved
Description
FULL PRODUCT VERSION :
java version "1.7.0_40"
Java(TM) SE Runtime Environment (build 1.7.0_40-b43)
Java HotSpot(TM) Server VM (build 24.0-b56, mixed mode)

also valid for

java version "1.7.0_25"
Java(TM) SE Runtime Environment (build 1.7.0_25-b15)
Java HotSpot(TM) Server VM (build 23.25-b01, mixed mode)


FULL OS VERSION :
SunOS <machine> 5.10 Generic_147440-01 sun4u sparc SUNW,SPARC-Enterprise

A DESCRIPTION OF THE PROBLEM :
We are currently in the process of upgrading our product from Java 6 to Java 7.

Effectively we are switching from Java 1.6.0_27 to Java 1.7.0_25. For the sake of testing with the most recent JVM, we also repeated our test with 1.7.0_40 - results were the same.

During performance tests we are seeing a growth in native heap (via pmap for the jvm's process) that we cannot explain. The growth continues until the 4gb boundary of the 32 bit process of the VM, and then the VM dies due to a OOM.

We noticed that this behavior seems to be specific to ParallelGC collector. When we switch to the CMS collector, the total native heap stays stable and hence the VM does not run out of heap space.

We also repeated the test for Java 1.6.0_27 with ParallelGC and there the problem is NOT present.

We are therefore assuming, that there's a potential memory leak in the implementation of the ParallelGC in Java 7.

THE PROBLEM WAS REPRODUCIBLE WITH -Xint FLAG: Did not try

THE PROBLEM WAS REPRODUCIBLE WITH -server FLAG: Yes

REGRESSION.  Last worked in version 6u43

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Sorry, as of now, I have no idea how to reproduce this outside of our application tests suite.

EXPECTED VERSUS ACTUAL BEHAVIOR :
expected result would be as of 1.6.0_27, i.e: nothing special, heap and total Kb are stable.


Actual result for 1.7.0_25++:
native heap keeps growing according to pmap
ERROR MESSAGES/STACK TRACES THAT OCCUR :
#
# There is insufficient memory for the Java Runtime Environment to continue.
# Native memory allocation (malloc) failed to allocate 11328 bytes for AllocateHeap
# Possible reasons:
#   The system is out of physical RAM or swap space
#   In 32 bit mode, the process size limit was hit
# Possible solutions:
#   Reduce memory load on the system
#   Increase physical memory or swap space
#   Check if swap backing store is full
#   Use 64 bit Java on a 64 bit OS
#   Decrease Java heap size (-Xmx/-Xms)
#   Decrease number of Java threads
#   Decrease Java thread stack sizes (-Xss)
#   Set larger code cache with -XX:ReservedCodeCacheSize=
# This output file may be truncated or incomplete.
#
#  Out of Memory Error (allocation.inline.hpp:60), pid=16518, tid=326
#
# JRE version: 7.0_25-b15
# Java VM: Java HotSpot(TM) Server VM (23.25-b01 mixed mode solaris-sparc )
# Core dump written. Default location: /data2/SFleege-Althoff/orgadata_system_sfleege_d25/core or core.16518
#

---------------  T H R E A D  ---------------

Current thread (0x12b21000):  JavaThread "WorkerThread#22[192.168.114.134:54628]" [_thread_in_vm, id=326, stack(0x4f100000,0x4f180000)]

Stack: [0x4f100000,0x4f180000],  sp=0x4f17d0c8,  free space=500k
Native frames: (J=compiled Java code, j=interpreted, Vv=VM code, C=native code)
V  [libjvm.so+0xabaf6c]  void VMError::report_and_die()+0x7bc
V  [libjvm.so+0x557ad8]  void report_vm_out_of_memory(const char*,int,unsigned,const char*)+0x88
V  [libjvm.so+0x856ccc]  void LoaderConstraintTable::extend_loader_constraint(LoaderConstraintEntry*,Handle,klassOopDesc*)+0x60
V  [libjvm.so+0x85616c]  bool LoaderConstraintTable::add_entry(Symbol*,klassOopDesc*,Handle,klassOopDesc*,Handle)+0x3b0
V  [libjvm.so+0xa434cc]  char*SystemDictionary::check_signature_loaders(Symbol*,Handle,Handle,bool,Thread*)+0x390
V  [libjvm.so+0x81b7f8]  bool klassVtable::update_inherited_vtable(instanceKlass*,methodHandle,int,bool,Thread*)+0x6bc
V  [libjvm.so+0x81ade8]  void klassVtable::initialize_vtable(bool,Thread*)+0x27c
V  [libjvm.so+0x63c2cc]  bool instanceKlass::link_class_impl(instanceKlassHandle,bool,Thread*)+0x66c
V  [libjvm.so+0x24d744]  void instanceKlass::link_class(Thread*)+0xac
V  [libjvm.so+0x252c6c]  JVM_GetClassDeclaredConstructors+0x1fc
j  java.lang.Class.getDeclaredConstructors0(Z)[Ljava/lang/reflect/Constructor;+8188
j  java.lang.Class.getDeclaredConstructors0(Z)[Ljava/lang/reflect/Constructor;+0
J  java.lang.Class.privateGetDeclaredConstructors(Z)[Ljava/lang/reflect/Constructor;
J  java.lang.Class.newInstance()Ljava/lang/Object;
j  sun.reflect.MethodAccessorGenerator$1.run()Lsun/reflect/MagicAccessorImpl;+24
j  sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x1b259c]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x320
V  [libjvm.so+0x20af78]  JVM_DoPrivileged+0xe58
C  [libjava.so+0x86b0]  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedAction_2+0x1c
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
j  sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;+755
j  sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(Ljava/lang/Class;[Ljava/lang/Class;[Ljava/lang/Class;ILjava/lang/Class;)Lsun/reflect/SerializationConstructorAccessorImpl;+15
j  sun.reflect.ReflectionFactory.newConstructorForSerialization(Ljava/lang/Class;Ljava/lang/reflect/Constructor;)Ljava/lang/reflect/Constructor;+34
j  java.io.ObjectStreamClass.getSerializableConstructor(Ljava/lang/Class;)Ljava/lang/reflect/Constructor;+64
j  java.io.ObjectStreamClass.access$1500(Ljava/lang/Class;)Ljava/lang/reflect/Constructor;+1
j  java.io.ObjectStreamClass$2.run()Ljava/lang/Void;+176
j  java.io.ObjectStreamClass$2.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
V  [libjvm.so+0x1b259c]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x320
V  [libjvm.so+0x20af78]  JVM_DoPrivileged+0xe58
C  [libjava.so+0x86b0]  Java_java_security_AccessController_doPrivileged__Ljava_security_PrivilegedAction_2+0x1c
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+106
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+75
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+75
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
J  java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V
J  java.io.ObjectOutputStream.defaultWriteFields(Ljava/lang/Object;Ljava/io/ObjectStreamClass;)V
J  java.io.ObjectOutputStream.writeSerialData(Ljava/lang/Object;Ljava/io/ObjectStreamClass;)V
J  java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V
J  java.io.ObjectOutputStream.writeObject(Ljava/lang/Object;)V
j  org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(Ljava/io/ObjectOutputStream;Ljava/lang/Object;)V+6
j  org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(Ljava/io/ObjectOutputStream;Ljava/lang/Object;I)V+48
j  org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(Ljava/lang/Object;Ljava/io/OutputStream;I)V+22
j  org.jboss.invocation.unified.marshall.InvocationMarshaller.write(Ljava/lang/Object;Ljava/io/OutputStream;I)V+8
j  org.jboss.remoting.transport.socket.ServerThread.versionedWrite(Ljava/io/OutputStream;Lorg/jboss/remoting/transport/socket/SocketServerInvoker;Ljava/lang/ClassLoader;Ljava/lang/Object;I)V+58
j  org.jboss.remoting.transport.socket.ServerThread.completeInvocation(Lorg/jboss/remoting/transport/socket/SocketWrapper;Ljava/io/InputStream;Ljava/io/OutputStream;ZI)V+389
j  org.jboss.remoting.transport.socket.ServerThread.processInvocation(Lorg/jboss/remoting/transport/socket/SocketWrapper;Ljava/io/InputStream;Ljava/io/OutputStream;)V+58
j  org.jboss.remoting.transport.socket.ServerThread.dorun()V+428
j  org.jboss.remoting.transport.socket.ServerThread.run()V+5
v  ~StubRoutines::call_stub
V  [libjvm.so+0x1b259c]  void JavaCalls::call_helper(JavaValue*,methodHandle*,JavaCallArguments*,Thread*)+0x320
V  [libjvm.so+0x674890]  void JavaCalls::call_virtual(JavaValue*,Handle,KlassHandle,Symbol*,Symbol*,Thread*)+0x19c
V  [libjvm.so+0x262420]  void thread_entry(JavaThread*,Thread*)+0x170
V  [libjvm.so+0xa5efa4]  void JavaThread::thread_main_inner()+0x90
V  [libjvm.so+0x25deb4]  void JavaThread::run()+0x3e4
V  [libjvm.so+0x94c828]  java_start+0x338

Java frames: (J=compiled Java code, j=interpreted, Vv=VM code)
j  java.lang.Class.getDeclaredConstructors0(Z)[Ljava/lang/reflect/Constructor;+0
J  java.lang.Class.privateGetDeclaredConstructors(Z)[Ljava/lang/reflect/Constructor;
J  java.lang.Class.newInstance()Ljava/lang/Object;
j  sun.reflect.MethodAccessorGenerator$1.run()Lsun/reflect/MagicAccessorImpl;+24
j  sun.reflect.MethodAccessorGenerator$1.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
j  sun.reflect.MethodAccessorGenerator.generate(Ljava/lang/Class;Ljava/lang/String;[Ljava/lang/Class;Ljava/lang/Class;[Ljava/lang/Class;IZZLjava/lang/Class;)Lsun/reflect/MagicAccessorImpl;+755
j  sun.reflect.MethodAccessorGenerator.generateSerializationConstructor(Ljava/lang/Class;[Ljava/lang/Class;[Ljava/lang/Class;ILjava/lang/Class;)Lsun/reflect/SerializationConstructorAccessorImpl;+15
j  sun.reflect.ReflectionFactory.newConstructorForSerialization(Ljava/lang/Class;Ljava/lang/reflect/Constructor;)Ljava/lang/reflect/Constructor;+34
j  java.io.ObjectStreamClass.getSerializableConstructor(Ljava/lang/Class;)Ljava/lang/reflect/Constructor;+64
j  java.io.ObjectStreamClass.access$1500(Ljava/lang/Class;)Ljava/lang/reflect/Constructor;+1
j  java.io.ObjectStreamClass$2.run()Ljava/lang/Void;+176
j  java.io.ObjectStreamClass$2.run()Ljava/lang/Object;+1
v  ~StubRoutines::call_stub
J  java.security.AccessController.doPrivileged(Ljava/security/PrivilegedAction;)Ljava/lang/Object;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+106
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+75
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
j  java.io.ObjectStreamClass.<init>(Ljava/lang/Class;)V+75
J  java.io.ObjectStreamClass.lookup(Ljava/lang/Class;Z)Ljava/io/ObjectStreamClass;
J  java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V
J  java.io.ObjectOutputStream.defaultWriteFields(Ljava/lang/Object;Ljava/io/ObjectStreamClass;)V
J  java.io.ObjectOutputStream.writeSerialData(Ljava/lang/Object;Ljava/io/ObjectStreamClass;)V
J  java.io.ObjectOutputStream.writeObject0(Ljava/lang/Object;Z)V
J  java.io.ObjectOutputStream.writeObject(Ljava/lang/Object;)V
j  org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObjectVersion2_2(Ljava/io/ObjectOutputStream;Ljava/lang/Object;)V+6
j  org.jboss.remoting.serialization.impl.java.JavaSerializationManager.sendObject(Ljava/io/ObjectOutputStream;Ljava/lang/Object;I)V+48
j  org.jboss.remoting.marshal.serializable.SerializableMarshaller.write(Ljava/lang/Object;Ljava/io/OutputStream;I)V+22
j  org.jboss.invocation.unified.marshall.InvocationMarshaller.write(Ljava/lang/Object;Ljava/io/OutputStream;I)V+8
j  org.jboss.remoting.transport.socket.ServerThread.versionedWrite(Ljava/io/OutputStream;Lorg/jboss/remoting/transport/socket/SocketServerInvoker;Ljava/lang/ClassLoader;Ljava/lang/Object;I)V+58
j  org.jboss.remoting.transport.socket.ServerThread.completeInvocation(Lorg/jboss/remoting/transport/socket/SocketWrapper;Ljava/io/InputStream;Ljava/io/OutputStream;ZI)V+389
j  org.jboss.remoting.transport.socket.ServerThread.processInvocation(Lorg/jboss/remoting/transport/socket/SocketWrapper;Ljava/io/InputStream;Ljava/io/OutputStream;)V+58
j  org.jboss.remoting.transport.socket.ServerThread.dorun()V+428
j  org.jboss.remoting.transport.socket.ServerThread.run()V+5
v  ~StubRoutines::call_stub

---------------  P R O C E S S  ---------------

Java Threads: ( => current thread )
  0x12b48000 JavaThread "WorkerThread#31[192.168.114.134:54618]" [_thread_in_Java, id=336, stack(0x4e700000,0x4e780000)]
  0x130cec00 JavaThread "WorkerThread#30[192.168.114.134:54616]" [_thread_blocked, id=335, stack(0x4e800000,0x4e880000)]
  0x132ef400 JavaThread "WorkerThread#29[192.168.114.134:54617]" [_thread_in_Java, id=334, stack(0x4e900000,0x4e980000)]
  0x132cdc00 JavaThread "WorkerThread#28[192.168.114.134:54615]" [_thread_in_Java, id=333, stack(0x4ea00000,0x4ea80000)]
  0x10ef3c00 JavaThread "WorkerThread#27[192.168.114.134:54614]" [_thread_in_Java, id=332, stack(0x4eb00000,0x4eb80000)]
  0x10e71400 JavaThread "WorkerThread#26[192.168.114.134:54613]" [_thread_in_Java, id=331, stack(0x4ec00000,0x4ec80000)]
  0x11c4a000 JavaThread "WorkerThread#25[192.168.114.134:54612]" [_thread_blocked, id=330, stack(0x4ed00000,0x4ed80000)]
  0x12348000 JavaThread "WorkerThread#24[192.168.114.134:54611]" [_thread_in_Java, id=329, stack(0x4ee00000,0x4ee80000)]
  0x12a97c00 JavaThread "RMI TCP Connection(11)-192.168.114.134" daemon [_thread_in_native, id=328, stack(0x4ef00000,0x4ef80000)]
  0x12b81c00 JavaThread "WorkerThread#23[192.168.114.134:54629]" [_thread_in_Java, id=327, stack(0x4f000000,0x4f080000)]
=>0x12b21000 JavaThread "WorkerThread#22[192.168.114.134:54628]" [_thread_in_vm, id=326, stack(0x4f100000,0x4f180000)]
  0x12a7bc00 JavaThread "WorkerThread#21[192.168.114.134:54627]" [_thread_in_Java, id=325, stack(0x4f200000,0x4f280000)]
  0x12b22800 JavaThread "WorkerThread#20[192.168.114.134:54626]" [_thread_in_Java, id=324, stack(0x4f300000,0x4f380000)]
  0x12309000 JavaThread "RMI TCP Connection(12)-192.168.114.134" daemon [_thread_in_native, id=323, stack(0x4f400000,0x4f480000)]
  0x125c2400 JavaThread "WorkerThread#19[192.168.114.134:54596]" [_thread_in_Java, id=322, stack(0x4f500000,0x4f580000)]
  0x126fe400 JavaThread "WorkerThread#18[192.168.114.134:54593]" [_thread_in_Java, id=321, stack(0x4f600000,0x4f680000)]
  0x005f4400 JavaThread "WorkerThread#17[192.168.114.134:54595]" [_thread_in_Java, id=320, stack(0x4f700000,0x4f780000)]
  0x121da000 JavaThread "WorkerThread#16[192.168.114.134:54594]" [_thread_in_Java, id=319, stack(0x4f800000,0x4f880000)]
  0x121d4c00 JavaThread "RMI TCP Connection(13)-192.168.114.134" daemon [_thread_in_native, id=318, stack(0x4f900000,0x4f980000)]
  0x10ffa000 JavaThread "RMI TCP Connection(14)-192.168.114.134" daemon [_thread_in_native, id=317, stack(0x4fa00000,0x4fa80000)]
  0x11b9fc00 JavaThread "WorkerThread#15[192.168.114.134:54623]" [_thread_in_Java, id=316, stack(0x4fb00000,0x4fb80000)]
  0x05a22c00 JavaThread "WorkerThread#14[192.168.114.134:54622]" [_thread_in_Java, id=315, stack(0x4fc00000,0x4fc80000)]
  0x017d4c00 JavaThread "WorkerThread#13[192.168.114.134:54621]" [_thread_blocked, id=314, stack(0x4fd00000,0x4fd80000)]
  0x11d95800 JavaThread "WorkerThread#12[192.168.114.134:54624]" [_thread_in_Java, id=313, stack(0x4fe00000,0x4fe80000)]
  0x1141ec00 JavaThread "WorkerThread#11[192.168.114.134:54630]" [_thread_in_Java, id=312, stack(0x4ff00000,0x4ff80000)]
  0x08a7ec00 JavaThread "WorkerThread#10[192.168.114.134:54634]" [_thread_in_Java, id=311, stack(0x50000000,0x50080000)]
  0x113a6000 JavaThread "WorkerThread#9[192.168.114.134:54631]" [_thread_in_Java, id=310, stack(0x50100000,0x50180000)]
  0x114bec00 JavaThread "WorkerThread#8[192.168.114.134:54633]" [_thread_in_Java, id=309, stack(0x50200000,0x50280000)]
  0x1118d400 JavaThread "RMI TCP Connection(15)-192.168.114.134" daemon [_thread_in_native, id=308, stack(0x50300000,0x50380000)]
  0x0f708000 JavaThread "WorkerThread#7[192.168.114.134:54602]" [_thread_in_Java, id=307, stack(0x50400000,0x50480000)]
  0x1070a000 JavaThread "WorkerThread#6[192.168.114.134:54604]" [_thread_in_Java, id=306, stack(0x50500000,0x50580000)]
  0x1022ac00 JavaThread "WorkerThread#5[192.168.114.134:54605]" [_thread_in_Java, id=305, stack(0x50700000,0x50780000)]
  0x10306800 JavaThread "WorkerThread#4[192.168.114.134:54603]" [_thread_in_Java, id=304, stack(0x50600000,0x50680000)]
  0x0fd91000 JavaThread "RMI TCP Connection(16)-192.168.114.134" daemon [_thread_in_native, id=303, stack(0x50800000,0x50880000)]
  0x10585000 JavaThread "JBoss System Threads(1)-10" daemon [_thread_blocked, id=302, stack(0x50900000,0x50980000)]
  0x10e1cc00 JavaThread "WorkerThread#3[192.168.114.134:54608]" [_thread_in_Java, id=297, stack(0x50a00000,0x50a80000)]
  0x10e1b400 JavaThread "WorkerThread#2[192.168.114.134:54609]" [_thread_in_Java, id=296, stack(0x50b00000,0x50b80000)]
  0x0b656c00 JavaThread "WorkerThread#1[192.168.114.134:54607]" [_thread_in_Java, id=295, stack(0x50c00000,0x50c80000)]
  0x0b655400 JavaThread "WorkerThread#0[192.168.114.134:54606]" [_thread_in_Java, id=294, stack(0x50d00000,0x50d80000)]
  0x10a2e400 JavaThread "ZipFile Lock Reaper" daemon [_thread_blocked, id=293, stack(0x50e00000,0x50e80000)]
  0x10e31000 JavaThread "RMI TCP Connection(17)-192.168.114.134" daemon [_thread_in_native, id=292, stack(0x50f00000,0x50f80000)]
  0x02dd9400 JavaThread "JBoss System Threads(1)-9" daemon [_thread_blocked, id=291, stack(0x5fe00000,0x5fe80000)]
  0x0163ec00 JavaThread "RMI TCP Connection(18)-192.168.114.134" daemon [_thread_in_native, id=290, stack(0x5bc00000,0x5bc80000)]
  0x0ac96c00 JavaThread "JBoss System Threads(1)-8" daemon [_thread_blocked, id=289, stack(0x60780000,0x60800000)]
  0x1036b800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-134" daemon [_thread_in_native, id=287, stack(0x51000000,0x51080000)]
  0x004cd800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-133" daemon [_thread_in_native, id=286, stack(0x51100000,0x51180000)]
  0x1091ec00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-132" daemon [_thread_in_native, id=285, stack(0x51200000,0x51280000)]
  0x10211800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-131" daemon [_thread_in_native, id=284, stack(0x51300000,0x51380000)]
  0x1073f000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-130" daemon [_thread_in_native, id=283, stack(0x59700000,0x59780000)]
  0x10385c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-129" daemon [_thread_in_native, id=282, stack(0x60980000,0x60a00000)]
  0x00028000 JavaThread "DestroyJavaVM" [_thread_blocked, id=2, stack(0xfdf00000,0xfdf80000)]
  0x10e31c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-128" daemon [_thread_in_native, id=281, stack(0x51400000,0x51480000)]
  0x10e2e800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-127" daemon [_thread_in_native, id=280, stack(0x51500000,0x51580000)]
  0x10e2d000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-126" daemon [_thread_in_native, id=279, stack(0x51600000,0x51680000)]
  0x10e2b000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-125" daemon [_thread_in_native, id=278, stack(0x51700000,0x51780000)]
  0x10e29c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-124" daemon [_thread_in_native, id=277, stack(0x51800000,0x51880000)]
  0x10dc7400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-123" daemon [_thread_in_native, id=276, stack(0x51900000,0x51980000)]
  0x10dc5c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-122" daemon [_thread_in_native, id=275, stack(0x51a00000,0x51a80000)]
  0x10dc3400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-121" daemon [_thread_in_native, id=274, stack(0x51b00000,0x51b80000)]
  0x10dc1800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-120" daemon [_thread_in_native, id=273, stack(0x51c00000,0x51c80000)]
  0x10dbec00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-119" daemon [_thread_in_native, id=272, stack(0x51d00000,0x51d80000)]
  0x10dbd400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-118" daemon [_thread_in_native, id=271, stack(0x51e00000,0x51e80000)]
  0x10dbc000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-117" daemon [_thread_in_native, id=270, stack(0x51f00000,0x51f80000)]
  0x107a1400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-116" daemon [_thread_in_native, id=269, stack(0x52000000,0x52080000)]
  0x10e73000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-115" daemon [_thread_in_native, id=268, stack(0x52100000,0x52180000)]
  0x10d7c000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-114" daemon [_thread_in_native, id=267, stack(0x52200000,0x52280000)]
  0x10b28800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-113" daemon [_thread_in_native, id=266, stack(0x52300000,0x52380000)]
  0x10d80800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-112" daemon [_thread_in_native, id=265, stack(0x52400000,0x52480000)]
  0x10e79800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-111" daemon [_thread_in_native, id=264, stack(0x52500000,0x52580000)]
  0x10d94400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-110" daemon [_thread_in_native, id=263, stack(0x52600000,0x52680000)]
  0x10d83000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-109" daemon [_thread_in_native, id=262, stack(0x52700000,0x52780000)]
  0x10d7d800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-108" daemon [_thread_in_native, id=261, stack(0x52800000,0x52880000)]
  0x10d85c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-107" daemon [_thread_in_native, id=260, stack(0x52900000,0x52980000)]
  0x10d84400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-106" daemon [_thread_in_native, id=259, stack(0x52a00000,0x52a80000)]
  0x10d81400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-105" daemon [_thread_in_native, id=258, stack(0x52b00000,0x52b80000)]
  0x10d7e400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-104" daemon [_thread_in_native, id=257, stack(0x52c00000,0x52c80000)]
  0x10e74c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-103" daemon [_thread_in_native, id=256, stack(0x52d00000,0x52d80000)]
  0x10e7a000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-102" daemon [_thread_in_native, id=255, stack(0x52e00000,0x52e80000)]
  0x10e76c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-101" daemon [_thread_in_native, id=254, stack(0x52f00000,0x52f80000)]
  0x10e7e000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-100" daemon [_thread_in_native, id=253, stack(0x53000000,0x53080000)]
  0x10e7b400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-99" daemon [_thread_in_native, id=252, stack(0x53100000,0x53180000)]
  0x10e77800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-98" daemon [_thread_in_native, id=251, stack(0x53200000,0x53280000)]
  0x10e75800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-97" daemon [_thread_in_native, id=250, stack(0x53300000,0x53380000)]
  0x10b2ac00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-96" daemon [_thread_in_native, id=249, stack(0x53400000,0x53480000)]
  0x10b26400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-95" daemon [_thread_in_native, id=248, stack(0x53500000,0x53580000)]
  0x10a44000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-94" daemon [_thread_in_native, id=247, stack(0x53600000,0x53680000)]
  0x10b20800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-93" daemon [_thread_in_native, id=246, stack(0x53700000,0x53780000)]
  0x10a3a000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-92" daemon [_thread_in_native, id=245, stack(0x53800000,0x53880000)]
  0x10a3bc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-91" daemon [_thread_in_native, id=244, stack(0x53900000,0x53980000)]
  0x10b25400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-90" daemon [_thread_in_native, id=243, stack(0x53a00000,0x53a80000)]
  0x10b22c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-89" daemon [_thread_in_native, id=242, stack(0x53b00000,0x53b80000)]
  0x10a45000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-88" daemon [_thread_in_native, id=241, stack(0x53c00000,0x53c80000)]
  0x10a3fc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-87" daemon [_thread_in_native, id=240, stack(0x53e00000,0x53e80000)]
  0x10a3e400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-86" daemon [_thread_in_native, id=239, stack(0x53f00000,0x53f80000)]
  0x10a3cc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-85" daemon [_thread_in_native, id=238, stack(0x54000000,0x54080000)]
  0x10a38c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-84" daemon [_thread_in_native, id=237, stack(0x54100000,0x54180000)]
  0x10a2f400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-83" daemon [_thread_in_native, id=236, stack(0x54200000,0x54280000)]
  0x10a2bc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-82" daemon [_thread_in_native, id=235, stack(0x54300000,0x54380000)]
  0x10d7a000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-81" daemon [_thread_in_native, id=234, stack(0x54400000,0x54480000)]
  0x10d78800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-80" daemon [_thread_in_native, id=233, stack(0x54500000,0x54580000)]
  0x10d76c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-79" daemon [_thread_in_native, id=232, stack(0x54600000,0x54680000)]
  0x10d75400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-78" daemon [_thread_in_native, id=231, stack(0x54700000,0x54780000)]
  0x10da2800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-77" daemon [_thread_in_native, id=230, stack(0x54800000,0x54880000)]
  0x10d68800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-76" daemon [_thread_in_native, id=229, stack(0x54900000,0x54980000)]
  0x10dd1800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-75" daemon [_thread_in_native, id=228, stack(0x54a00000,0x54a80000)]
  0x10da3c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-74" daemon [_thread_in_native, id=227, stack(0x54b00000,0x54b80000)]
  0x10d98800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-73" daemon [_thread_in_native, id=226, stack(0x54c00000,0x54c80000)]
  0x10e42400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-72" daemon [_thread_in_native, id=225, stack(0x54d00000,0x54d80000)]
  0x10e4e000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-71" daemon [_thread_in_native, id=224, stack(0x54e00000,0x54e80000)]
  0x10dd2000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-70" daemon [_thread_in_native, id=223, stack(0x54f00000,0x54f80000)]
  0x10e3b800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-69" daemon [_thread_in_native, id=222, stack(0x55000000,0x55080000)]
  0x10dd3400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-68" daemon [_thread_in_native, id=221, stack(0x55100000,0x55180000)]
  0x10dd3000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-67" daemon [_thread_in_native, id=220, stack(0x55200000,0x55280000)]
  0x10d70800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-66" daemon [_thread_in_native, id=219, stack(0x55300000,0x55380000)]
  0x10d6d000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-65" daemon [_thread_in_native, id=218, stack(0x55400000,0x55480000)]
  0x10d6b800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-64" daemon [_thread_in_native, id=217, stack(0x55500000,0x55580000)]
  0x10d6a000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-63" daemon [_thread_in_native, id=216, stack(0x55600000,0x55680000)]
  0x10d63400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-62" daemon [_thread_in_native, id=215, stack(0x55700000,0x55780000)]
  0x10d61c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-61" daemon [_thread_in_native, id=214, stack(0x55800000,0x55880000)]
  0x10e50000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-60" daemon [_thread_in_native, id=213, stack(0x55900000,0x55980000)]
  0x10e49400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-59" daemon [_thread_in_native, id=212, stack(0x55a00000,0x55a80000)]
  0x10e47c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-58" daemon [_thread_in_native, id=211, stack(0x55b00000,0x55b80000)]
  0x10e46400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-57" daemon [_thread_in_native, id=210, stack(0x55c00000,0x55c80000)]
  0x10e43c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-56" daemon [_thread_in_native, id=209, stack(0x55d00000,0x55d80000)]
  0x10e3c000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-55" daemon [_thread_in_native, id=208, stack(0x55e00000,0x55e80000)]
  0x10e3c800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-54" daemon [_thread_in_native, id=207, stack(0x55f00000,0x55f80000)]
  0x10e39000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-53" daemon [_thread_in_native, id=206, stack(0x56000000,0x56080000)]
  0x10dad000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-52" daemon [_thread_in_native, id=205, stack(0x56100000,0x56180000)]
  0x10dab800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-51" daemon [_thread_in_native, id=204, stack(0x56200000,0x56280000)]
  0x10da7800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-50" daemon [_thread_in_native, id=203, stack(0x56300000,0x56380000)]
  0x10da5000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-49" daemon [_thread_in_native, id=202, stack(0x56400000,0x56480000)]
  0x10d9bc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-48" daemon [_thread_in_native, id=201, stack(0x56500000,0x56580000)]
  0x10da0800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-47" daemon [_thread_in_native, id=200, stack(0x56600000,0x56680000)]
  0x10d9f000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-46" daemon [_thread_in_native, id=199, stack(0x56700000,0x56780000)]
  0x10d9d400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-45" daemon [_thread_in_native, id=198, stack(0x56800000,0x56880000)]
  0x10d99800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-44" daemon [_thread_in_native, id=197, stack(0x56900000,0x56980000)]
  0x10d95800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-43" daemon [_thread_in_native, id=196, stack(0x56a00000,0x56a80000)]
  0x10a50800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-42" daemon [_thread_in_native, id=195, stack(0x56b00000,0x56b80000)]
  0x10a4f000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-41" daemon [_thread_in_native, id=194, stack(0x56c00000,0x56c80000)]
  0x10a4b000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-40" daemon [_thread_in_native, id=193, stack(0x56d00000,0x56d80000)]
  0x10a49400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-39" daemon [_thread_in_native, id=192, stack(0x56e00000,0x56e80000)]
  0x10a45800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-38" daemon [_thread_in_native, id=191, stack(0x56f00000,0x56f80000)]
  0x10b31c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-37" daemon [_thread_in_native, id=190, stack(0x57000000,0x57080000)]
  0x10b30000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-36" daemon [_thread_in_native, id=189, stack(0x57100000,0x57180000)]
  0x10e84c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-35" daemon [_thread_in_native, id=188, stack(0x57200000,0x57280000)]
  0x10e82c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-34" daemon [_thread_in_native, id=187, stack(0x57300000,0x57380000)]
  0x10dcd000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-33" daemon [_thread_in_native, id=186, stack(0x57400000,0x57480000)]
  0x10dcb400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-32" daemon [_thread_in_native, id=185, stack(0x57500000,0x57580000)]
  0x10dc9c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-31" daemon [_thread_in_native, id=184, stack(0x57600000,0x57680000)]
  0x10dc8400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-30" daemon [_thread_in_native, id=183, stack(0x57700000,0x57780000)]
  0x10dcf800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-29" daemon [_thread_in_native, id=182, stack(0x57800000,0x57880000)]
  0x10e89000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-28" daemon [_thread_in_native, id=181, stack(0x57900000,0x57980000)]
  0x10e87800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-27" daemon [_thread_in_native, id=180, stack(0x57a00000,0x57a80000)]
  0x10e85c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-26" daemon [_thread_in_native, id=179, stack(0x57b00000,0x57b80000)]
  0x10e51400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-25" daemon [_thread_in_native, id=178, stack(0x57c00000,0x57c80000)]
  0x10dfcc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-24" daemon [_thread_in_native, id=177, stack(0x57d00000,0x57d80000)]
  0x10df8c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-23" daemon [_thread_in_native, id=176, stack(0x57e00000,0x57e80000)]
  0x10df7400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-22" daemon [_thread_in_native, id=175, stack(0x57f00000,0x57f80000)]
  0x10df5c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-21" daemon [_thread_in_native, id=174, stack(0x58000000,0x58080000)]
  0x10ddc000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-20" daemon [_thread_in_native, id=173, stack(0x58100000,0x58180000)]
  0x10dd9400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-19" daemon [_thread_in_native, id=172, stack(0x58200000,0x58280000)]
  0x10dd7800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-18" daemon [_thread_in_native, id=171, stack(0x58300000,0x58380000)]
  0x10dd4c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-17" daemon [_thread_in_native, id=170, stack(0x58400000,0x58480000)]
  0x106f4800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-16" daemon [_thread_in_native, id=169, stack(0x58500000,0x58580000)]
  0x106f3000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-15" daemon [_thread_in_native, id=168, stack(0x58600000,0x58680000)]
  0x106f1800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-14" daemon [_thread_in_native, id=167, stack(0x58700000,0x58780000)]
  0x106efc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-13" daemon [_thread_in_native, id=166, stack(0x58800000,0x58880000)]
  0x106ee400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-12" daemon [_thread_in_native, id=165, stack(0x58900000,0x58980000)]
  0x106ecc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-11" daemon [_thread_in_native, id=164, stack(0x58a00000,0x58a80000)]
  0x1070f800 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-10" daemon [_thread_in_native, id=163, stack(0x58b00000,0x58b80000)]
  0x1070e000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-9" daemon [_thread_in_native, id=162, stack(0x58c00000,0x58c80000)]
  0x1072a400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-8" daemon [_thread_in_native, id=161, stack(0x58d00000,0x58d80000)]
  0x10729400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-7" daemon [_thread_in_native, id=160, stack(0x58e00000,0x58e80000)]
  0x10ccbc00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-6" daemon [_thread_in_native, id=159, stack(0x58f00000,0x58f80000)]
  0x10d14400 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-5" daemon [_thread_in_native, id=158, stack(0x59000000,0x59080000)]
  0x10906c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-4" daemon [_thread_in_native, id=157, stack(0x59100000,0x59180000)]
  0x10d28c00 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-3" daemon [_thread_in_native, id=156, stack(0x59200000,0x59280000)]
  0x10a5d000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-2" daemon [_thread_in_native, id=155, stack(0x59300000,0x59380000)]
  0x10a5c000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-1" daemon [_thread_in_native, id=154, stack(0x59400000,0x59480000)]
  0x10a73000 JavaThread "http-padsol49-d0-jb1%2F192.168.41.85-24458-Acceptor-0" daemon [_thread_in_native, id=153, stack(0x59500000,0x59580000)]
  0x10746000 JavaThread "JBossLifeThread" [_thread_blocked, id=152, stack(0x59600000,0x59680000)]
  0x0fd49400 JavaThread "Connection Consumer for dest JBossTopic[OPSC-Gold_Broadcast] id=7" daemon [_thread_blocked, id=150, stack(0x59800000,0x59880000)]
  0x1018e000 JavaThread "WorkManager(2)-7" daemon [_thread_blocked, id=149, stack(0x59900000,0x59980000)]
  0x101e3c00 JavaThread "Connection Consumer for dest JBossTopic[OPSC-Gold_Broadcast] id=6" daemon [_thread_blocked, id=148, stack(0x59a00000,0x59a80000)]
  0x10317400 JavaThread "WorkManager(2)-6" daemon [_thread_blocked, id=147, stack(0x59b00000,0x59b80000)]
  0x0d281c00 JavaThread "Connection Consumer for dest JBossTopic[OPSC-Gold_Broadcast] id=5" daemon [_thread_blocked, id=146, stack(0x59c00000,0x59c80000)]
  0x0ab9c000 JavaThread "WorkManager(2)-5" daemon [_thread_blocked, id=145, stack(0x59d00000,0x59d80000)]
  0x0d281800 JavaThread "Connection Consumer for dest JBossTopic[OPSC-Gold_Broadcast] id=4" daemon [_thread_blocked, id=144, stack(0x59e00000,0x59e80000)]
  0x05bc0400 JavaThread "WorkManager(2)-4" daemon [_thread_blocked, id=143, stack(0x5a400000,0x5a480000)]
  0x0d55d800 JavaThread "Connection Consumer for dest JBossQueue[CbfNotificationQueue] id=3" daemon [_thread_blocked, id=142, stack(0x5a000000,0x5a080000)]
  0x03b88800 JavaThread "WorkManager(2)-3" daemon [_thread_blocked, id=141, stack(0x5a200000,0x5a280000)]
  0x0aa1a800 JavaThread "Connection Consumer for dest JBossQueue[CbfNotificationDLQ] id=2" daemon [_thread_blocked, id=140, stack(0x5a300000,0x5a380000)]
  0x0b60f400 JavaThread "WorkManager(2)-2" daemon [_thread_blocked, id=138, stack(0x5a100000,0x5a180000)]
  0x0d6cd800 JavaThread "Connection Consumer for dest JBossTopic[OPSC-Gold_Broadcast] id=1" daemon [_thread_blocked, id=132, stack(0x5a500000,0x5a580000)]
  0x089d2000 JavaThread "WorkManager(2)-1" daemon [_thread_blocked, id=130, stack(0x5a700000,0x5a780000)]
  0x05a5c000 JavaThread "OOB-2,192.168.41.85:24478" [_thread_blocked, id=129, stack(0x5a600000,0x5a680000)]
  0x055db000 JavaThread "JCA PoolFiller" [_thread_blocked, id=127, stack(0x5a800000,0x5a880000)]
  0x05844c00 JavaThread "IdleRemover" daemon [_thread_blocked, id=126, stack(0x5a900000,0x5a980000)]
  0x02763800 JavaThread "DefaultQuartzScheduler_QuartzSchedulerThread" [_thread_blocked, id=125, stack(0x5aa00000,0x5aa80000)]
  0x006d8000 JavaThread "DefaultQuartzScheduler_Worker-9" [_thread_blocked, id=124, stack(0x5ab00000,0x5ab80000)]
  0x031ae800 JavaThread "DefaultQuartzScheduler_Worker-8" [_thread_blocked, id=123, stack(0x5ac00000,0x5ac80000)]
  0x03b35000 JavaThread "DefaultQuartzScheduler_Worker-7" [_thread_blocked, id=122, stack(0x5ad00000,0x5ad80000)]
  0x03b33c00 JavaThread "DefaultQuartzScheduler_Worker-6" [_thread_blocked, id=121, stack(0x5ae00000,0x5ae80000)]
  0x0598c400 JavaThread "DefaultQuartzScheduler_Worker-5" [_thread_blocked, id=120, stack(0x5af00000,0x5af80000)]
  0x089b6800 JavaThread "DefaultQuartzScheduler_Worker-4" [_thread_blocked, id=119, stack(0x5b000000,0x5b080000)]
  0x0890f000 JavaThread "DefaultQuartzScheduler_Worker-3" [_thread_blocked, id=118, stack(0x5b100000,0x5b180000)]
  0x08b08400 JavaThread "DefaultQuartzScheduler_Worker-2" [_thread_blocked, id=117, stack(0x5b200000,0x5b280000)]
  0x089cd000 JavaThread "DefaultQuartzScheduler_Worker-1" [_thread_blocked, id=116, stack(0x5b300000,0x5b380000)]
  0x08b07c00 JavaThread "DefaultQuartzScheduler_Worker-0" [_thread_blocked, id=115, stack(0x5b400000,0x5b480000)]
  0x057ad000 JavaThread "ContainerBackgroundProcessor[StandardEngine[jboss.web]]" daemon [_thread_blocked, id=114, stack(0x5b500000,0x5b580000)]
  0x03aa7c00 JavaThread "OOB-20,192.168.41.85:24477" [_thread_blocked, id=113, stack(0x5b600000,0x5b680000)]
  0x02ec6c00 JavaThread "OOB-19,192.168.41.85:24477" [_thread_blocked, id=112, stack(0x5b700000,0x5b780000)]
  0x085cbc00 JavaThread "OOB-18,192.168.41.85:24477" [_thread_blocked, id=111, stack(0x5b800000,0x5b880000)]
  0x02de9800 JavaThread "OOB-17,192.168.41.85:24477" [_thread_blocked, id=110, stack(0x5b900000,0x5b980000)]
  0x03e85800 JavaThread "OOB-16,192.168.41.85:24477" [_thread_blocked, id=109, stack(0x5ba00000,0x5ba80000)]
  0x02526000 JavaThread "OOB-15,192.168.41.85:24477" [_thread_blocked, id=108, stack(0x5bb00000,0x5bb80000)]
  0x08dc0c00 JavaThread "OOB-14,192.168.41.85:24477" [_thread_blocked, id=106, stack(0x5bd00000,0x5bd80000)]
  0x00f38400 JavaThread "OOB-13,192.168.41.85:24477" [_thread_blocked, id=105, stack(0x5be00000,0x5be80000)]
  0x01d6e000 JavaThread "Timer-4,192.168.41.85:24478" daemon [_thread_blocked, id=104, stack(0x5bf00000,0x5bf80000)]
  0x0ac86c00 JavaThread "OOB-12,192.168.41.85:24477" [_thread_blocked, id=103, stack(0x5c000000,0x5c080000)]
  0x0b37b400 JavaThread "OOB-11,192.168.41.85:24477" [_thread_blocked, id=102, stack(0x5c100000,0x5c180000)]
  0x01bee800 JavaThread "Timer-3,192.168.41.85:24478" daemon [_thread_blocked, id=101, stack(0x5c200000,0x5c280000)]
  0x013ed000 JavaThread "Timer-2,192.168.41.85:24478" daemon [_thread_blocked, id=100, stack(0x5c300000,0x5c380000)]
  0x05edd000 JavaThread "ConnectionTable.AcceptorThread,192.168.41.85:24478" [_thread_in_native, id=99, stack(0x5c400000,0x5c480000)]
  0x0ad22400 JavaThread "ReceiverThread" daemon [_thread_in_native, id=98, stack(0x5c500000,0x5c580000)]
  0x0a6c7000 JavaThread "FD_SOCK server socket acceptor,null" daemon [_thread_in_native, id=97, stack(0x5c600000,0x5c680000)]
  0x0a6c6400 JavaThread "Timer-1,192.168.41.85:24478" daemon [_thread_blocked, id=96, stack(0x5c700000,0x5c780000)]
  0x092bf400 JavaThread "OOB-10,192.168.41.85:24477" [_thread_blocked, id=94, stack(0x5c800000,0x5c880000)]
  0x042e5c00 JavaThread "OOB-9,192.168.41.85:24477" [_thread_blocked, id=93, stack(0x5c900000,0x5c980000)]
  0x092ac000 JavaThread "FD_SOCK server socket acceptor,192.168.41.85:24477" daemon [_thread_in_native, id=92, stack(0x5ca00000,0x5ca80000)]
  0x0ac7b800 JavaThread "pool-8-thread-1" [_thread_blocked, id=91, stack(0x5cb00000,0x5cb80000)]
  0x09ac0400 JavaThread "Thread-13" daemon [_thread_blocked, id=89, stack(0x5cc00000,0x5cc80000)]
  0x0a19d800 JavaThread "PooledInvokerAcceptor#0-24472" [_thread_in_native, id=88, stack(0x5cd00000,0x5cd80000)]
  0x02b12400 JavaThread "PooledInvokerAcceptor#0-24470" [_thread_in_native, id=87, stack(0x5ce00000,0x5ce80000)]
  0x09590000 JavaThread "RMI TCP Accept-24479" daemon [_thread_in_native, id=86, stack(0x5cf00000,0x5cf80000)]
  0x09a7a400 JavaThread "RMI TCP Accept-24469" daemon [_thread_in_native, id=85, stack(0x5d000000,0x5d080000)]
  0x0ab04c00 JavaThread "Thread-12" [_thread_blocked, id=84, stack(0x5d100000,0x5d180000)]
  0x09f7cc00 JavaThread "Listener:24475" daemon [_thread_in_native, id=83, stack(0x5d200000,0x5d280000)]
  0x0aae8c00 JavaThread "Thread-11" daemon [_thread_blocked, id=80, stack(0x5d300000,0x5d380000)]
  0x09b8c800 JavaThread "Thread-10" daemon [_thread_blocked, id=79, stack(0x5d400000,0x5d480000)]
  0x09b8c000 JavaThread "Thread-9" daemon [_thread_blocked, id=78, stack(0x5d500000,0x5d580000)]
  0x0aaf1000 JavaThread "Listener:24476" daemon [_thread_in_native, id=77, stack(0x5d600000,0x5d680000)]
  0x05091400 JavaThread "AcceptorThread[ServerSocket[addr=padsol49-d0-jb1/192.168.41.85,localport=24473]]" [_thread_in_native, id=76, stack(0x5d700000,0x5d780000)]
  0x0511b400 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=75, stack(0x5d800000,0x5d880000)]
  0x06e1e000 JavaThread "JBoss System Threads(1)-7" daemon [_thread_in_native, id=74, stack(0x5d900000,0x5d980000)]
  0x05473000 JavaThread "JBoss System Threads(1)-6" daemon [_thread_in_native, id=73, stack(0x5da00000,0x5da80000)]
  0x09170800 JavaThread "RMI TCP Accept-24462" daemon [_thread_in_native, id=72, stack(0x5db00000,0x5db80000)]
  0x0432cc00 JavaThread "AsynchViewChangeHandler Thread" [_thread_blocked, id=71, stack(0x5dc00000,0x5dc80000)]
  0x0432c400 JavaThread "AsynchKeyChangeHandler Thread" [_thread_blocked, id=70, stack(0x5dd00000,0x5dd80000)]
  0x042b9c00 JavaThread "Timer-12,192.168.41.85:24477" daemon [_thread_blocked, id=69, stack(0x5de00000,0x5de80000)]
  0x042b9400 JavaThread "Timer-11,192.168.41.85:24477" daemon [_thread_blocked, id=68, stack(0x5df00000,0x5df80000)]
  0x05518400 JavaThread "Timer-10,192.168.41.85:24477" daemon [_thread_blocked, id=67, stack(0x5e000000,0x5e080000)]
  0x05518000 JavaThread "Timer-9,192.168.41.85:24477" daemon [_thread_blocked, id=66, stack(0x5e100000,0x5e180000)]
  0x09aa1400 JavaThread "Timer-8,192.168.41.85:24477" daemon [_thread_blocked, id=65, stack(0x5e200000,0x5e280000)]
  0x09cd1000 JavaThread "OOB-8,192.168.41.85:24477" [_thread_blocked, id=64, stack(0x5e300000,0x5e380000)]
  0x09aa0c00 JavaThread "OOB-7,192.168.41.85:24477" [_thread_blocked, id=63, stack(0x5e400000,0x5e480000)]
  0x09aa0800 JavaThread "OOB-6,192.168.41.85:24477" [_thread_blocked, id=62, stack(0x5e500000,0x5e580000)]
  0x04ff6400 JavaThread "Timer-7,192.168.41.85:24477" daemon [_thread_blocked, id=61, stack(0x5e600000,0x5e680000)]
  0x031d8400 JavaThread "OOB-5,192.168.41.85:24477" [_thread_blocked, id=60, stack(0x5e700000,0x5e780000)]
  0x09ac1400 JavaThread "OOB-4,192.168.41.85:24477" [_thread_blocked, id=59, stack(0x5e800000,0x5e880000)]
  0x01b1fc00 JavaThread "Timer-6,192.168.41.85:24477" daemon [_thread_blocked, id=58, stack(0x5e900000,0x5e980000)]
  0x05398800 JavaThread "OOB-3,192.168.41.85:24477" [_thread_blocked, id=57, stack(0x5ea00000,0x5ea80000)]
  0x0122e800 JavaThread "Timer-5,192.168.41.85:24477" daemon [_thread_blocked, id=56, stack(0x5eb00000,0x5eb80000)]
  0x03a54800 JavaThread "FD_SOCK server socket acceptor,192.168.41.85:24477" daemon [_thread_in_native, id=55, stack(0x5ec00000,0x5ec80000)]
  0x09b0b400 JavaThread "Timer-4,192.168.41.85:24477" daemon [_thread_blocked, id=54, stack(0x5ed00000,0x5ed80000)]
  0x092a6000 JavaThread "OOB-2,192.168.41.85:24477" [_thread_blocked, id=52, stack(0x5ee00000,0x5ee80000)]
  0x00db1800 JavaThread "Timer-3,192.168.41.85:24477" daemon [_thread_blocked, id=51, stack(0x5ef00000,0x5ef80000)]
  0x094b3800 JavaThread "OOB-1,192.168.41.85:24477" [_thread_blocked, id=50, stack(0x5f000000,0x5f080000)]
  0x00c58400 JavaThread "Timer-2,192.168.41.85:24477" daemon [_thread_blocked, id=49, stack(0x5f100000,0x5f180000)]
  0x08628400 JavaThread "UDP mcast,192.168.41.85:24477" [_thread_in_native, id=48, stack(0x5f200000,0x5f280000)]
  0x09a24c00 JavaThread "UDP ucast,192.168.41.85:24477" [_thread_in_native, id=47, stack(0x5f300000,0x5f380000)]
  0x094b4000 JavaThread "FD_SOCK server socket acceptor,null" daemon [_thread_in_native, id=46, stack(0x5f400000,0x5f480000)]
  0x09164800 JavaThread "Timer-1,192.168.41.85:24477" daemon [_thread_blocked, id=45, stack(0x5f500000,0x5f580000)]
  0x09209800 JavaThread "JBoss System Threads(1)-5" daemon [_thread_blocked, id=44, stack(0x5f600000,0x5f680000)]
  0x08cde000 JavaThread "AcceptorThread[ServerSocket[addr=/192.168.41.85,localport=24474]]" [_thread_in_native, id=42, stack(0x5f700000,0x5f780000)]
  0x08cdd800 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=41, stack(0x5f800000,0x5f880000)]
  0x02625800 JavaThread "secondaryServerSocketThread[0]" daemon [_thread_in_native, id=40, stack(0x5f900000,0x5f980000)]
  0x04342800 JavaThread "AcceptorThread[ServerSocket[addr=padsol49-d0-jb1/192.168.41.85,localport=24468]]" [_thread_in_native, id=39, stack(0x5fa00000,0x5fa80000)]
  0x03fc8400 JavaThread "ServerSocketRefresh" daemon [_thread_blocked, id=38, stack(0x5fb00000,0x5fb80000)]
  0x03b94000 JavaThread "JBoss System Threads(1)-4" daemon [_thread_blocked, id=37, stack(0x5fc00000,0x5fc80000)]
  0x01062c00 JavaThread "RMI Scheduler(0)" daemon [_thread_blocked, id=36, stack(0x5fd00000,0x5fd80000)]
  0x0177e800 JavaThread "JBoss System Threads(1)-3" daemon [_thread_blocked, id=34, stack(0x5ff00000,0x5ff80000)]
  0x0223ec00 JavaThread "JBoss System Threads(1)-2" daemon [_thread_in_native, id=33, stack(0x60000000,0x60080000)]
  0x03a37c00 JavaThread "GC Daemon" daemon [_thread_blocked, id=32, stack(0x60100000,0x60180000)]
  0x03b47800 JavaThread "RMI Reaper" [_thread_blocked, id=31, stack(0x60200000,0x60280000)]
  0x03b7e400 JavaThread "RMI TCP Accept-24461" daemon [_thread_in_native, id=30, stack(0x60300000,0x60380000)]
  0x038c3400 JavaThread "JBoss System Threads(1)-1" daemon [_thread_in_native, id=29, stack(0x60400000,0x60480000)]
  0x03927400 JavaThread "Timer-Log4jService" daemon [_thread_blocked, id=28, stack(0x60500000,0x60580000)]
  0x01d7fc00 JavaThread "AuthenticationCacheFlushThread" [_thread_blocked, id=27, stack(0x60680000,0x60700000)]
  0x00c77000 JavaThread "Timer-0" daemon [_thread_blocked, id=25, stack(0x60880000,0x60900000)]
  0x00490c00 JavaThread "Service Thread" daemon [_thread_blocked, id=22, stack(0x60b80000,0x60c00000)]
  0x0048f000 JavaThread "C2 CompilerThread1" daemon [_thread_blocked, id=21, stack(0x60c80000,0x60d00000)]
  0x0048cc00 JavaThread "C2 CompilerThread0" daemon [_thread_blocked, id=20, stack(0x60d80000,0x60e00000)]
  0x0048b400 JavaThread "Signal Dispatcher" daemon [_thread_blocked, id=19, stack(0x60e80000,0x60f00000)]
  0x0044dc00 JavaThread "Finalizer" daemon [_thread_blocked, id=18, stack(0x61200000,0x61280000)]
  0x00448c00 JavaThread "Reference Handler" daemon [_thread_blocked, id=17, stack(0x61300000,0x61380000)]

Other Threads:
  0x00443000 VMThread [stack: 0x62b00000,0x62b80000] [id=16]
  0x0049b000 WatcherThread [stack: 0x60a80000,0x60b00000] [id=23]

VM state:not at safepoint (normal execution)

VM Mutex/Monitor currently owned by a thread:  ([mutex/lock_event])
[0x00026bb8] SystemDictionary_lock - owner thread: 0x12b21000

Heap
 PSYoungGen      total 678592K, used 602316K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656768K, 89% used [0xd0c00000,0xf47b8980,0xf8d60000)
  from space 21824K, 77% used [0xf8d60000,0xf9dda900,0xfa2b0000)
  to   space 21760K, 0% used [0xfa2c0000,0xfa2c0000,0xfb800000)
 ParOldGen       total 1400832K, used 903570K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 64% used [0x7b400000,0xb2664988,0xd0c00000)
 PSPermGen       total 94208K, used 94127K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7ebcf8,0x6e800000)

Card table byte_map: [0x68400000,0x68898000] byte_map_base: 0x680ba000

Polling page: 0xff3f6000

Code Cache  [0xfbc00000, 0xfd608000, 0xfdc00000)
 total_blobs=8037 nmethods=7768 adapters=179 free_code_cache=6259Kb largest_free_block=6264384

Compilation events (10 events):
Event: 278.213 Thread 0x0048cc00 8378             com.sun.org.apache.xerces.internal.impl.dtd.XMLDTDValidator::setProperty (1 bytes)
Event: 278.221 Thread 0x0048cc00 nmethod 8378 0xfd5f6c08 code [0xfd5f6ce0, 0xfd5f6d3c]
Event: 278.238 Thread 0x0048f000 8379   !         sun.reflect.GeneratedConstructorAccessor342::newInstance (49 bytes)
Event: 278.242 Thread 0x0048f000 nmethod 8379 0xfd5fe248 code [0xfd5fe340, 0xfd5fe43c]
Event: 278.312 Thread 0x0048cc00 8380             com.orga.opsc.gold.bas.service.businessop.internal.io.EnsureSoldComponentTypeIsOKInput::getProductId (5 bytes)
Event: 278.313 Thread 0x0048cc00 nmethod 8380 0xfd5fe0c8 code [0xfd5fe1a0, 0xfd5fe1fc]
Event: 278.340 Thread 0x0048f000 8381             com.orga.opsc.gold.bas.persistence.AbstractAccount::getRteInstance (5 bytes)
Event: 278.342 Thread 0x0048f000 nmethod 8381 0xfd5fdf48 code [0xfd5fe020, 0xfd5fe07c]
Event: 278.378 Thread 0x0048cc00 8382             java.io.ObjectOutputStream$HandleTable::clear (27 bytes)
Event: 278.383 Thread 0x0048cc00 nmethod 8382 0xfd5f81c8 code [0xfd5f82c0, 0xfd5f849c]

GC Heap History (10 events):
Event: 273.954 GC heap before
{Heap before GC invocations=70 (full 1):
 PSYoungGen      total 678592K, used 669949K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656192K, 100% used [0xd0c00000,0xf8cd0000,0xf8cd0000)
  from space 22400K, 61% used [0xfa220000,0xfaf8f790,0xfb800000)
  to   space 21824K, 0% used [0xf8cd0000,0xf8cd0000,0xfa220000)
 ParOldGen       total 1400832K, used 880457K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 62% used [0x7b400000,0xb0fd27b0,0xd0c00000)
 PSPermGen       total 94208K, used 94074K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7deac0,0x6e800000)
Event: 274.006 GC heap after
Heap after GC invocations=70 (full 1):
 PSYoungGen      total 678016K, used 17126K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656192K, 0% used [0xd0c00000,0xd0c00000,0xf8cd0000)
  from space 21824K, 78% used [0xf8cd0000,0xf9d89900,0xfa220000)
  to   space 22016K, 0% used [0xfa280000,0xfa280000,0xfb800000)
 ParOldGen       total 1400832K, used 884598K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb13ddbc0,0xd0c00000)
 PSPermGen       total 94208K, used 94074K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7deac0,0x6e800000)
}
Event: 275.250 GC heap before
{Heap before GC invocations=71 (full 1):
 PSYoungGen      total 678016K, used 673318K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656192K, 100% used [0xd0c00000,0xf8cd0000,0xf8cd0000)
  from space 21824K, 78% used [0xf8cd0000,0xf9d89900,0xfa220000)
  to   space 22016K, 0% used [0xfa280000,0xfa280000,0xfb800000)
 ParOldGen       total 1400832K, used 884601K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb13de670,0xd0c00000)
 PSPermGen       total 94208K, used 94084K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e1340,0x6e800000)
Event: 275.293 GC heap after
Heap after GC invocations=71 (full 1):
 PSYoungGen      total 678912K, used 13900K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656896K, 0% used [0xd0c00000,0xd0c00000,0xf8d80000)
  from space 22016K, 63% used [0xfa280000,0xfb013330,0xfb800000)
  to   space 21504K, 0% used [0xf8d80000,0xf8d80000,0xfa280000)
 ParOldGen       total 1400832K, used 888921K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb1816580,0xd0c00000)
 PSPermGen       total 94208K, used 94084K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e1340,0x6e800000)
}
Event: 276.069 GC heap before
{Heap before GC invocations=72 (full 1):
 PSYoungGen      total 678912K, used 670796K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656896K, 100% used [0xd0c00000,0xf8d80000,0xf8d80000)
  from space 22016K, 63% used [0xfa280000,0xfb013330,0xfb800000)
  to   space 21504K, 0% used [0xf8d80000,0xf8d80000,0xfa280000)
 ParOldGen       total 1400832K, used 889075K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb183cf48,0xd0c00000)
 PSPermGen       total 94208K, used 94092K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e3318,0x6e800000)
Event: 276.108 GC heap after
Heap after GC invocations=72 (full 1):
 PSYoungGen      total 678400K, used 17528K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656896K, 0% used [0xd0c00000,0xd0c00000,0xf8d80000)
  from space 21504K, 81% used [0xf8d80000,0xf9e9e3e0,0xfa280000)
  to   space 21824K, 0% used [0xfa2b0000,0xfa2b0000,0xfb800000)
 ParOldGen       total 1400832K, used 893588K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb1ca5358,0xd0c00000)
 PSPermGen       total 94208K, used 94092K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e3318,0x6e800000)
}
Event: 276.859 GC heap before
{Heap before GC invocations=73 (full 1):
 PSYoungGen      total 678400K, used 674424K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656896K, 100% used [0xd0c00000,0xf8d80000,0xf8d80000)
  from space 21504K, 81% used [0xf8d80000,0xf9e9e3e0,0xfa280000)
  to   space 21824K, 0% used [0xfa2b0000,0xfa2b0000,0xfb800000)
 ParOldGen       total 1400832K, used 893592K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 63% used [0x7b400000,0xb1ca6250,0xd0c00000)
 PSPermGen       total 94208K, used 94105K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e6640,0x6e800000)
Event: 276.900 GC heap after
Heap after GC invocations=73 (full 1):
 PSYoungGen      total 678592K, used 17060K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656768K, 0% used [0xd0c00000,0xd0c00000,0xf8d60000)
  from space 21824K, 78% used [0xfa2b0000,0xfb3590c8,0xfb800000)
  to   space 21824K, 0% used [0xf8d60000,0xf8d60000,0xfa2b0000)
 ParOldGen       total 1400832K, used 900045K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 64% used [0x7b400000,0xb22f36b0,0xd0c00000)
 PSPermGen       total 94208K, used 94105K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e6640,0x6e800000)
}
Event: 277.691 GC heap before
{Heap before GC invocations=74 (full 1):
 PSYoungGen      total 678592K, used 673828K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656768K, 100% used [0xd0c00000,0xf8d60000,0xf8d60000)
  from space 21824K, 78% used [0xfa2b0000,0xfb3590c8,0xfb800000)
  to   space 21824K, 0% used [0xf8d60000,0xf8d60000,0xfa2b0000)
 ParOldGen       total 1400832K, used 900045K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 64% used [0x7b400000,0xb22f36b0,0xd0c00000)
 PSPermGen       total 94208K, used 94119K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e9de8,0x6e800000)
Event: 277.733 GC heap after
Heap after GC invocations=74 (full 1):
 PSYoungGen      total 678592K, used 16874K [0xd0c00000, 0xfb800000, 0xfb800000)
  eden space 656768K, 0% used [0xd0c00000,0xd0c00000,0xf8d60000)
  from space 21824K, 77% used [0xf8d60000,0xf9dda900,0xfa2b0000)
  to   space 21760K, 0% used [0xfa2c0000,0xfa2c0000,0xfb800000)
 ParOldGen       total 1400832K, used 903570K [0x7b400000, 0xd0c00000, 0xd0c00000)
  object space 1400832K, 64% used [0x7b400000,0xb2664988,0xd0c00000)
 PSPermGen       total 94208K, used 94119K [0x68c00000, 0x6e800000, 0x7b400000)
  object space 94208K, 99% used [0x68c00000,0x6e7e9de8,0x6e800000)
}

Deoptimization events (10 events):
Event: 261.595 Thread 0x126fe400 Uncommon trap -12 fr.pc 0xfd27e23c
Event: 261.607 Thread 0x121da000 Uncommon trap -83 fr.pc 0xfd2c6b38
Event: 278.427 Thread 0x12b21000 Uncommon trap -83 fr.pc 0xfd414544
Event: 278.427 Thread 0x12b21000 Uncommon trap -34 fr.pc 0xfc5bf088
Event: 278.427 Thread 0x130cec00 Uncommon trap -83 fr.pc 0xfd414544
Event: 278.427 Thread 0x12b21000 Uncommon trap 162 fr.pc 0xfce69580
Event: 278.428 Thread 0x130cec00 Uncommon trap -34 fr.pc 0xfc5bf088
Event: 278.428 Thread 0x130cec00 Uncommon trap 162 fr.pc 0xfce69580
Event: 278.429 Thread 0x12b21000 Uncommon trap -34 fr.pc 0xfcb51dc8
Event: 278.438 Thread 0x130cec00 Uncommon trap -34 fr.pc 0xfcb51dc8

Internal exceptions (10 events):
Event: 277.087 Thread 0x10ef3c00 Threw 0xda7274c0 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.099 Thread 0x121da000 Threw 0xdaa74d10 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.099 Thread 0x121da000 Threw 0xdaa7cad8 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.100 Thread 0x121da000 Threw 0xdaa848a0 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.100 Thread 0x121da000 Threw 0xdaa8c668 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.100 Thread 0x121da000 Threw 0xdaa94430 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.100 Thread 0x121da000 Threw 0xdaa9c1f8 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 277.100 Thread 0x121da000 Threw 0xdb170bb0 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/interpreter/interpr
Event: 278.425 Thread 0x12b21000 Threw 0xf2784030 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/prims/jni.cpp:742
Event: 278.425 Thread 0x130cec00 Threw 0xf2ce1030 at /export/HUDSON/workspace/jdk7u25-2-build-solaris-sparc-product/jdk7u25/hotspot/src/share/vm/prims/jni.cpp:742

Events (10 events):
Event: 278.428 Thread 0x130cec00 DEOPT PACKING pc=0xfc5bf088 sp=0x4e87ee10
Event: 278.428 Thread 0x130cec00 DEOPT UNPACKING pc=0xfbc32524 sp=0x4e87ed70 mode 2
Event: 278.428 Thread 0x130cec00 DEOPT PACKING pc=0xfce69580 sp=0x4e87ef08
Event: 278.428 Thread 0x130cec00 DEOPT UNPACKING pc=0xfbc32524 sp=0x4e87ee48 mode 2
Event: 278.429 Thread 0x12b21000 DEOPT PACKING pc=0xfcb51dc8 sp=0x4f17e8b0
Event: 278.429 Thread 0x12b21000 DEOPT UNPACKING pc=0xfbc32524 sp=0x4f17e830 mode 2
Event: 278.429 Executing VM operation: RevokeBias
Event: 278.435 Executing VM operation: RevokeBias done
Event: 278.438 Thread 0x130cec00 DEOPT PACKING pc=0xfcb51dc8 sp=0x4e87ea30
Event: 278.438 Thread 0x130cec00 DEOPT UNPACKING pc=0xfbc32524 sp=0x4e87e9b0 mode 2


Dynamic libraries:
0x00010000 /nfs/local/java/jdk1.7.0_25/bin/java
0xff370000 /lib/libthread.so.1
0xff330000 /nfs/local/java/jdk1.7.0_25/bin/../jre/lib/sparc/jli/libjli.so
0xff310000 /lib/libdl.so.1
0xff180000 /lib/libc.so.1
0xff350000 /platform/SUNW,SPARC-Enterprise/lib/libc_psr.so.1
0xfe000000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/server/libjvm.so
0xff2d0000 /lib/libsocket.so.1
0xff2f0000 /usr/lib/libsched.so.1
0xff160000 /lib/libm.so.1
0xff120000 /usr/lib/libCrun.so.1
0xff100000 /lib/libdoor.so.1
0xff0c0000 /usr/lib/libdemangle.so.1
0xff0a0000 /lib/libkstat.so.1
0xfef80000 /lib/libnsl.so.1
0xfee80000 /lib/libm.so.2
0xff070000 /lib/libscf.so.1
0xff040000 /lib/libuutil.so.1
0xfef60000 /lib/libgen.so.1
0xfef30000 /lib/libmd.so.1
0xfee50000 /lib/libmp.so.2
0xfedd0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libverify.so
0xfed90000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libjava.so
0xfdfb0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libzip.so
0x60f80000 /usr/lib/locale/en_US.UTF-8/en_US.UTF-8.so.3
0xfde90000 /usr/lib/locale/en_US.UTF-8/methods_en_US.UTF-8.so.3
0xfbbd0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libnio.so
0xfbbb0000 /lib/librt.so.1
0xfb8e0000 /lib/libsendfile.so.1
0xfb8b0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libnet.so
0xfb890000 /lib/libaio.so.1
0xfb870000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libj2ucrypto.so
0xfb850000 /usr/lib/libsoftcrypto.so
0xfb830000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libmanagement.so
0x613d0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libj2pkcs11.so
0x61390000 /usr/lib/libpkcs11.so
0x612e0000 /usr/lib/libcryptoutil.so.1
0x60f10000 /usr/lib/security/pkcs11_softtoken_extra.so
0x611c0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/librmi.so
0x67eb0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libsunec.so
0x59f00000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/libawt.so
0x681e0000 /nfs/local/java/jdk1.7.0_25/jre/lib/sparc/headless/libmawt.so

VM Arguments:
jvm_args: -Xms2048m -Xmx2048m -Dsun.rmi.dgc.client.gcInterval=3600000 -Dsun.rmi.dgc.server.gcInterval=3600000 -Dsun.lang.ClassLoader.allowArraySyntax=true -Djgroups.bind_addr=padsol49-d0-jb1 -Doracle.jdbc.Trace=false -Doracle.jdbc.TcpNoDelay=false -Djboss.messaging.ServerPeerID=1 -Djava.net.preferIPv4Stack=true -Djava.awt.headless=true -XX:-UseVMInterruptibleIO -verbose:gc -Xloggc:/data2/SFleege-Althoff/orgadata_system_sfleege_d25/jboss1/server/opsc-gold/log/gc_20131007170053.log -XX:+PrintGCDetails -XX:+PrintTenuringDistribution -XX:+PrintGCApplicationConcurrentTime -XX:+PrintGCApplicationStoppedTime -XX:+PrintGCTimeStamps -XX:+PrintGCDateStamps -XX:-TraceClassUnloading -XX:+HeapDumpOnOutOfMemoryError -XX:MaxPermSize=192m -Duser.timezone=Europe/Berlin -XX:MaxPermSize=296m -Djava.endorsed.dirs=/data2/SFleege-Althoff/orgadata_system_sfleege_d25/jboss1/lib/endorsed
java_command: org.jboss.Main -b padsol49-d0-jb1 -c opsc-gold -g test_cbf_sflee_d25_Sun -u 225.142.181.50 -m 24484
Launcher Type: SUN_STANDARD

Environment Variables:
JAVA_HOME=/usr/local/java/jdk1.7.0_25
PATH=/usr/local/bash-4.2/bin:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/openldap/bin:/usr/local/java/jdk1.7.0_25/bin:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/run/bin:/data2/SFleege-Althoff/ws/OPSC_Gold_system_dev_OZ75355_Sun/Component/ant/bin:/usr/local/gcc-4.6.3/bin:/usr/local/bin:/usr/ccs/bin:/usr/openwin/bin:/usr/bin/X11:/usr/X11/bin:/bin:/pcs/bin:/usr/bin:/data2/SFleege-Althoff/ws/OPSC_Gold_system_dev_OZ75355_Sun/goldcommon/tools:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/test/bin:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/test/scripts:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/test/perf:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/test/scenarios:/usr/ucb:/oracle/u00/oracle/product/11g/bin:/sbin:/usr/X11R6/bin:/nfs/uh12/sfleege/bin:/usr/local:.:/nfs/uh03/cv/kcell:/nfs/install/software/ssc/tools/bin:.:/nfs/uh12/sfleege/bin:/pcs/bin:/bin:/pcs/bin:/usr/ucb:/usr/bin:/usr/ccs/bin:/usr/xmotif/bin:/usr/openwin/bin:/usr/local/bin:/usr/local/gnu/bin:/usr/local/X11/bin:/nfs/uh03/cv/kcell:/nfs/install/software/ssc/tools/bin
LD_LIBRARY_PATH=/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/openldap/lib:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/db/lib:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/run/lib:/usr/local/java/jdk1.7.0_25/lib:/usr/local/gcc-4.6.3/lib:/usr/openwin/lib:/usr/dt/lib::/usr/local/lib:/usr/local/lib/sparcv9:/lib/sparcv9:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/test/lib:/oracle/u00/oracle/product/11g/lib:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/subversion/lib:/data2/SFleege-Althoff/bd/OPSC_Gold_system_dev_OZ75355_Sun/OPSCGOLD/apache2/lib
SHELL=/bin/bash
DISPLAY=padpssuse01:1002
OSTYPE=solaris
ARCH=64

Signal Handlers:
SIGSEGV: [libjvm.so+0xabbdfc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGBUS: [libjvm.so+0xabbdfc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGFPE: [libjvm.so+0x21b4fc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGPIPE: [libjvm.so+0x21b4fc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGXFSZ: [libjvm.so+0x21b4fc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGILL: [libjvm.so+0x21b4fc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c
SIGUSR1: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGUSR2: SIG_DFL, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGQUIT: [libjvm.so+0x94ffe8], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIGHUP: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGINT: SIG_IGN, sa_mask[0]=0x00000000, sa_flags=0x00000000
SIGTERM: [libjvm.so+0x94ffe8], sa_mask[0]=0xffbffeff, sa_flags=0x00000004
SIG39: [libjvm.so+0x954150], sa_mask[0]=0x00000000, sa_flags=0x00000008
SIG40: [libjvm.so+0x21b4fc], sa_mask[0]=0xffbffeff, sa_flags=0x0000000c


---------------  S Y S T E M  ---------------

OS:                   Oracle Solaris 10 9/10 s10s_u9wos_14a SPARC
     Copyright (c) 2010, Oracle and/or its affiliates. All rights reserved.
                            Assembled 11 August 2010

                  Solaris 10 8/11 (Update 10) Patchset applied.

uname:SunOS 5.10 Generic_147440-01 sun4u
  (T2 libthread)
rlimit: STACK 8192k, CORE infinity, NOFILE 65536, AS infinity
load average:10.47 4.98 5.89

CPU:total 32 v9, popc, vis1, vis2, sparc64

Memory: 8k page, physical 33554432k(5412336k free)

vm_info: Java HotSpot(TM) Server VM (23.25-b01) for solaris-sparc JRE (1.7.0_25-b15), built on Jun  5 2013 23:56:12 by "" with Sun Studio 12u1

time: Mon Oct  7 17:05:33 2013
elapsed time: 279 seconds



REPRODUCIBILITY :
This bug can be reproduced always.

CUSTOMER SUBMITTED WORKAROUND :
Switching from ParallelGC to CMS circumvents the issue.
Comments
See comment from submitter in previous comment.
16-12-2013
Comment from submitter. This issue will be closed as not reproducible. ====================================== Hi Jon, I managed to have the test repeated. Unfortunately I have to report that the issue is not reproducible anymore. From my point of view this does not make sense, since we did not change the command line params in the meantime. Since the problem does not really hinder us (we do not use ParallelGC in production anyhow), I don't have much leverage to push this topic harder for reproduction. I hate to say it, but I will have to ask you to close the ticket as INVALID (or similar) since I will not be able to have it reproduced. ... bye, Michael
16-12-2013
Submitter: mhess@orga-systems.com
12-11-2013
In 6u27, -XX:+UseParallelGC gives you ParallelScavenge + SerialOld. In 7u25 (and 7u40), -XX:+UseParallelGC gives you ParallelScavenge + ParallelOld. ParallelOld uses more malloc'd memory various data structures. On Solaris, the malloc does not use mmap for large allocations, and since we place the Java heap rather low in the virtual memory address space to be able to use compressed oops, we do not have a lot of memory that we can malloc. If the above analysis is correct, then the customer should be able to reproduce the failure with 6u27 by using the flags: -XX:+UseParallelGC -XX:+UseParallelOldGC. A workaround is to increase -XX:HeapBaseMinAddress (they can check the current value with -XX:+PrintFlagsFinal), but that might affect the kind of compressed oops they get, which might affect performance. Please ask the customer to run with -XX:+UseParallelGC -XX:+UseParallelOldGC, then we can discuss workarounds.
29-10-2013
We don't really have enough information to go on here, but moving over to JDK since this was a regression between JDK6 and JDK7. It is possible that this is just caused by the permgen removal. More data has been moved outside the Java heap, so some increase in native memory usage is expected. One thing for the filer to test could be to just lower the Java heap with 300MB or something. To the dev team: 1) Is this a known issue, or do you have an idea what the problem is? 2) If not, please advice the filer what additional information we would need to debug this? Any logs we could request Feel free to resolve the bug as incomplete after that, I don't expect that we can make any fixes from just this information.
29-10-2013