FULL PRODUCT VERSION :
java version  " 1.6.0_38 " 
Java(TM) SE Runtime Environment (build 1.6.0_38-b05)
Java HotSpot(TM) 64-Bit Server VM (build 20.13-b02, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux magnifico.aimsv.com 2.6.18-308.24.1.el5xen #1 SMP Tue Dec 4 18:33:26 EST 2012 x86_64 x86_64 x86_64 GNU/Linux


EXTRA RELEVANT SYSTEM CONFIGURATION :
Our application runs in JBOSS-6.1 server and one of its components uses JBOSS MQ (i know that MQ is deprecated but this component is legacy that we need to carry on for some more time).

Several messages get transferred between various components of the system via JMS (asynchronous communication), perhaps tens of messages per second at a peak load.







A DESCRIPTION OF THE PROBLEM :
We have been running this application for several years without any known issues. We have regularly upgrading our JDK/JRE version at regular intervals. We tested our application with JRE-1.6.33 and did not face this deadlock issue.

However, after upgrading to JRE-1.6.38 and running our application for about a week (roughly 5-7 days), we are certainly seeing deadlocks appearing in JDK core classes like SSLScoketImpl.java.

On searching on the internet for this particular  " deadlock in SSLSocketImpl.readDataRecord() or SSLSocketImpl.writeRecord() " , we found several references to this exact same issue in various open source  products, and they claim that this issue is in core JRE and that this issue is still present in JRE-1.7 as well, and is not yet fixed in any release of JRE.

Here is one reference to the above claim:
https://jira.atlassian.com/browse/BAM-12699

This thread from Oracle Forums claims that this issue is still present in JRE-1.7.x
https://forums.oracle.com/forums/thread.jspa?messageID=10748121

We found that this issue seems to have been reported on JRE-1.4 and/or JRE-1.5 and was fixed (http://bugs.sun.com/view_bug.do?bug_id=6569529)
but looks like it reappeared in JDK-1.6.38+ or JDK-1.7.x

If we continue to run the test even after seeing the deadlocks, the number of deadlocks are increasing at a rate of 5 per week


Here is the extract from thread-dump
----------------------------------------------------------------------
 " UIL2.SocketManager.WriteTask#204296 client=10.64.2.123:8193 " :
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.getConnectionState(SSLSocketImpl.java:601)
        - waiting to lock <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.isClosed(SSLSocketImpl.java:1328)
        at java.net.Socket.getTcpNoDelay(Socket.java:877)
        at com.sun.net.ssl.internal.ssl.BaseSSLSocketImpl.getTcpNoDelay(BaseSSLSocketImpl.java:325)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecordInternal(SSLSocketImpl.java:770)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:753)
        at com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:100)
        - locked <0x000000076ab43a18> (a com.sun.net.ssl.internal.ssl.AppOutputStream)
        at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:65)
        at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:123)
        - locked <0x000000076ab5afb0> (a org.jboss.util.stream.NotifyingBufferedOutputStream)
        at java.io.ObjectOutputStream$BlockDataOutputStream.flush(ObjectOutputStream.java:1793)
        at java.io.ObjectOutputStream.flush(ObjectOutputStream.java:699)
        at org.jboss.mq.il.uil2.SocketManager$WriteTask.run(SocketManager.java:558)
        at java.lang.Thread.run(Thread.java:662)
 " UIL2.SocketManager.ReadTask#204295 client=10.64.2.123:8193 " :
        at sun.misc.Unsafe.park(Native Method)
        - parking to wait for  <0x000000076ab40898> (a java.util.concurrent.locks.ReentrantLock$NonfairSync)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireQueued(AbstractQueuedSynchronizer.java:842)
        at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquire(AbstractQueuedSynchronizer.java:1178)
        at java.util.concurrent.locks.ReentrantLock$NonfairSync.lock(ReentrantLock.java:186)
        at java.util.concurrent.locks.ReentrantLock.lock(ReentrantLock.java:262)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:751)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.writeRecord(SSLSocketImpl.java:624)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.sendAlert(SSLSocketImpl.java:1875)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.warning(SSLSocketImpl.java:1713)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.closeInternal(SSLSocketImpl.java:1481)
        - locked <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.recvAlert(SSLSocketImpl.java:1819)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readRecord(SSLSocketImpl.java:1019)
        - locked <0x000000076ab4ca18> (a com.sun.net.ssl.internal.ssl.SSLSocketImpl)
        - locked <0x000000076ab41738> (a java.lang.Object)
        at com.sun.net.ssl.internal.ssl.SSLSocketImpl.readDataRecord(SSLSocketImpl.java:820)
        at com.sun.net.ssl.internal.ssl.AppInputStream.read(AppInputStream.java:75)
- locked <0x000000076ab43988> (a com.sun.net.ssl.internal.ssl.AppInputStream)
        at java.io.BufferedInputStream.fill(BufferedInputStream.java:218)
        at java.io.BufferedInputStream.read(BufferedInputStream.java:237)
        - locked <0x000000076ab55098> (a org.jboss.util.stream.NotifyingBufferedInputStream)
        at org.jboss.util.stream.NotifyingBufferedInputStream.read(NotifyingBufferedInputStream.java:79)
        at java.io.ObjectInputStream$PeekInputStream.peek(ObjectInputStream.java:2248)
        at java.io.ObjectInputStream$BlockDataInputStream.readBlockHeader(ObjectInputStream.java:2428)
        at java.io.ObjectInputStream$BlockDataInputStream.refill(ObjectInputStream.java:2498)
        at java.io.ObjectInputStream$BlockDataInputStream.read(ObjectInputStream.java:2570)
        at java.io.ObjectInputStream$BlockDataInputStream.readByte(ObjectInputStream.java:2719)
        at java.io.ObjectInputStream.readByte(ObjectInputStream.java:894)
        at org.jboss.mq.il.uil2.SocketManager$ReadTask.run(SocketManager.java:340)
        at java.lang.Thread.run(Thread.java:662)


REGRESSION.  Last worked in version 6u31

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Heavy JMS load with Publisher and Receiver both on the same host, running for a  one or two weeks.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
NO Deadlocks in the Thread dump taken by JSTACK
ACTUAL -
these deadlocks continue to increase over the time and is slowing down the system below acceptable levels.

REPRODUCIBILITY :
This bug can be reproduced always.