JDK-8130025 : Tomcat enter stuck status, one thread hold the lock, and all others are blocked
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 7u72
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux
  • CPU: x86_64
  • Submitted: 2015-06-23
  • Updated: 2015-09-01
  • Resolved: 2015-09-01
Related Reports
Duplicate :  
JDK-6432000 - Static certificate cache can limit scalability
Description
FULL PRODUCT VERSION :
java version "1.7.0_72"
Java(TM) SE Runtime Environment (build 1.7.0_72-b14)
Java HotSpot(TM) 64-Bit Server VM (build 24.72-b04, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
CentOS release 6.5 (Final)

EXTRA RELEVANT SYSTEM CONFIGURATION :
Tomcat 7.0.57

A DESCRIPTION OF THE PROBLEM :
Hold the lock thread:
"http-bio-10886-exec-919" daemon prio=10 tid=0x00007fa54c57b800 nid=0x4489 runnable [0x00007fa4579ee000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.removeEntryForKey(Unknown Source)
	at java.util.HashMap.remove(Unknown Source)
	at sun.security.util.MemoryCache.emptyQueue(Unknown Source)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- locked <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)

all other threads are in the status as below:
"http-bio-10886-exec-921" daemon prio=10 tid=0x00007fa54c3fd000 nid=0x448e waiting for monitor entry [0x00007fa4576b5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
it happened in Prod env with high traffic

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Tomcat Server response normally
ACTUAL -
server stuck there.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
"http-bio-10886-exec-921" daemon prio=10 tid=0x00007fa54c3fd000 nid=0x448e waiting for monitor entry [0x00007fa4576b5000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa3a416730> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa3a41f3c0> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa3a41f398> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa3a42d0f0> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-919" daemon prio=10 tid=0x00007fa54c57b800 nid=0x4489 runnable [0x00007fa4579ee000]
   java.lang.Thread.State: RUNNABLE
	at java.util.HashMap.removeEntryForKey(Unknown Source)
	at java.util.HashMap.remove(Unknown Source)
	at sun.security.util.MemoryCache.emptyQueue(Unknown Source)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- locked <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa489031f0> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa4890be80> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa4890be58> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa48919bb0> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-916" daemon prio=10 tid=0x00007fa54c0d3000 nid=0x4463 waiting for monitor entry [0x00007fa59d1d3000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa4a9e6b48> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa4a9ef7d8> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa4a9ef7b0> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa4a9fd508> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-913" daemon prio=10 tid=0x00007fa54c1a5800 nid=0x445a waiting for monitor entry [0x00007fa4598d7000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa406b2ce8> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa406bb978> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa406bb950> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa406c96a8> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-915" daemon prio=10 tid=0x00007fa54c13e000 nid=0x4459 waiting for monitor entry [0x00007fa44e7fd000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa448210e0> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa44829d70> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa44829d48> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa44837aa0> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-914" daemon prio=10 tid=0x00007fa460080000 nid=0x4458 waiting for monitor entry [0x00007fa58065a000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa3e5c52a8> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa3e5cdf38> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa3e5cdf10> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa3e5dbc68> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

"http-bio-10886-exec-912" daemon prio=10 tid=0x00007fa460079800 nid=0x4455 waiting for monitor entry [0x00007fa58135f000]
   java.lang.Thread.State: BLOCKED (on object monitor)
	at sun.security.util.MemoryCache.put(Unknown Source)
	- waiting to lock <0x00007fab9d70f118> (a sun.security.util.MemoryCache)
	at sun.security.ssl.SSLSessionContextImpl.put(Unknown Source)
	at sun.security.ssl.ServerHandshaker.clientFinished(Unknown Source)
	at sun.security.ssl.ServerHandshaker.processMessage(Unknown Source)
	at sun.security.ssl.Handshaker.processLoop(Unknown Source)
	at sun.security.ssl.Handshaker.process_record(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.readRecord(Unknown Source)
	- locked <0x00007faa3c4a8aa8> (a sun.security.ssl.SSLSocketImpl)
	- locked <0x00007faa3c4b1738> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.performInitialHandshake(Unknown Source)
	- locked <0x00007faa3c4b1710> (a java.lang.Object)
	at sun.security.ssl.SSLSocketImpl.startHandshake(Unknown Source)
	at sun.security.ssl.SSLSocketImpl.getSession(Unknown Source)
	at org.apache.tomcat.util.net.jsse.JSSESocketFactory.handshake(JSSESocketFactory.java:243)
	at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:302)
	- locked <0x00007faa3c4bf468> (a org.apache.tomcat.util.net.SocketWrapper)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
	at java.lang.Thread.run(Unknown Source)

REPRODUCIBILITY :
This bug can be reproduced occasionally.

CUSTOMER SUBMITTED WORKAROUND :
Not sure if this solution (https://github.com/netty/netty/issues/832) is workable

SUPPORT :
YES
Comments
From the code it's clear that the thread isn't blocked, but makes progress too slow, as it sits in a loop pooling values from a reference queue and removing entries from a map. So the thread isn't really stuck, but just processing too slow, limiting the scalability. The issue is the same as in JDK-6432000, as it is caused by limitations of the same class sun.util.Cache. Closing as a duplicate.
01-09-2015
Received following response from the submitter: ----------------------------------------------------------------------------------------------- On 7/2/2015 7:28 AM, ....... wrote: > Hi, .........: > It happened in our Production Env, we didn't find chance to upgrade to 7u80, is there any suspect fixing between 7u72 and 7u80? 7u72 is our Java version > Thanks > .......... -----------------------------------------------------------------------------------------------
02-07-2015
Looking at the attached dump, this looks similar to JDK-6432000
01-07-2015
Received following response from the submitter: ---------------------------------------------------------------------------------------- On 7/1/2015 8:42 AM, .......... wrote: > > HI, .....: > > Thanks for your response. > > Please see attachment. > > it seems below thread hold the lock, and others Http thread are blocked to wait for this lock > > "http-bio-10886-exec-919" daemon prio=10 tid=0x00007fa54c57b800 nid=0x4489 runnable [0x00007fa4579ee000] > > Thanks > > ........ -----------------------------------------------------------------------------------------
01-07-2015
Sent an email to the submitter: -------------------------------------------------------------------------------------- On 6/29/2015 9:41 AM, ......... wrote: > Hi ..........., > > We would require more information in order to evaluate this issue. Can you provide the snapshots of the stack trace taken at 1 min intervals to see if progress is being made by the app? > Also, could you confirm if this issue is reproducible with latest JDK 7u80 and 8u45? > > Regards, > .................. ------------------------------------------------------------------------------------------------
29-06-2015