Relates :
|
Customer's app is not scaling well with double load causing about a 6 fold increase in cpu as well as high thread contention which we can see in the Java Flight Recorder(JFR)recording. See attached. Contention is due to threads competing for the classloader. The recording shows numerous blocked threads on classloader. We do observe weblogic.utils.classloaders.ChangeAwareClassLoader (21K count) in the JFR recording. One of the highest caller is from com.sun.org.apache.xalan.internal.utils. The customer Note we has already ruled out things like GC performance, heap size, physical memory, etc. The GC logs, CA Introscope monitoring, and the JFR recording all show GCs are not a concern. We also have the same issue with a 10g heap in our labs (the attached JFR is from production with a 5g heap). The application is not scaling as it should with increased load. A doubling of load causes several times more cpu usage, several times more threads, very high amounts of contention (lots of blocked threads), and significant delays in response times. Doubling the load has caused response times to increase exponentially. The attached JFR shows very high contention among threads. We are trying to understand why this contention is happening. We have 16 instances per server so 4% cpu per instance is a lot in this scenario. We normally run at about 0 - 1% and don't expect a small increase in load to put us at 4% cpu per instance. The issue gets far worse if load gets higher and has caused a production outage before with 100% system cpu. Our biggest issue is that response times for the app increase exponentially. This is unacceptable and we cannot scale the application to handle upcoming production load increases. Check out at all the blocking in the JFR. It is really apparent when you look at the Graph tab for Events. We are trying to understand why there is so much contention as we do not see this behavior with lower load. It appears to be related to the classloader but we don't understand why this is happening.