United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-6651382 The Java JVM SNMP provider reports incorrect stats when asked for multiple OIDs
JDK-6651382 : The Java JVM SNMP provider reports incorrect stats when asked for multiple OIDs

Details
Type:
Bug
Submit Date:
2008-01-16
Status:
Closed
Updated Date:
2012-10-01
Project Name:
JDK
Resolved Date:
2011-04-19
Component:
core-svc
OS:
solaris_10
Sub-Component:
java.lang.management
CPU:
x86
Priority:
P2
Resolution:
Fixed
Affected Versions:
6u3
Fixed Versions:

Related Reports
Backport:
Backport:
Backport:
Backport:

Sub Tasks

Description
The Java JVM SNMP provider reports incorrect stats when asked for multiple OIDs within JVM-MANAGEMENT-MIB::jvmMemPoolUsed and JVM-MANAGEMENT-MIB::jvmMemPoolMaxSize. I've traced this as far as something within  Java_sun_management_MemoryPoolImpl_getUsage0 within libmanagement, but that's as far as I was able to take it.  
 
Good results via snmpwalk example: 
snmpwalk -v2c  -cremovedpw 127.0.0.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11 
 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.1 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.2 = Counter64: 100364776 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.3 = Counter64: 3451328 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.4 = Counter64: 378413648 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.5 = Counter64: 44289744 bytes 
 
Now, when you query multiple OIDs at once, the JVM "sticks" on the first value it looked up and returns it for all data points: 
 
Bad results via requesting multiple OIDs: 
snmpget -v2c -cremovedpw 127.0.0.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.2 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.3 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.4 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.5 
 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.1 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.2 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.3 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.4 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.5 = Counter64: 13562176 bytes 
 
I've seen this problem for both the JvmMemPoolUsed and jvmMemPoolMaxSize tree, but it may exist for other components. 
 
The java version is: 
java version "1.6.0_03" 
Java(TM) SE Runtime Environment (build 1.6.0_03-b05) 
Java HotSpot(TM) Client VM (build 1.6.0_03-b05, mixed mode, sharing) 

The problem was first noticed when we tried to gather these stats for the first time from the JVM. Other SNMP stats provided by the JVM are working for us.
.
The following nightly tests are affected by this bug:-

closed/sun/management/snmp/jvmMemory/JvmMemPoolTest.sh
sun/management/jmxremote/bootstrap/JvmstatCountersTest.java

Also include the non-closed name for the test just in case
it ever moves:

sun/management/snmp/jvmMemory/JvmMemPoolTest.sh

                                    

Comments
EVALUATION

This is indeed a bug.
The culprit seems to be:
sun/management/snmp/jvminstr/JvmMemPoolEntryImpl.java

This class uses the keys 'memoryTag', 'peakMemoryTag' and
'collectMemoryTag' to cache the memory Usage, memory PeakUsage,
and memory CollectionUsage in the request-contextual cache - but
these keys do not contain any instance (=line) identification.
As a result all the lines in the table will use the same key, which is wrong.

The fix is quite simple.
I believe the methods getMemoryUsage(), getPeakMemoryUsage(), and
getCollectMemoryUsage() should use a key composed of:
   memoryTag+"."+jvmMemPoolIndex,
   peakMemoryTag+"."+jvmMemPoolIndex,
   collectMemoryTag+"."+jvmMemPoolIndex
instead.

What happens is that since the key doesn't contain the index id,
the first data which is put into the cache is returned for all
the lines, since all the lines are using the same keys.

Each line should use its own key, and the way it's done elsewhere
in the code is by appending "." + the value of the index to the common
root. If I'm not mistaken that's the root of the bug.

The only methods that should need to change are the method which use the
memoryTag, peakMemoryTag, and collectMemoryTag variables, and I believe
only the three methods getMemoryUsage(), getPeakMemoryUsage(), and
getCollectMemoryUsage() are concerned by this.

It would be good to check whether the other XXXXXEntryImpl classes
in the jvminstr package also have these bug.
I suspect that any table entry representing an MXBean exposing a
complex attribute (like MemoryUsage) will have this bug - they're
all built on the same model.
                                     
2008-01-24
SUGGESTED FIX

test: see attached closed.patch diff file (code base jdk7)
fix:  see attached jdk_fixes.patch diff file (code base jdk7)
                                     
2008-01-30
WORK AROUND

send one SNMP request per memory pool
                                     
2008-01-30



Hardware and Software, Engineered to Work Together