JDK-6651382 : The Java JVM SNMP provider reports incorrect stats when asked for multiple OIDs
  • Type: Bug
  • Component: core-svc
  • Sub-Component: java.lang.management
  • Affected Version: 6u3
  • Priority: P2
  • Status: Closed
  • Resolution: Fixed
  • OS: solaris_10
  • CPU: x86
  • Submitted: 2008-01-16
  • Updated: 2012-10-01
  • Resolved: 2011-04-19
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other Other JDK 6 JDK 7 Other
5.0u18-rev,OpenJDK6Fixed 5.0u19Fixed 6u10Fixed 7 b25Fixed OpenJDK6Fixed
Description
The Java JVM SNMP provider reports incorrect stats when asked for multiple OIDs within JVM-MANAGEMENT-MIB::jvmMemPoolUsed and JVM-MANAGEMENT-MIB::jvmMemPoolMaxSize. I've traced this as far as something within  Java_sun_management_MemoryPoolImpl_getUsage0 within libmanagement, but that's as far as I was able to take it.  
 
Good results via snmpwalk example: 
snmpwalk -v2c  -cremovedpw 127.0.0.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11 
 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.1 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.2 = Counter64: 100364776 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.3 = Counter64: 3451328 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.4 = Counter64: 378413648 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.5 = Counter64: 44289744 bytes 
 
Now, when you query multiple OIDs at once, the JVM "sticks" on the first value it looked up and returns it for all data points: 
 
Bad results via requesting multiple OIDs: 
snmpget -v2c -cremovedpw 127.0.0.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.1 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.2 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.3 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.4 1.3.6.1.4.1.42.2.145.3.163.1.1.2.110.1.11.5 
 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.1 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.2 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.3 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.4 = Counter64: 13562176 bytes 
JVM-MANAGEMENT-MIB::jvmMemPoolUsed.5 = Counter64: 13562176 bytes 
 
I've seen this problem for both the JvmMemPoolUsed and jvmMemPoolMaxSize tree, but it may exist for other components. 
 
The java version is: 
java version "1.6.0_03" 
Java(TM) SE Runtime Environment (build 1.6.0_03-b05) 
Java HotSpot(TM) Client VM (build 1.6.0_03-b05, mixed mode, sharing) 

The problem was first noticed when we tried to gather these stats for the first time from the JVM. Other SNMP stats provided by the JVM are working for us.
.
The following nightly tests are affected by this bug:-

closed/sun/management/snmp/jvmMemory/JvmMemPoolTest.sh
sun/management/jmxremote/bootstrap/JvmstatCountersTest.java

Also include the non-closed name for the test just in case
it ever moves:

sun/management/snmp/jvmMemory/JvmMemPoolTest.sh
Comments
WORK AROUND send one SNMP request per memory pool
30-01-2008
SUGGESTED FIX test: see attached closed.patch diff file (code base jdk7) fix: see attached jdk_fixes.patch diff file (code base jdk7)
30-01-2008
EVALUATION This is indeed a bug. The culprit seems to be: sun/management/snmp/jvminstr/JvmMemPoolEntryImpl.java This class uses the keys 'memoryTag', 'peakMemoryTag' and 'collectMemoryTag' to cache the memory Usage, memory PeakUsage, and memory CollectionUsage in the request-contextual cache - but these keys do not contain any instance (=line) identification. As a result all the lines in the table will use the same key, which is wrong. The fix is quite simple. I believe the methods getMemoryUsage(), getPeakMemoryUsage(), and getCollectMemoryUsage() should use a key composed of: memoryTag+"."+jvmMemPoolIndex, peakMemoryTag+"."+jvmMemPoolIndex, collectMemoryTag+"."+jvmMemPoolIndex instead. What happens is that since the key doesn't contain the index id, the first data which is put into the cache is returned for all the lines, since all the lines are using the same keys. Each line should use its own key, and the way it's done elsewhere in the code is by appending "." + the value of the index to the common root. If I'm not mistaken that's the root of the bug. The only methods that should need to change are the method which use the memoryTag, peakMemoryTag, and collectMemoryTag variables, and I believe only the three methods getMemoryUsage(), getPeakMemoryUsage(), and getCollectMemoryUsage() are concerned by this. It would be good to check whether the other XXXXXEntryImpl classes in the jvminstr package also have these bug. I suspect that any table entry representing an MXBean exposing a complex attribute (like MemoryUsage) will have this bug - they're all built on the same model.
24-01-2008