JDK-8319548 : Unexpected internal name for Filler array klass causes error in VisualVM
  • Type: Bug
  • Component: hotspot
  • Sub-Component: gc
  • Affected Version: 21,22,23
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2023-11-06
  • Updated: 2024-04-04
  • Resolved: 2023-12-21
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21 JDK 22 JDK 23
21.0.3Fixed 22.0.2Fixed 23 b03Fixed
Related Reports
Relates :  
Description
I am investigating the following VisualVM bug https://github.com/oracle/visualvm/issues/523 and it looks to me that the name of FillerArray in jmap -histo does not follow the naming convention. Below is the current output from JDK 21:

jmap -histo 334286
 num     #instances         #bytes  class name (module)
-------------------------------------------------------
   1:         22613        1141112  [B (java.base@21.0.1)
   2:         21488         515712  java.lang.String (java.base@21.0.1)
   3:          3799         470352  java.lang.Class (java.base@21.0.1)
   4:           290         350480  Ljdk.internal.vm.FillerArray; (java.base@21.0.1)
   5:          4081         312000  [Ljava.lang.Object; (java.base@21.0.1)
   6:           974         177520  [I (java.base@21.0.1)
   7:          5397         172704  java.util.HashMap$Node (java.base@21.0.1)
   8:          4819         154208  java.util.concurrent.ConcurrentHashMap$Node (java.base@21.0.1)
   9:           948         137208  [C (java.base@21.0.1)
  10:          3264         104448  java.util.Hashtable$Entry (java.base@21.0.1)
  11:           116          86200  [J (java.base@21.0.1)
  12:           622          81920  [Ljava.util.HashMap$Node; (java.base@21.0.1)
  13:          2486          79552  javax.swing.text.html.parser.ContentModel (java.desktop@21.0.1)


I guess that the correct name should be [Ljdk.internal.vm.FillerArray; - currently it is missing the array prefix [. Alternatively if FillerArray should be reported as an instance of a class (not an array) that the name should be jdk.internal.vm.FillerArray like any other class in the output produced by jmap -histo. However if FillerArray is reported as instance of a class, this means (based on the example above) that there are 290 instances of the FillerArray class, which take up 350480 bytes and so one instance is 1208.5517 bytes - which looks really strange to me.

FYI: I was able to "fix" the FillerArray name with the small naive patch below:

diff --git a/src/hotspot/share/memory/universe.cpp b/src/hotspot/share/memory/universe.cpp
index 1c0a5461ffe..1f5c1ab03bd 100644
--- a/src/hotspot/share/memory/universe.cpp
+++ b/src/hotspot/share/memory/universe.cpp
@@ -337,7 +337,7 @@ void Universe::genesis(TRAPS) {
       // Initialization of the fillerArrayKlass must come before regular
       // int-TypeArrayKlass so that the int-Array mirror points to the
       // int-TypeArrayKlass.
-      _fillerArrayKlassObj = TypeArrayKlass::create_klass(T_INT, "Ljdk/internal/vm/FillerArray;", CHECK);
+      _fillerArrayKlassObj = TypeArrayKlass::create_klass(T_INT, "[Ljdk/internal/vm/FillerArray;", CHECK);
       for (int i = T_BOOLEAN; i < T_LONG+1; i++) {
         _typeArrayKlassObjs[i] = TypeArrayKlass::create_klass((BasicType)i, CHECK);
       }

Comments
Yeah, apologies, your analysis is correct, regarding the the shipping release on said dates. My substantive point is that 22u did not have this fix, so if one runs into this bug on a JDK 22 release, the only course of action is to downgrade to a 21 update - which seems wrong to me. It should be possible to upgrade to a 22 update to get the fix (before 23 is shipped). Again, maybe this is particular bug fix is not worth getting overly excited about. I'm more surprised by the general approach to porting, and if it could be improved.
04-04-2024

The original commit was on Dec 21, 2023, the backport to 21u landed on Jan 05, 2024, targeting the closest release, 21.0.3 in April 2024. (It was too late for 21.0.2 in January 2024, which was deep in rampdown.) This looks completely in line with the idea of delivering fixes in the latest shipping LTS.
04-04-2024

There is a more general concern here - the lack of porting a bug fix to an update of the latest shipping release (whether STS or LTS), since it disadvantages the deployment model of "always on the latest Java" - since the latest shipping release may be missing some crucial bug fixes. [ In this particular case, I mistakenly thought that this bug was the cause of the error we were seeing. At best, it is just misleading and confusing ]
04-04-2024

For this particular patch, no particular reason, just avoiding the churn in STS release that would EOL soon anyway.
04-04-2024

[~shade] is there a reason why you didn't port this to the jdk22u release first ?
04-04-2024

Elasticsearch 8.13 is the first release to use JDK 22, and fails with `java.lang.IncompatibleClassChangeError: Class Ljdk.internal.vm.FillerArray; does not implement the requested interface java.util.concurrent.locks.Lock`. Further details in https://github.com/elastic/elasticsearch/issues/106987 I assume that the above error only happens when dumping the heap on OOM, since the process is run with `-XX:+HeapDumpOnOutOfMemoryError` and `-XX:+ExitOnOutOfMemoryError`? Is this assumption correct? Is there any workaround for the issue? [EDIT - this is not correct - I believe we see this error at non OOM times - see JDK-8329528] Full stacktrace: [2024-03-31T00:01:29,450][ERROR][o.e.b.ElasticsearchUncaughtExceptionHandler] [xxx] fatal error in thread [elasticsearch[xxx][write][T#7]], exiting java.lang.IncompatibleClassChangeError: Class Ljdk.internal.vm.FillerArray; does not implement the requested interface java.util.concurrent.locks.Lock at org.elasticsearch.common.util.concurrent.ReleasableLock.acquire(ReleasableLock.java:43) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.index.translog.Translog.add(Translog.java:578) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.index.engine.InternalEngine.index(InternalEngine.java:1223) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.index.shard.IndexShard.index(IndexShard.java:1072) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.index.shard.IndexShard.applyIndexOperation(IndexShard.java:997) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.index.shard.IndexShard.applyIndexOperationOnPrimary(IndexShard.java:915) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.bulk.TransportShardBulkAction.executeBulkItemRequest(TransportShardBulkAction.java:378) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.bulk.TransportShardBulkAction$2.doRun(TransportShardBulkAction.java:235) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.bulk.TransportShardBulkAction.performOnPrimary(TransportShardBulkAction.java:305) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:151) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.bulk.TransportShardBulkAction.dispatchedShardOperationOnPrimary(TransportShardBulkAction.java:79) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.action.support.replication.TransportWriteAction$1.doRun(TransportWriteAction.java:216) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.common.util.concurrent.TimedRunnable.doRun(TimedRunnable.java:33) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.common.util.concurrent.ThreadContext$ContextPreservingAbstractRunnable.doRun(ThreadContext.java:984) ~[elasticsearch-8.13.0.jar:?] at org.elasticsearch.common.util.concurrent.AbstractRunnable.run(AbstractRunnable.java:26) ~[elasticsearch-8.13.0.jar:?] at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1144) ~[?:?] at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:642) ~[?:?] at java.lang.Thread.run(Thread.java:1570) ~[?:?
02-04-2024

[jdk22u-fix-request] Approval Request from Chris Hegarty Clean backport to improve heap dump reliability w.r.t. filler objects. Applies cleanly.
02-04-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk22u/pull/119 Date: 2024-04-02 08:45:48 +0000
02-04-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk21u-dev/pull/123 Date: 2024-01-04 09:48:55 +0000
04-01-2024

[jdk21u-fix-request] Approval Request from Aleksey Shipilëv Clean backport to improve heap dump reliability w.r.t. filler objects. Only happens after JDK 19. Applies cleanly.
04-01-2024

Changeset: 05745e3f Author: Thomas Schatzl <tschatzl@openjdk.org> Date: 2023-12-21 09:17:31 +0000 URL: https://git.openjdk.org/jdk/commit/05745e3f1d56f71d7647e81fa5933c9f4ed18430
21-12-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/17155 Date: 2023-12-19 10:08:14 +0000
19-12-2023

Another option is to not dump these filler objects.
08-11-2023