While investigating a memory leak in one of my applications, I created a simple reproducer in which the symbol table grows unbounded.
It is caused by an accidental increment of the Symbol refcount during bucket cleanup - the concurrentHashTable delete_in_bucket routine uses (abuses?) the lookup function given to it, which for symbol table increments the refcount, under the assumption that a successful lookup means a new reference.
This new test case for test/hotspot/gtest/classfile/test_symbolTable.cpp shows the issue succintly:
```
TEST_VM(SymbolTable, test_cleanup_leak) {
// Check that dead entry cleanup doesn't increment refcount of live entry in same bucket.
// Create symbol and release ref, marking it available for cleanup.
Symbol* entry1 = SymbolTable::new_symbol("hash_collision_123");
entry1->decrement_refcount();
// Create a new symbol in the same bucket, which will notice the dead entry and trigger cleanup.
// Note: relies on SymbolTable's use of String::hashCode which collides for these two values.
Symbol* entry2 = SymbolTable::new_symbol("hash_collision_397476851");
ASSERT_EQ(entry2->refcount(), 1) << "Symbol refcount just created is 1";
}
```
This test fails, entry2's refcount is actually 2 at this point because of the cleanup logic incrementing the refcount via equals. I have a patch to fix this.
Note that I observed this behaviour in a real application which churns through a lot of short-lived LambdaForms classes. I have attached a reproducer (ClassChurn.java) for the more realistic class churn scenario, where this leak (and possibly some others) can be observed. You can observe the RSS of the process growing over time, and the symbol table size can be observed growing via NativeMemoryTracking or jcmd VM.symboltable. I run it with `java -Xms200M -Xmx200m -XX:MaxMetaspaceSize=200M -XX:+AlwaysPreTouch ClassChurn.java`.