JDK-8256844 : Make NMT late-initializable
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 16
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2020-11-23
  • Updated: 2021-09-19
  • Resolved: 2021-08-04
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 18
18 b09Fixed
Related Reports
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
(Was: "NMT is not usable from gtestlauncher", but it turns out this is a more general restriction).

Today, NMT is of limited use due to the following restrictions:

- NMT cannot be used if the hotspot is embedded into a custom launcher unless the launcher actively cooperates. Just creating and invoking the JVM is not enough, it needs to do some steps prior to loading the hotspot. This limitation is not well known (nor, do I believe, documented). Many products don't do this, e.g., you cannot use NMT with IntelliJ. For us at SAP this problem limits NMT usefulness greatly since our VMs are often embedded into custom launchers and modifying every launcher is impossible.

- Worse, if that custom launcher links the libjvm statically there is just no way to activate NMT at all. This is the reason NMT cannot be used in the gtestlauncher.

- Related to that is that we cannot pass NMT options via JAVA_TOOL_OPTIONS and -XX:Flags=<file>.

- The fact that NMT cannot be used in gtests is really a pity since it would allow us to both test NMT itself more rigorously and check for memory leaks while testing other stuff.

The reason for all this is that NMT initialization happens very early, on the first call to os::malloc(). And those calls happen already during dynamic C++ initialization - a long time before the VM gets around parsing arguments. So, regular VM argument parsing is too late to parse NMT arguments.

The current solution is to pass NMT arguments via a specially prepared environment variable: NMT_LEVEL_<PID>=<NMT arguments>. That environment variable has to be set by the embedding launcher, before it loads the libjvm. Since its name contains the PID, we cannot even set that variable in the shell before starting the launcher.

All that means that every launcher needs to especially parse and process the NMT arguments given at the command line (or via whatever method) and prepare the environment variable. java itself does this. This only works before the libjvm.so is loaded, before its dynamic C++ initialization. For that reason, it does not work if the launcher links statically against the hotspot, since in that case C++ initialization of the launcher and hotspot are folded into one phase with no possibility of executing code beforehand.

And since it bypasses argument handling in the VM, it bypasses a number of argument processing ways, e.g., JAVA_TOOL_OPTIONS.

Comments
Changeset: eec64f55 Author: Thomas Stuefe <stuefe@openjdk.org> Date: 2021-08-04 12:19:02 +0000 URL: https://git.openjdk.java.net/jdk/commit/eec64f55870cf51746755d8fa59098a82109e826
04-08-2021

I did a quick test to see how much was allocated before NMT argument parsing happens. See attached https://bugs.openjdk.java.net/secure/attachment/93483/log1.txt. The category "EarlyBird" shows early allocations. This was done with the normal argument parsing untouched. We could (and should) move NMT initialization up to the very start of CreateJavaVM, which would reduce the window for early initializations to happen and make it more predictable. I did not see any reallocs or frees happen before argument parsing, and I did not see any virtual memory allocations. I was surprised btw that so much was allocated from the logging framework, even though logging was off. This is the initialization of the logging tag sets. In an ideal world this would just be a static array. All tags and their number are known at compile time, why do we pay here at startup for dynamic allocation?
06-03-2021

No problem :) just wanted to make sure I understood you correctly.
24-02-2021

Yea, how I missed that one :-( Sorry.
24-02-2021

Is that not the same idea as using a static buffer for early memory? Just instead of a global static buffer a malloc'ed one?
24-02-2021

An alternative: Allocate all bootstrap memory (before NMT level is known) into a memory pool from ::malloc(), assume the size is small, we don't mind to leak it. Since NMT level will be known while JVM still in single thread mode, that make things a lot simpler. e.g. allocation from pool just a pointer bumping, then os::free() just need additional range checking, detecting overflow the memory pool also trivial.
24-02-2021

Always allocating the malloc header, independent from NMT on off, would be a possible and simple solution, but the memory footprint increase would not be acceptable. Note that malloc headers have one hidden disadvantage in that they bump allocation sizes - usually nice power2 numbers since programmers are tidy - into crooked values, for which C-Heap-allocators may not be optimized, so the hidden waste may be larger.
24-02-2021

Hi Kim, I had the same idea (these were the "more complex" ideas I mentioned in the mail). Some remarks: Alternative to using a side table, I considered a preallocated static buffer into which we would place early allocations. The "is it an early allocation" would have been just two pointer comparisons (buffer start <= p < end). Depending on how many allocations we have, and if we also free and/or realloc, this could have been a simple buffer with pointer bump logic, or maybe with a simple freeblock list chain added. But regardless of the mechanism used to track early allocations, this approach has the disadvantage that - we pay on every free ever, regardless how little - tracking information can be out of sync since we miss early allocations. Zhengyu does not like that since he says users will spot the discrepancies and ask questions. Early allocation numbers seem quite fixed in size - especially if we take my proposal to move NMT initialization up to the very start of CreateJavaVM. I looked at these sites and I could not spot any runtime dependencies which would shake that number much. A lot of allocations come from early UL initialization btw, I was surprised we even pay for that if logging is off. I still like the very simple "dont allocate early at all" approach best though. But I am prepared to go this route if people don't like it. (Edit: Missed your last sentence, which is the same buffer suggestion I described)
24-02-2021

A possible alternative to always allocating the NMT malloc header. (I don't have an informed opinion about whether this would actually be good; measurements would be needed.) Start with NMT malloc headers. Start with a side table in which early allocations are recorded. Once we know whether NMT malloc headers are needed, either (1) If NMT malloc headers are needed, delete the side table and assume all allocations have the header. (2) If NMT malloc headers are not needed, stop including them for new allocations, stop recording allocations in the side table, and when freeing first look in the side table to see if there is an NMT header involved. A couple more questions that might be interesting. Do we know anything about the number of early allocations? So do we know how big such a side table would need to be? Is it more or less fixed in size (up to future code changes), or is it hard to predict? Or if early allocation is (reasonably) predictable in amount, pre-allocate a big enough block and do all the early allocation out of that. free can check whether the memory is in that block (range check, faster than a table lookup).
24-02-2021

We also could, of course, just always allocate malloc headers, NMT off or on. That would be a very simple solution, at the expense of some memory. Melting them with debug malloc headers, which we also have in addition of the NMT headers, would zero out the cost for debug builds. For release builds we would pay 8 bytes per malloc but it would have some nice side effects (beside easy NMT tracking): canaries, and using the same memory layout as in debug.
19-01-2021

Okay. I thought malloc headers were the reason, but maybe you never ran into this problem since you always initialized early. What I meant was that within a VM, you either have to always use malloc headers or can never use them. Because when someone calls os::free(p), p is opaque. You need to know if it is safe to substract the header size from that pointer. Failing to substract it or substracting it wrongly would mean you pass an invalid pointer to CRT free(), and that could corrupt the C-heap. So if we had late initialization, and pre-init malloc calls would have had no malloc header, but post-init malloc calls would have a malloc header, at os::free() you'd need to know if p was allocated before or after init.
19-01-2021

Dropping tracking level was designed in response of memory pressure (Maybe I went a little to far for that). Dropping from detail tracking, it can free the hashtable, But for summary tracking, it uses a static data structure, so there is no memory it can free by dropping to "off". Further more, NMT data is really useful in this circumstance and -XX:+PrintNMTStatistics was designed only report summary initially. It is not hard to add flag for os::malloc to track if it needs to create malloc tracking headers, event it is downgraded to off, but it is quite pointless.
19-01-2021

Malloc headers may be the biggest hurdle to late NMT initialization. The point is not that we would loose early initializations. The problem is that at os::free time, we need to know whether to subtract a header from the freed pointer or not. Allocations before late initialization would have no header; later allocations would have a header. That may also be the reason why level cannot be dropped to "off" once it had been enabled. There are several options to deal with this: - replacing tracking via headers with a hashmap (the SAP internal VM uses a custom hash map for similar purposes) but that seems overblown. - tracking early initializations separately, either by taking them from a separate pool, or by tracking those pointers in some map.
19-01-2021

Hi Zhengyu, >Precise stack tracking should be in JDK11 (JDK-8199067), should we backport it to 8u? My personal opinion is that I would not put too much work into old releases. jdk11 slowly replaces jdk8 for our customers, and as you say, there are plenty other ways to misinterprete the output of NMT.
19-01-2021

[~zgu] Thanks for confirming that. I can't see any reason the other tools pose any different consideration so that is a relief.
19-01-2021

[~dholmes] Supporting other tools discussion never came up, so it was not a deliberate decision.
19-01-2021

Today I had a customer issue (funny how things come together) where I really needed NMT but could not use it because customer uses the VM with their own launcher and that, of course, does not set up NMT. This reliance on a cooperating launchers is really bad. Therefore I reframed this issue to analyze if there is a possibility to get NMT running without launcher cooperation.
19-01-2021

[~zgu] Thanks for clarifying the "pollution" issue. I understand what you mean now, and yes I expect that remains a concern. In light of that constraint was there a deliberate decision to only enable the java launcher and not the other tools? (ref JDK-8258917).
18-01-2021

Hi Thomas, > (4) at least up to JDK15(?) NMT overreported thread stack usage (what you now do with mincore()), which confused some of our customers, and I have read more than one blog article trying to "fix" high thread stack usage in jdk8. Precise stack tracking should be in JDK11 (JDK-8199067), should we backport it to 8u? I thought about applying os::committed_in_range() from JDK-8199067 to committed regions to get accurate counting. But there are chances to have backing memory for the regions that JVM has yet touched, and that probably confuses JVM developers ... What do you think?
18-01-2021

Hi Zhengyu, >@stuefe: Yes, the earliest mallocs come from static initialization. However, even you eliminating those mallocs, there is no guarantee that there are no > mallocs before parameter passing. It is probably doable, but could be fragile. I agree. Probably not worth the effort. > NMT is really a JVM developer facing diagnostic tool, what are your cases for customers to run with NMT on? Yes, its for us developers. But at least with our customers, I often face situations where a NMT report would have been nice but NMT is off and it is often very difficult to convince customers to restart systems to change those parameters. > Recent years, I do see customers trying to NMT to exam Java applications' memory usage (e.g. comparing RSS). I really don't see it is a right tool: Another point for your list: (4) at least up to JDK15(?) NMT overreported thread stack usage (what you now do with mincore()), which confused some of our customers, and I have read more than one blog article trying to "fix" high thread stack usage in jdk8.
18-01-2021

I am trying my best to answer above questions: @dholmes: I am not sure if it is still the case: JVM is not supposed to be "polluted" by env-var and JVM's behaviors can only be affected by command line options. Without pid, a global env-var, e.g. NMT_LEVEL=xxx will enable NMT for all JVM instances. It was a no-no at the time, even JRocket uses env-var in its NMT implementation. Due to this restriction, we had an early implementation that used side-table for tracking malloc, but scaled poorly. Current implementation was a compromise: JVM takes env-var, but a private env-var between launcher and JVM. The scaling problem with side-table, may not be an issue today. At the time, there is only one "native" lock: ThreadCritical, it is not only a heavyweight lock, but also getting "hot" when using with NMT. @Joakim: NMT level can be downgraded (BTW, I think it is doable to downgrade to off). However, if you start NMT =detail, then downgrade to summary/off, NMT still needs to allocate tracking header for each allocation. So default to detail, then degrade, is a no-go, you always pay memory overhead ... @stuefe: Yes, the earliest mallocs come from static initialization. However, even you eliminating those mallocs, there is no guarantee that there are no mallocs before parameter passing. It is probably doable, but could be fragile. NMT is really a JVM developer facing diagnostic tool, what are your cases for customers to run with NMT on? Recent years, I do see customers trying to NMT to exam Java applications' memory usage (e.g. comparing RSS). I really don't see it is a right tool: 1) It does not track Java/native libraries' memory 2) Under reporting malloc memory (does not take malloc header/alignment into account) 3) Over reporting virtual memory (commit does not guarantee to have backing memory) IIRC, design goal was 5-10% performance penalty, I believe it is about ~5%.
18-01-2021

After further investigation, it seems you cannot transition NMT to off; you can only transition from "detail" to "summary" due to thread safety (allocating the structures thread-safe would require more locking). I've however also experimented by adding an ENV-variable (without pid) to set the level. It looks somewhat promising, but I guess this "pollutes" the JVM, and also makes the pid requirement unnecessary.
18-01-2021

Interesting idea, but we'd always pay for the full execution of MallocTracker::initialize() at startup. Not sure if that matters.
18-01-2021

There was a bug with different threads changing NMT level causing a crash (https://bugs.openjdk.java.net/browse/JDK-8059100), so apparently a fix was made that only allows the NMT level to be decreased, but not increased. Given that NMT can be decreased, perhaps the default when the NMT_LEVEL-ENV variable isn't set, should to set it high, and then change it according to the "-XX:NativeMemoryTrack"-argument later on.
18-01-2021

I don't see any concern with env var pollution here as this is not something you would set in your .bashrc but something set only in the JVM launch script when needed. I'm also surprised to learn that lack of the env-var disables NMT completely! Why would we do that instead of just allowing NMT to commence once VM init has reached a NMT initialization point? Sure it would be incomplete, but seems much better than nothing.
18-01-2021

Well, ideally we would have no code running at C++ dynamic initialization and no-one should allocate any memory before main(). But since NMT is supposed to track that, it needs to be able to run at C++ initialization time. At least the malloc tracker, that is. Since NMT is so useful, I will take some measurements. For our downstream VMs, we actually have long considered to just always enable it. Maybe the solution would be to make it that cheap that always-on or always-on-then-off (the Joakim-Approach) are feasible.
18-01-2021

I expect it would matter for startup. IMO we can't assume NMT is fully on until argument checking tells us otherwise. We have to assume it is off until argument checking tells us otherwise.
18-01-2021

@Zhengyu: Just a thought, but the ability to switch NMT on at any time would be really useful though (even with the caveat that we miss prior allocations and have to work around orphaned frees/releases). I always have to ask customers to restart their systems to get an NMT trace. Either that or just enabling it by default; the costs are not that high. What do you think?
16-01-2021

Requiring the PID in the env-var was intentional, to avoid environment variables "polluting" JVM (I was told that it was a decision that made long time ago). And "No", if the env-var does not present, NMT won't start. Arg processing just for warning user why NMT is not started, even NativeMemoryTracking is set.
16-01-2021

BTW this should just be a warning and NMT should work for anything that happens after it is enabled during arg processing.
14-01-2021

Requiring the PID in the env-var may have been overly proscriptive. If the pid is not used then the code that launches the statically linked launcher could set the env-var.
14-01-2021

Yes, as far as I understand it, you're correct. I hadn't thought of the static vs. dynamic loading of libjvm, but that makes sense. It's definitely worth checking out. I'll do some more testing and investigation.
13-01-2021

In order for NMT to work, there must be an environment variable set, "NMT_LEVEL_<PID>=detail|summary". Setting this is in gtestLauncher.cpp (by looking for argument "-XX:NativeMemoryTracking" as in libjli's java.c) doesn't work however, since the mem tracker is initialized (by a LogTagSet) before any code in gtestLauncher is run.
13-01-2021

Joakim, thanks for checking! Lets see if I understand correctly: - NMT is partly initialized as part of the static C++ initialization of the libjvm. Parts are initialized later (MemTracker::init()) called from arguments.cpp as part of arg parsing in hotspot. - The option given to CreateJavaVM and parsed by hotspot arg parsing has to match the content of the NMT_LEVEL_<PID> env variable set up before (MemTracker::check_launcher_nmt_support(tail)). It does not actually set the NMT level. The NMT level must have been set before via NMT_LEVEL_<PID>. - All that stuff only works for the java launcher because it dynamically loads the libjvm.so. And before that, also parses the "-XX:NativeMemoryTracking" option separately from hotpot, and sets up the env variable. By loading dynamically, static C++ initialization of NMT is delayed until after the env var is set. Have I got this right? If yes, this is not only a problem for gtestlauncher, but any launcher linking to the libjvm statically. NMT would not be usable in these circumstances. Since one cannot know the pid beforehand, one cannot even set the environment variable. I think this would be worth solving. NMT is so very useful.
13-01-2021