JDK-8269345 : Add Linux-specific jcmd to trim the C-heap.
  • Type: CSR
  • Component: hotspot
  • Sub-Component: runtime
  • Priority: P4
  • Status: Closed
  • Resolution: Approved
  • Fix Versions: 18
  • Submitted: 2021-06-25
  • Updated: 2024-01-03
  • Resolved: 2021-07-22
Related Reports
CSR :  
JDK-8268893 - jcmd to trim the glibc heap
Description
Summary
-------

Add Linux-specific jcmd to trim the native C-heap.

Problem
-------

When returning memory back to the glibc C-heap using `free(3)`, that memory is often retained instead of being returned to the Operating System. The glibc is somewhat notorious for this behavior, whereas other libc-implementations release memory more promptly. This effect depends on allocation pattern, but usually the finer granular allocations are, the more pronounced is this effect. As a result the JVM process could experience permanent increases in working set size due to temporary spikes in C-heap usage.

Beside the obvious disadvantage of using too much memory it also confuses support engineers: faced with a JVM showing a high working set size, they cannot know if that footprint is really memory used by this process, or if part of that memory consists of unused glibc C-heap. Even though they can see how much C-heap the JVM thinks it uses (using NMT - and beware the sometimes considerable glibc-overhead), if NMT does not show anything untoward nothing is answered.

Solution
--------

A new Linux-specific command is introduced to the jcmd command collection. That command would trigger a call to `malloc_trim(3)` call inside the JVM process when running on glibc. In the future this could be expanded to other `libc` environments if needed and if they provide a suitable API.

`malloc_trim(3)` is a glibc-specific API to explicitly trim its C-heap [1]. When this API is called, the glibc attempts to reduce the C-heap footprint by shrinking the process heap (decreasing the brk) and discard dirty unneeded pages using `madvise(2)`. This usually has immediate effect and releases superfluous memory to the Operating System.

Being able to manually trigger `malloc_trim(3)` would have the following advantages:
- obviously it reduces memory pressure as a stop gap measure
- when analyzing cases of high memory footprint, it would allow to distinguish "real" footprint from cases where the glibc just holds on to memory. This is especially important for cases where the C-heap spike was used by code outside the JVM itself and therefore does not show up on NMT.

Specification
-------------

The proposed command takes the form:

`jcmd System.trim_native_heap`

and has no arguments.

Description: Attempts to free up memory by trimming the native C-heap

When running under glibc, calling it would cause `malloc_trim(3)` to be executed synchronously. `jcmd` will wait for the trim to finish and return with information about how much memory had been freed by this operation.

When running on non-glibc platforms (e.g. with muslc on Alpine), the command prints "Not available".

For example:
```
thomas@starfish:~$ jcmd 18770 System.trim_native_heap
18770:
Attempting trim...
Done.
Virtual size before: 28849744k, after: 28849724k, (-20k)
RSS before: 8685896k, after: 920740k, (-7765156k)  <<<<
Swap before: 0k, after: 0k, (0k)
```

A preliminary RFR (which may be adapted depending on the outcome of this CSR) can be found here: [4]


[1] https://man7.org/linux/man-pages/man3/malloc_trim.3.html

[2] https://bugs.ruby-lang.org/issues/15667

[3] https://github.com/openjdk/jdk/pull/4510#issuecomment-864762379

[4] https://github.com/openjdk/jdk/pull/4510
Comments
Moving revised request to Approved.
22-07-2021
I think all the issues have been resolved. David Holmes has additionally reviewd the CSR last week so I'm moving it to finalized. @JoeDarcy, can you please have a new look at it?
21-07-2021
Is there anything left to decide?
14-07-2021
Right, I misread your comment completely. My current implementation prints "Not available" on non-glibc-platforms. I'll modify the text.
08-07-2021
[~stuefe] Still Linux only. My comments and edits just pertain to non-glibc environments.
08-07-2021
@David: the original plan was to provide this command only for Linux. Even with the more generic name I still prefer that over a command which does nothing on the majority of platforms. We can add the command to other platforms if the need arises (if we flesh out that platform as well). Does that sound okay or do you prefer the command to be available everywhere and stubbing it out?
08-07-2021
[~dholmes] and/or [~stuefe] - what about the synopsis line? It still says "glibc".
08-07-2021
[~stuefe] I made an editing pass to make the descriptions more generic to fit the new non-glibc-specific name. One thing we need to decide is what happens when not running on glibc: does this immediately and silently return? does it report there is nothing to do? Or ... ?
08-07-2021
Ok, I updated the name to "trim_native_heap".
07-07-2021
A name like "trim_native_heap" for this functinality sounds more appropriate to me.
06-07-2021
That then takes me back to the even more general `trim_native_heap` which could be expanded as applicable to any platform and "libc" equivalent.
06-07-2021
I have no strong emotions about the name, I just like to move the issue forward. Thinking about this some more, I now agree with Joe - maybe we should leave the "glibc" out of the name completely. At least on AIX we have another libc being hesitant about returning cached memory, so it is not unthinkable we may want to extend that command to different platforms in the future. How about "libc_trim_heap" or "trim_libc_heap" instead?
06-07-2021
Hi [~darcy], The naming was discussed in the PR and the general view is that this is glibc specific and so the name should reflect that. I also asked in the PR if we already have platform specific jcmds and the answer is yes: PerfMap is Linux only.
06-07-2021
Hmm. Moving to Provisional, not Approved. Are there existing platform-specific jcmd options? Could / should this command be given a more neural name?
06-07-2021
Hi David, makes absolutely sense. Since CSRs are usually looked over by very few overworked approvers. ..Thomas
28-06-2021
Hi Thomas, I hadn't really noticed what the template states. I think "Alternative solutions may be discussed" is somewhat too broad - perhaps "mentioned" would be a better turn of phrase. Any in-depth discussion of alternatives should be happening in the JEP, in the main JBS issue or on a mailing list - otherwise it will just obscure rather than highlight the actual solution. That's not to say such discussion may not arise as a consequence of the CSR review - obviously a reviewer may ask "did you think about ...?" (though ideally there will already be a link to such discussion so the reviewer won't need to ask that question). Thanks for streamlining the content anyway.
28-06-2021
Thank you Volker! I followed David's advice and removed the "alternatives" section from the CSR text. For completeness sake, to preserve the discussion history, I duplicate it here: ---------------------- Alternatives: 1) calling trim automatically In theory, one could call `malloc_trim(3)` automatically, e.g. in a very naive implementation after ever os::free(). Or, more reasonably, tied to some sort of spike recognition inside os::free(), or to arena management. Other language runtimes call malloc_trim(3) tied to certain events, e.g. c-ruby after a full gc cycle [2], but that would not really help us since C-heap usage inside the JVM is usually not tied to events which are easy to isolate. It would be difficult to estimate when trimming makes sense: the JVM cannot predict future C-heap consumption pattern. Even if it notices a C-heap usage spike it does not know if that spike is an isolated incident and worth a trim, or if subsequent allocations could happen right away and trimming the C-heap now would result in an unnecessary performance penalty. Much more importantly, the JVM shares the glibc heap with outside consumers in the same process. So it may not be responsible for C-heap usage spikes, may not even be aware of them. An automatic trimming in addition to this new command is certainly possible but out of scope for this CSR. 2) a more generic trim command When this idea was first discussed, it had been proposed to add a generic command with a more vague description instead of adding a Linux-specific command [3]. That more generic command would be specified to "release system resources" - intentionally leaving the description vague - and platforms would fill out that implementation as they see fit. For now all platforms but Linux would leave that command empty. While this would be more elegant, this may be one of the cases where the vagueness attempts to hide complexity which cannot be hidden - the supporter needs to know exactly what this command does, to estimate the effects of calling it, and to analyze the results. In addition to that, leaving the implementation empty for most platform would be more confusing to users that a clearly named and specified glibc-specific command.
25-06-2021
Changed "Interface Kind" from "add/remove command in $JDK/bin" to "add/remove/modify command line option" because this CSR isreally just about adding a new option to the existing jcmd command. Also set scope to "JDK". This is consistent with previous CSRs which extended jcmd with new options. Otherwise this looks good to me.
25-06-2021
Hi David, thanks! The pre-filled CSR template is a bit confusing then, since I think it explicitly mentions alternatives in the "Solution" parts. In this case don't you think the alternatives make sense to avoid rehashing old discussions?
25-06-2021
[~stuefe] Hi Thomas, Please note as CSR request is only concerned with the details of the specification change. It is not a JEP and does not need to discuss pros and cons and alternatives. Such discussion belongs in a JEP (if a JEP is needed) or else a preliminary RFC mail thread on the appropriate mailing list(s). Once the change has been fleshed out then the specific proposed change is presented in the CSR request. Thanks.
25-06-2021