Bug ID: JDK-8343205 CompileBroker::possibly_add_compiler_threads excessively polls available memory

JDK-8343205 : CompileBroker::possibly_add_compiler_threads excessively polls available memory

Type: Bug
Component: hotspot
Sub-Component: compiler
Affected Version: 11,17,21,24

Priority: P4
Status: Resolved
Resolution: Fixed
OS: linux
CPU: x86_64

Submitted: 2024-10-25
Updated: 2024-12-16
Resolved: 2024-11-04

Versions (Unresolved/Resolved/Fixed)

The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.

JDK 21	JDK 24
21.0.7Fixed	24 b23Fixed

Related Reports

Relates :

JDK-8198756 - Lazy allocation of compiler threads

Description

A DESCRIPTION OF THE PROBLEM :
possibly_add_compiler_threads always fetches the free memory, before considering if a compiler thread is required, which we've observed causing performance issues with both cgroupv1 and cgroupv2 due to the frequency of polling (even with the mitigations added in JDK-8232207), particularly with large running large numbers of relatively short lived tasks on container platforms.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Believe this is enough to demonstrate the correlation between compilations and calls:

java -Xlog:os+container=trace,jit+compilation=debug
[0.000s][trace][os,container] OSContainer::init: Initializing Container Support
[0.000s][debug][os,container] Detected optional pids controller entry in /proc/cgroups
[0.001s][debug][os,container] Detected cgroups v2 unified hierarchy
[0.001s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/cpu.max
[0.001s][trace][os,container] CPU Quota is: 400000
[0.001s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/cpu.max
[0.001s][trace][os,container] CPU Period is: 100000
[0.001s][trace][os,container] CPU Quota count based on quota/period: 4
[0.001s][trace][os,container] OSContainer::active_processor_count: 4
[0.004s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 4
[0.004s][trace][os,container] total physical memory: 790776565760
[0.004s][trace][os,container] Path to /memory.max is /sys/fs/cgroup/memory.max
[0.004s][trace][os,container] Memory Limit is: 8589934592
[0.005s][trace][os,container] CgroupSubsystem::active_processor_count (cached): 4
[0.025s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/cpu.max
[0.025s][trace][os,container] CPU Quota is: 400000
[0.025s][trace][os,container] Path to /cpu.max is /sys/fs/cgroup/cpu.max
[0.025s][trace][os,container] CPU Period is: 100000
[0.025s][trace][os,container] CPU Quota count based on quota/period: 4
[0.025s][trace][os,container] OSContainer::active_processor_count: 4
[0.029s][debug][jit,compilation]    1       3       java.lang.String::hashCode (60 bytes)
[0.029s][trace][os,container   ] total physical memory: 790776565760
[0.029s][trace][os,container   ] Path to /memory.max is /sys/fs/cgroup/memory.max
[0.029s][trace][os,container   ] Memory Limit is: 8589934592
[0.029s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.029s][trace][os,container   ] Memory Usage is: 8166121472
[0.030s][debug][jit,compilation]    2       3       java.lang.Object::<init> (1 bytes)
[0.030s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.030s][trace][os,container   ] Memory Usage is: 8166121472
[0.030s][debug][jit,compilation]    3     n 0       jdk.internal.misc.Unsafe::getReferenceVolatile (native)
[0.030s][debug][jit,compilation]    4     n 0       jdk.internal.vm.Continuation::enterSpecial (native)   (static)
[0.030s][debug][jit,compilation]    5     n 0       jdk.internal.vm.Continuation::doYield (native)   (static)
[0.031s][debug][jit,compilation]    6       3       java.lang.Byte::toUnsignedInt (6 bytes)
[0.031s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.031s][trace][os,container   ] Memory Usage is: 8166383616
[0.032s][debug][jit,compilation]    7       3       java.lang.String::coder (15 bytes)
[0.032s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.032s][trace][os,container   ] Memory Usage is: 8166383616
[0.032s][debug][jit,compilation]   10       3       java.lang.String::isLatin1 (19 bytes)
[0.032s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.032s][trace][os,container   ] Memory Usage is: 8166383616
[0.032s][debug][jit,compilation]    8       3       jdk.internal.util.Preconditions::checkIndex (18 bytes)
[0.032s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.032s][trace][os,container   ] Memory Usage is: 8166383616
[0.032s][debug][jit,compilation]    9       3       java.lang.String::charAt (25 bytes)
[0.032s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.032s][trace][os,container   ] Memory Usage is: 8166383616
[0.032s][debug][jit,compilation]   11     n 0       java.lang.Object::hashCode (native)
[0.032s][debug][jit,compilation]   12     n 0       java.lang.invoke.MethodHandle::linkToStatic(LLLLLLL)L (native)   (static)
[0.033s][debug][jit,compilation]   13       1       java.lang.Enum::ordinal (5 bytes)
[0.033s][trace][os,container   ] Path to /memory.current is /sys/fs/cgroup/memory.current
[0.033s][trace][os,container   ] Memory Usage is: 8166383616


EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Free memory is only considered if a thread would otherwise be added.
ACTUAL -
Free memory is considered for each compiler loop iteration.

CUSTOMER SUBMITTED WORKAROUND :
Configure -XX:-UseDynamicNumberOfCompilerThreads

FREQUENCY : often

Comments

[jdk17u-fix-request] Approval Request from Martin JDK17 is affected by this performance issue, too. The fix is not complicated and the backport from 21u-dev applies cleanly. Tier1-4 plus extra tests have passed many times. No regression observed.
16-12-2024
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk17u-dev/pull/3106 Date: 2024-12-11 11:39:50 +0000
11-12-2024
Please note that it didn't make it into 23.0.2 because it was too late. The workaround -XX:-UseDynamicNumberOfCompilerThreads has other drawbacks.
18-11-2024
This is a relatively new change and should see some more real world runs e.g. in JDK 23 before putting into a backport release. Deferring to April for OpenJDK 21. Work-around is to use: -XX:-UseDynamicNumberOfCompilerThreads in the meantime.
18-11-2024
[jdk21u-fix-request] Approval Request from Martin JDK21 is affected by this performance issue, too. This solution is not complicated, the backport applies almost cleanly and has been reviewed. Tier1-4 plus extra tests and benchmarks have passed several times. No regression observed.
11-11-2024
[jdk23u-fix-request] Approval Request from Martin JDK23 is affected by this performance issue. This solution is not complicated and the backport applies cleanly. Tier1-4 plus extra tests and benchmarks have passed several times. No regression observed.
07-11-2024
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk21u-dev/pull/1123 Date: 2024-11-04 18:07:40 +0000
04-11-2024
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk23u/pull/214 Date: 2024-11-04 18:00:25 +0000
04-11-2024
Changeset: 75801992 Branch: master Author: Martin Doerr <mdoerr@openjdk.org> Date: 2024-11-04 09:58:51 +0000 URL: https://git.openjdk.org/jdk/commit/75801992a7c626d409f66e2491082dba84c6fe45
04-11-2024
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/21812 Date: 2024-10-31 17:03:33 +0000
31-10-2024
We can do the quick checks first and query os::free_memory() afterwards. This will make it less frequent. I'll take a look. Thanks for reporting!
31-10-2024
Martin, what do you think?
30-10-2024
> EXPECTED - > Free memory is only considered if a thread would otherwise be added. The current logic needs the free memory to determine if a new compiler thread should be added and therefore has to eagerly poll that information: https://github.com/openjdk/jdk/blob/master/src/hotspot/share/compiler/compileBroker.cpp#L1036 I think that logic should be changed to not use a MIN4 but first check if the compile queue is even filled enough to justify adding a new compiler thread. Only then we should check if there's enough memory (first checking the code cache and then system memory).
30-10-2024
ILW = Logic that determines if a new compiler thread should be added polls os::free_memory() too often causing performance issues similar to JDK-8232207, with a large number of short lived tasks on container platforms, -XX:-UseDynamicNumberOfCompilerThreads = MML = P4
30-10-2024