JDK-8303154 : Investigate and improve instruction cache flushing during compilation
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: compiler
  • Affected Version: 11,17,21
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2023-02-24
  • Updated: 2023-07-12
  • Resolved: 2023-03-16
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 17 JDK 21
17.0.9-oracleFixed 21 b15Fixed
Related Reports
Relates :  
Description
It was noticed that we flush the instruction cache three times for a single C1 compilation:

(1) Flushing of the temporary code buffer:
AbstractICache::invalidate_range
AbstractAssembler::flush
Compilation::emit_code_epilog
Compilation::emit_code_body
Compilation::compile_java_method
Compilation::compile_method

Why is that even required? We don't execute code in the buffer but copy it to the code cache first.

(2) When copying from the temporary buffer into the code cache:
AbstractICache::invalidate_range
CodeBuffer::copy_code_to
CodeBuffer::copy_code_and_locs_to
nmethod::nmethod
nmethod::new_nmethod
ciEnv::register_method
Compilation::install_code
Compilation::compile_method

(3) And again when committing the code in the code cache:
AbstractICache::invalidate_range
CodeCache::commit
nmethod::nmethod
nmethod::new_nmethod
ciEnv::register_method
Compilation::install_code
Compilation::compile_method

C2 seems to omit (1) but does (2) and (3) as well.

From [~kvn]:
"C2 does not flush his temp buffer. But it does the same 2 flushes during code installation. We need to investigate that"
"CodeBuffer::copy_code_to() is used also for runtime stubs and other codes generation.
CodeCache::commit() is used for adapters too. But adapters uses RuntimeBlob which calls copy_code_to().
It seems we can try to skip flashing in CodeCache::commit() in such case.
It could be some historical conflicting changes.
We need RFE and track all paths to AbstractICache::invalidate_range() when we publish code to CodeCache."
Comments
Fix request [17u] I backport this for parity with 17.0.9-oracle. Medium risk, simple change. Clean backport. SAP nightly testing passed.
11-07-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk17u-dev/pull/1559 Date: 2023-07-10 06:30:20 +0000
10-07-2023

Changeset: b7945bc9 Author: Damon Fenacci <damon.fenacci@oracle.com> Committer: Tobias Hartmann <thartmann@openjdk.org> Date: 2023-03-16 08:28:55 +0000 URL: https://git.openjdk.org/jdk/commit/b7945bc9e5db5761f17a9e56246424fbcab21627
16-03-2023

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/12877 Date: 2023-03-06 08:37:50 +0000
10-03-2023

In addition to multiple flushes for a single compilation unit, I also see multiple flushes for multiple compilations to the same base address (but presumably, different offsets). For example, I see Thread 2 "java" hit Breakpoint 3, 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 __aarch64_sync_cache_range: base: 0xd9102880 end: 0xd9106498 delta: 15384 #0 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 #1 0x0000fffff773b1cc in StubCodeMark::~StubCodeMark() () #2 0x0000fffff775e340 in StubGenerator::generate_bigIntegerRightShift() () #3 0x0000fffff776dcf4 in StubGenerator::generate_all() () #4 0x0000fffff773da00 in StubGenerator_generate(...) () #5 0x0000fffff776e218 in StubRoutines::initialize2() [clone .part.30] () #6 0x0000fffff71b78b0 in init_globals() () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #7 0x0000fffff77da370 in Threads::create_vm(...) () #8 0x0000fffff7270d5c in JNI_CreateJavaVM () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #9 0x0000fffff7f139e4 in JavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #10 0x0000fffff7f1784c in ThreadJavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #11 0x0000fffff7ed78f8 in start_thread () from /lib64/libpthread.so.0 #12 0x0000fffff7df21fc in thread_start () from /lib64/libc.so.6 Thread 2 "java" hit Breakpoint 3, 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 __aarch64_sync_cache_range: base: 0xd9102880 end: 0xd9106590 delta: 15632 #0 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 #1 0x0000fffff773b1cc in StubCodeMark::~StubCodeMark() () #2 0x0000fffff775ee7c in StubGenerator::generate_bigIntegerLeftShift() () #3 0x0000fffff776dd08 in StubGenerator::generate_all() () #4 0x0000fffff773da00 in StubGenerator_generate(...) () #5 0x0000fffff776e218 in StubRoutines::initialize2() [clone .part.30] () #6 0x0000fffff71b78b0 in init_globals() () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #7 0x0000fffff77da370 in Threads::create_vm(...) () #8 0x0000fffff7270d5c in JNI_CreateJavaVM () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #9 0x0000fffff7f139e4 in JavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #10 0x0000fffff7f1784c in ThreadJavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #11 0x0000fffff7ed78f8 in start_thread () from /lib64/libpthread.so.0 #12 0x0000fffff7df21fc in thread_start () from /lib64/libc.so.6 ... Thread 2 "java" hit Breakpoint 3, 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 __aarch64_sync_cache_range: base: 0xd9102880 end: 0xd91093cc delta: 27468 #0 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 #1 0x0000fffff773b1cc in StubCodeMark::~StubCodeMark() () #2 0x0000fffff7756274 in StubGenerator::generate_sha256_implCompress(...) () #3 0x0000fffff776de9c in StubGenerator::generate_all() () #4 0x0000fffff773da00 in StubGenerator_generate(...) () #5 0x0000fffff776e218 in StubRoutines::initialize2() [clone .part.30] () #6 0x0000fffff71b78b0 in init_globals() () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #7 0x0000fffff77da370 in Threads::create_vm(...) () #8 0x0000fffff7270d5c in JNI_CreateJavaVM () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #9 0x0000fffff7f139e4 in JavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #10 0x0000fffff7f1784c in ThreadJavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #11 0x0000fffff7ed78f8 in start_thread () from /lib64/libpthread.so.0 #12 0x0000fffff7df21fc in thread_start () from /lib64/libc.so.6 Thread 2 "java" hit Breakpoint 3, 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 __aarch64_sync_cache_range: base: 0xd9102880 end: 0xd91095a8 delta: 27944 #0 0x0000fffff6736f6c in __aarch64_sync_cache_range () from /lib64/libgcc_s.so.1 #1 0x0000fffff773b1cc in StubCodeMark::~StubCodeMark() () #2 0x0000fffff775d60c in StubGenerator::generate_updateBytesAdler32() () #3 0x0000fffff776ddd0 in StubGenerator::generate_all() () #4 0x0000fffff773da00 in StubGenerator_generate(...) () #5 0x0000fffff776e218 in StubRoutines::initialize2() [clone .part.30] () #6 0x0000fffff71b78b0 in init_globals() () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #7 0x0000fffff77da370 in Threads::create_vm(...) () #8 0x0000fffff7270d5c in JNI_CreateJavaVM () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/lib/server/libjvm.so #9 0x0000fffff7f139e4 in JavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #10 0x0000fffff7f1784c in ThreadJavaMain () from /usr/lib/jvm/java-17-openjdk-17.0.4.1.1-2.el8_6.aarch64/bin/../lib/libjli.so #11 0x0000fffff7ed78f8 in start_thread () from /lib64/libpthread.so.0 #12 0x0000fffff7df21fc in thread_start () from /lib64/libc.so.6 0xd9102880 to 0xd91095a8 is 400+ cache lines, many of which get flushed dozens of times. All those flushes happen during JNI_CreateJavaVM, but some people might care about JVM startup time.
25-02-2023

I found it useful to run under gdb and use (gdb) break __aarch64_sync_cache_range Breakpoint 3 at 0xfffff6736f6c (gdb) commands Type commands for breakpoint(s) 3, one per line. End with a line saying just "end". >printf "__aarch64_sync_cache_range: base: 0x%x end: 0x%x delta: %d\n", $x0, $x1, ($x1 - $x0) >where >continue >end (gdb) run Starting program: /usr/bin/java -XX:+PrintCompilation HelloWorld to get a log of all the base and end values, and thread stacks for all the call sites.
25-02-2023

We also need to look on buffer flush when adapters are generated (before their installation in CodeCache). This is most numerous case.
24-02-2023