JDK-8296507 : GCM using more memory than necessary with in-place operations
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.crypto
  • Affected Version: 18,19
  • Priority: P2
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2022-11-08
  • Updated: 2025-02-14
  • Resolved: 2022-12-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 20
20 b27Fixed
Related Reports
Relates :  
JDK-8267125 - AES Galois CounterMode (GCM) interleaved implementation using AVX512 + VAES instructions
Description
With the addition of the AVX512 intrinsic that combined gctr and ghash operations, caused a check for in-place operations to require a new buffer for decryption to avoid complications with tag mismatches.  This below mail thread and test showed that TLS, who uses in-place byte[] or heap bytebuffers can drive up memory usage when the system.  Direct ByteBuffers are unaffected as the intrinsic does not call them directly.

https://mail.openjdk.org/pipermail/security-dev/2022-October/032723.html
https://mail.openjdk.org/pipermail/security-dev/2022-November/032757.html
Comments
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/11121 Date: 2022-11-13 02:54:10 +0000
14-02-2025
Changeset: b4da0ee7 Author: Anthony Scarpino <ascarpino@openjdk.org> Date: 2022-12-06 21:37:12 +0000 URL: https://git.openjdk.org/jdk/commit/b4da0ee706b6a274e6ba4e5483ef972f45c9f81e
06-12-2022
While I saw a small ops/sec difference between jdk17 and 19 when I ran the performance numbers, the memory usage was a lot larger when looking at JFR. The test mentioned in the mailing links showed ops/sec went up (bad direction) from 77 to 90, but JFR byte[] allocations went from 700MB to 13.7GB. After change in this fix, the numbers returned to normal. No longer are in-place operations using the combined GCM intrinsic. They must go back to first running ghash over the whole buffer, then decrypting like it was in jdk17. From what I have seen in perf testing, the numbers have increased despite using slower intrinsics at small and larger data sizes. This shows the memory usage slowed the system despite using the faster intrinsic. With a modified perf test for in-place ops decrypt: JDK 17: 122913.554 ops/sec JDK 19: 94885.008 ops/sec Post fix: 122735.804 ops/sec
15-11-2022