JDK-8298249 : Excessive memory allocation in CipherInputStream AEAD decryption
  • Type: Bug
  • Component: security-libs
  • Sub-Component: javax.crypto
  • Affected Version: 11,17,19
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2022-11-28
  • Updated: 2024-04-11
  • Resolved: 2022-12-15
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 21
21 b03Fixed
Related Reports
Blocks :  
Relates :  
Relates :  
Description
ADDITIONAL SYSTEM INFORMATION :
Tested on Linux, Mac, Java 11, 17, 19

A DESCRIPTION OF THE PROBLEM :
The title pretty much sums up the issue, the runtime of GCM decryption is O(n²) with respect to the size of the input.

issue was also described https://stackoverflow.com/questions/74575538/why-is-the-runtime-complexity-of-gcm-mode-encryption-on²-in-java

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Looking at the code, it is buffering all input before it would push it out, but that should be (at worst) O(lg(n)) as the buffer doubles in size each time to copy the input bytes.



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
That the runtime of the decrypt would be linear with respect to the size of the input, that is, doubling the size of the input should take roughly double the time to decrypt.
ACTUAL -
On my machine, this gives:

```
*** Run 3 ***
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=1M  Encrypted=13ms  Decrypted1=91ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=2M  Encrypted=25ms  Decrypted1=236ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=4M  Encrypted=56ms  Decrypted1=854ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=8M  Encrypted=104ms  Decrypted1=3552ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=16M  Encrypted=202ms  Decrypted1=13896ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE  Size=32M  Encrypted=394ms  Decrypted1=53576ms result1=true
```

The O(n²) runtime of the decrypt is quite clear to see!


---------- BEGIN SOURCE ----------
import java.io.BufferedInputStream;
import java.io.ByteArrayInputStream;
import java.io.ByteArrayOutputStream;
import java.io.FileInputStream;
import java.io.FileOutputStream;
import java.io.InputStream;
import java.io.OutputStream;
import java.util.Arrays;
import java.util.List;
import java.util.Random;
import java.util.function.BiFunction;
import java.util.function.Function;

import javax.crypto.Cipher;
import javax.crypto.CipherInputStream;
import javax.crypto.CipherOutputStream;
import javax.crypto.spec.GCMParameterSpec;
import javax.crypto.spec.IvParameterSpec;
import javax.crypto.spec.SecretKeySpec;
public class AES_Test {

  public static void main(String[] args) throws Exception {
    Random r = new Random();
    byte[] key = new byte[32];
    byte[] spec = new byte[12];
    byte[] iv = new byte[16];
    r.nextBytes(key);
    r.nextBytes(spec);
    r.nextBytes(iv);

    List<BiFunction<Integer, SecretKeySpec, Cipher>> cipherCreators = List.of(
      (mode, serverKey) -> {
        GCMParameterSpec eGcmParameterSpec = new GCMParameterSpec(16 * 8, spec);
        try
        {
          Cipher eCipher = Cipher.getInstance("AES/GCM/NoPadding");
          eCipher.init(mode, serverKey, eGcmParameterSpec);
          return eCipher;
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
      },
      (mode, serverKey) -> {
        IvParameterSpec ivSpec = new IvParameterSpec(iv);
        Cipher eCipher;
        try {
          eCipher = Cipher.getInstance("AES/CBC/PKCS5Padding");
          eCipher.init(mode, serverKey, ivSpec);
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
        return eCipher;
      },
      (mode, serverKey) -> {
        IvParameterSpec ivSpec = new IvParameterSpec(iv);
        Cipher eCipher;
        try {
          eCipher = Cipher.getInstance("AES/CTR/NoPadding");
          eCipher.init(mode, serverKey, ivSpec);
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
        return eCipher;
      },
      (mode, serverKey) -> {
        IvParameterSpec ivSpec = new IvParameterSpec(iv);
        Cipher eCipher;
        try {
          eCipher = Cipher.getInstance("AES/CTS/NoPadding");
          eCipher.init(mode, serverKey, ivSpec);
        } catch (Exception e) {
          throw new RuntimeException(e);
        }
        return eCipher;
      }
    );

    SecretKeySpec serverKey = new SecretKeySpec(key, "AES");
    GCMParameterSpec gcmParameterSpec = new GCMParameterSpec(16 * 8, spec);

    for (int j = 0; j < 3; j++) {
      System.out.println("*** Run " + (j + 1) + " ***");
      for (BiFunction<Integer, SecretKeySpec, Cipher> cipherCreator : cipherCreators) {
        for (int i = 1; i <= 32; i *= 2) {
          byte[] randomBytes = new byte[i * 1024 * 1024];
          r.nextBytes(randomBytes);
            long start = System.currentTimeMillis();
          // Encrypt
          ByteArrayOutputStream bout = new ByteArrayOutputStream(randomBytes.length);
          {
            Cipher encryptCipher = cipherCreator.apply(Cipher.ENCRYPT_MODE, serverKey);
            ByteArrayInputStream fin = new ByteArrayInputStream(randomBytes);
            OutputStream cout = new CipherOutputStream(bout, encryptCipher);

            fin.transferTo(cout);
            cout.close();
          }
          byte[] encBytes = bout.toByteArray();
          long encrypted = System.currentTimeMillis();
          // Decrypt
          {
            InputStream fin = new ByteArrayInputStream(encBytes);
            Cipher decryptCipher = cipherCreator.apply(Cipher.DECRYPT_MODE, serverKey);
            InputStream cin = new CipherInputStream(fin, decryptCipher);
            bout = new ByteArrayOutputStream(randomBytes.length);

            cin.transferTo(bout);
          }
          long decrypted = System.currentTimeMillis();

          System.out.println(cipherCreator.apply(Cipher.ENCRYPT_MODE, serverKey).toString() + "  Size=" + i + "M  Encrypted=" + (encrypted - start) + "ms  Decrypted1=" + (decrypted - encrypted) + "ms result1=" + Arrays.equals(randomBytes, bout.toByteArray()));
        }
      }
    }
  }
}
---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
It is something specific to the JDK implementation, since Bouncycastle does not exhibit this behaviour:
```
*** Run 3 ***
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=1M  Encrypted=15ms  Decrypted1=16ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=2M  Encrypted=28ms  Decrypted1=30ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=4M  Encrypted=51ms  Decrypted1=59ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=8M  Encrypted=111ms  Decrypted1=124ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=16M  Encrypted=196ms  Decrypted1=222ms result1=true
Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: BC  Size=32M  Encrypted=362ms  Decrypted1=443ms result1=true
```

FREQUENCY : always



Comments
If one backports this to jdk17u-dev, then these tests wold fail: javax/crypto/Cipher/CipherInputStreamExceptions.java com/sun/crypto/provider/Cipher/AEAD/ReadWriteSkip.java There looks to be an implicit dependency on JDK-8267125 that redid a few things in GCM code. Applying this patch to commit before JDK-8267125 fails the tests, applying on top of JDK-8267125 passes the tests. I think it is risky to backport. The mitigation for JDK 17 might be JDK-8330108.
11-04-2024

A pull request was submitted for review. URL: https://git.openjdk.org/jdk17u-dev/pull/2393 Date: 2024-04-11 14:29:10 +0000
11-04-2024

Changeset: b9074fa1 Author: Daniel Jeliński <djelinski@openjdk.org> Date: 2022-12-15 06:54:33 +0000 URL: https://git.openjdk.org/jdk/commit/b9074fa1ed489993d60ce873fd8105a95d30782a
15-12-2022

A pull request was submitted for review. URL: https://git.openjdk.org/jdk/pull/11597 Date: 2022-12-08 16:33:06 +0000
09-12-2022

Thanks for reporting! There's a problem in the interaction between CipherInputStream and AEAD decryption. CipherInputStream reads data from the input in 512 byte chunks, allocating a new internal array if the expected output size is too large for the existing one, see CipherInputStream.getMoreData: https://github.com/openjdk/jdk/blob/f804f2ce8ef7a859aae021b20cbdcd9e34f9fb94/src/java.base/share/classes/javax/crypto/CipherInputStream.java#L156 AEAD decryption only outputs data from doFinal(); it buffers data received in update(), increasing the expected output size. As a result, CipherInputStream allocates a new output array every time it reads a new 512 byte chunk. When larger arrays are allocated, GC runs more frequently, which explains the observed behavior. I'll see if we can modify the CipherInputStream to better work with AEAD ciphers. BC returns decrypted data before verifying the AEAD tag, which is why this issue did not reproduce with BC provider. It reduces the memory pressure, but requires careful handling to make sure that no data is used before the tag is verified (i.e. CryptoInputStream.read returns -1)
08-12-2022

Issue is reproduced. The runtime complexity of GCM mode decryption looks to be O(n²) with respect to the size of input. OS: Mac OS Monterey (12.5) JDK 11.0.17 :Fail JDK 17.0.5: Fail JDK 19.0.1: Fail Output: ***Run 3*** Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=1M Encrypted=15ms Decrypted1=52ms result1=true Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=2M Encrypted=31ms Decrypted1=166ms result1=true Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=4M Encrypted=59ms Decrypted1=605ms result1=true Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=8M Encrypted=121ms Decrypted1=2156ms result1=true Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=16M Encrypted=235ms Decrypted1=8453ms result1=true Cipher.AES/GCM/NoPadding, mode: encryption, algorithm from: SunJCE Size=32M Encrypted=489ms Decrypted1=33225ms result1=true ILW = issue in GA build, reproducible with single test , no workaround available = MLM = P4 Moving it to dev team for further analysis
07-12-2022