JDK-8179389 : X509Certificate generateCRLs is extremely slow using a PEM crl list
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 8,9
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • OS: linux
  • CPU: x86_64
  • Submitted: 2017-04-26
  • Updated: 2017-05-23
  • Resolved: 2017-05-10
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 10
10 b08Fixed
Description
FULL PRODUCT VERSION :
java -version
openjdk version "1.8.0_121"
OpenJDK Runtime Environment (build 1.8.0_121-b14)
OpenJDK 64-Bit Server VM (build 25.121-b14, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux 4.10.9-200.fc25.x86_64 #1 SMP Mon Apr 10 14:48:16 UTC 2017 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Loading a PEM CRL list with 110 CRLs and 200MB in size is 25 minutes in my laptop. Using a DER list is 12 seconds.

The main reason is the X509Factory.java class uses a buffer of 2048 to store the base64 data and if it needs more it adds chucks by 1024 each (using Arrays.copyOf). That means to create a 10MB pem it calls to Arrays.copyOf 10000 times (allocate and copy). That's the reason to be painfully slow.

See here:
http://hg.openjdk.java.net/jdk8/jdk8/jdk/file/687fd7c7986d/src/share/classes/sun/security/provider/X509Factory.java#l482



STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Just download a big CRL, convert to PEM and try the generateCRLs method. See the difference between PEM and DER.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Load the PEM crl in a reasonable time. My patch below does the load of 200MB in 83s and a 33MB CRL in 14s.
ACTUAL -
A 33MB CRL is 6 minutes in time to be loaded. A 200MB file with several CRLs is 25 minutes.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
import java.io.File;
import java.io.FileInputStream;
import java.util.Collection;
import java.security.cert.Certificate;
import java.security.cert.CertificateFactory;

public class LoadCerts {

    public static void main(String[] args) throws Exception {
        if (args.length != 1) {
            throw new IllegalArgumentException("The first argument should be the PEM file.");
        }
        File f = new File(args[0]);
        if (!f.exists() || !f.canRead()) {
            throw new IllegalArgumentException(String.format("Invalid file %s", args[0]));
        }
        try (FileInputStream is = new FileInputStream(f)) {
            CertificateFactory cf = CertificateFactory.getInstance("X.509");
            long start = System.currentTimeMillis();
            Collection crls = cf.generateCRLs(is);
            long end = System.currentTimeMillis();
            System.out.println(String.format("Loaded %s certificates from %s in %d seconds.", crls.size(), args[0], (end - start) / 1000L));
        }
    }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
Consider this patch, it uses the new java8 Base64.Decoder to just use an InputStream over the Base64 (avoiding intermediary buffers). It loads a 33MB crl in 14s instead of 6 minutes.

--- /home/rmartinc/jdk8/jdk/src/share/classes/sun/security/provider/X509Factory.java	2017-04-26 09:23:14.807876940 +0200
+++ X509Factory.java	2017-04-26 13:28:51.687191779 +0200
@@ -546,22 +546,17 @@
             }
 
             // Step 3: Read the data
-            while (true) {
-                int next = is.read();
-                if (next == -1) {
-                    throw new IOException("Incomplete data");
-                }
-                if (next != '-') {
-                    data[pos++] = (char)next;
-                    if (pos >= data.length) {
-                        data = Arrays.copyOf(data, data.length+1024);
-                    }
-                } else {
-                    break;
-                }
-            }
+            InputStream b64is = Base64.getMimeDecoder().wrap(is);
+            ByteArrayOutputStream bout = new ByteArrayOutputStream(2048);
+            c = b64is.read();
+            bout.write(c);
+            readBERInternal(b64is, bout, c);
 
             // Step 4: Consume the footer
+            c = is.read();
+            while (c != '-' && c != -1) {
+	        c = is.read();
+            }
             StringBuffer footer = new StringBuffer("-");
             while (true) {
                 int next = is.read();
@@ -575,7 +570,7 @@
 
             checkHeaderFooter(header.toString(), footer.toString());
 
-            return Base64.getMimeDecoder().decode(new String(data, 0, pos));
+            return bout.toByteArray();
         }
     }