JDK-8223061 : Missing Read Timeout for CRL load
  • Type: Bug
  • Component: security-libs
  • Sub-Component: java.security
  • Affected Version: 8,11,12,13
  • Priority: P4
  • Status: Closed
  • Resolution: Duplicate
  • OS: linux_ubuntu
  • CPU: x86_64
  • Submitted: 2019-04-25
  • Updated: 2019-05-08
  • Resolved: 2019-04-29
Related Reports
Duplicate :  
Description
ADDITIONAL SYSTEM INFORMATION :
Sample stuck thread (from thread dump):

Thread 73765: (state = IN_NATIVE)
 - java.net.SocketInputStream.socketRead0(java.io.FileDescriptor, byte[], int, int, int) @bci=0 (Compiled frame; information may be imprecise)
 - java.net.SocketInputStream.socketRead(java.io.FileDescriptor, byte[], int, int, int) @bci=8, line=116 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int, int) @bci=79, line=170 (Compiled frame)
 - java.net.SocketInputStream.read(byte[], int, int) @bci=11, line=141 (Compiled frame)
 - java.io.BufferedInputStream.fill() @bci=214, line=246 (Compiled frame)
 - java.io.BufferedInputStream.read1(byte[], int, int) @bci=44, line=286 (Compiled frame)
 - java.io.BufferedInputStream.read(byte[], int, int) @bci=49, line=345 (Compiled frame)
 - sun.net.www.http.HttpClient.parseHTTPHeader(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) @bci=51, line=704 (Interpreted frame)
 - sun.net.www.http.HttpClient.parseHTTP(sun.net.www.MessageHeader, sun.net.ProgressSource, sun.net.www.protocol.http.HttpURLConnection) @bci=56, line=647 (Interpreted frame)
 - sun.net.www.protocol.http.HttpURLConnection.getInputStream0() @bci=327, line=1569 (Interpreted frame)
 - sun.net.www.protocol.http.HttpURLConnection.getInputStream() @bci=52, line=1474 (Interpreted frame)
 - sun.security.provider.certpath.URICertStore.engineGetCRLs(java.security.cert.CRLSelector) @bci=151, line=396 (Interpreted frame)
 - java.security.cert.CertStore.getCRLs(java.security.cert.CRLSelector) @bci=5, line=181 (Interpreted frame)
 - sun.security.provider.certpath.DistributionPointFetcher.getCRL(sun.security.x509.URIName) @bci=92, line=245 (Interpreted frame)
 - sun.security.provider.certpath.DistributionPointFetcher.getCRLs(java.security.cert.X509CRLSelector, sun.security.x509.X509CertImpl, sun.security.x509.DistributionPoint, boolean[], boolean, java.security.PublicKey, java.security.cert.X509Certificate, java.lang.String, java.util.List, java.util.Set, java.util.Date) @bci=199, line=189 (Interpreted frame)
 - sun.security.provider.certpath.DistributionPointFetcher.getCRLs(java.security.cert.X509CRLSelector, boolean, java.security.PublicKey, java.security.cert.X509Certificate, java.lang.String, java.util.List, boolean[], java.util.Set, java.util.Date) @bci=165, line=121 (Interpreted frame)
 - sun.security.provider.certpath.RevocationChecker.checkCRLs(java.security.cert.X509Certificate, java.security.PublicKey, java.security.cert.X509Certificate, boolean, boolean, java.util.Set, java.util.Set) @bci=449, line=552 (Interpreted frame)
 - sun.security.provider.certpath.RevocationChecker.checkCRLs(java.security.cert.X509Certificate, java.util.Collection, java.util.Set, java.security.PublicKey, boolean) @bci=16, line=465 (Interpreted frame)
 - sun.security.provider.certpath.RevocationChecker.check(java.security.cert.X509Certificate, java.util.Collection, java.security.PublicKey, boolean) @bci=152, line=367 (Interpreted frame)
 - sun.security.provider.certpath.RevocationChecker.check(java.security.cert.Certificate, java.util.Collection) @bci=14, line=337 (Interpreted frame)
 - sun.security.provider.certpath.PKIXMasterCertPathValidator.validate(java.security.cert.CertPath, java.util.List, java.util.List) @bci=316, line=125 (Interpreted frame)
 - sun.security.provider.certpath.PKIXCertPathValidator.validate(java.security.cert.TrustAnchor, sun.security.provider.certpath.PKIX$ValidatorParams) @bci=333, line=219 (Interpreted frame)
 - sun.security.provider.certpath.PKIXCertPathValidator.validate(sun.security.provider.certpath.PKIX$ValidatorParams) @bci=217, line=140 (Interpreted frame)
 - sun.security.provider.certpath.PKIXCertPathValidator.engineValidate(java.security.cert.CertPath, java.security.cert.CertPathParameters) @bci=7, line=79 (Interpreted frame)
 - java.security.cert.CertPathValidator.validate(java.security.cert.CertPath, java.security.cert.CertPathParameters) @bci=6, line=292 (Interpreted frame)
 - sun.security.validator.PKIXValidator.doValidate(java.security.cert.X509Certificate[], java.security.cert.PKIXBuilderParameters) @bci=34, line=347 (Interpreted frame)
 - sun.security.validator.PKIXValidator.engineValidate(java.security.cert.X509Certificate[], java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object) @bci=278, line=260 (Interpreted frame)
 - sun.security.validator.Validator.validate(java.security.cert.X509Certificate[], java.util.Collection, java.security.AlgorithmConstraints, java.lang.Object) @bci=6, line=260 (Interpreted frame)
 - sun.security.ssl.X509TrustManagerImpl.validate(sun.security.validator.Validator, java.security.cert.X509Certificate[], java.security.AlgorithmConstraints, java.lang.String) @bci=10, line=324 (Interpreted frame)
 - sun.security.ssl.X509TrustManagerImpl.checkTrusted(java.security.cert.X509Certificate[], java.lang.String, java.net.Socket, boolean) @bci=218, line=229 (Interpreted frame)
 - sun.security.ssl.X509TrustManagerImpl.checkServerTrusted(java.security.cert.X509Certificate[], java.lang.String, java.net.Socket) @bci=5, line=124 (Interpreted frame)
 - sun.security.ssl.ClientHandshaker.serverCertificate(sun.security.ssl.HandshakeMessage$CertificateMsg) @bci=163, line=1491 (Interpreted frame)
 - sun.security.ssl.ClientHandshaker.processMessage(byte, int) @bci=237, line=216 (Interpreted frame)
 - sun.security.ssl.Handshaker.processLoop() @bci=96, line=979 (Compiled frame)
 - sun.security.ssl.Handshaker.process_record(sun.security.ssl.InputRecord, boolean) @bci=24, line=914 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.readRecord(sun.security.ssl.InputRecord, boolean) @bci=357, line=1062 (Compiled frame)
 - sun.security.ssl.SSLSocketImpl.performInitialHandshake() @bci=84, line=1375 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.startHandshake(boolean) @bci=13, line=1403 (Interpreted frame)
 - sun.security.ssl.SSLSocketImpl.startHandshake() @bci=2, line=1387 (Interpreted frame)
 - org.apache.http.conn.ssl.SSLConnectionSocketFactory.createLayeredSocket(java.net.Socket, java.lang.String, int, org.apache.http.protocol.HttpContext) @bci=258, line=396 (Interpreted frame)
 - org.apache.http.conn.ssl.SSLConnectionSocketFactory.connectSocket(int, java.net.Socket, org.apache.http.HttpHost, java.net.InetSocketAddress, java.net.InetSocketAddress, org.apache.http.protocol.HttpContext) @bci=195, line=355 (Interpreted frame)



A DESCRIPTION OF THE PROBLEM :
When JVM is run with the flags:
-Dcom.sun.security.enableCRLDP=true -Dcom.sun.net.ssl.checkRevocation=true 

To enable CRL verification, CRL request is being issued without any read-timeout. Implication of this is that given some network condition / bad backend behaviour, network connections might be blocked indifinitely.

We've seen that happen in production systems with CRL request being blocked indefinitely against http://crl3.digicert.com/DigiCertGlobalRootCA.crl

Offending code can be found in URICertStore::engineGetCRLs:
            URLConnection connection = uri.toURL().openConnection();
            if (lastModified != 0) {
                connection.setIfModifiedSince(lastModified);
            }
            long oldLastModified = lastModified;
            connection.setConnectTimeout(CRL_CONNECT_TIMEOUT);
>> missing setReadTimeout() <<
            try (InputStream in = connection.getInputStream()) {

Would probably make sense to use the same connect timeout also as read timeout to ensure read requests always return at some point.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Given production system doing ALOT of secure connections requiring CRL verifications, under certain network / CRL-backend system condition one would see such threads get blocked.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Such network request fails on timeout after some max amount of time (can probably use the same CONNECT timeout)
ACTUAL -
Thread/System might get stuck indefinitely.

CUSTOMER SUBMITTED WORKAROUND :
None.

FREQUENCY : rarely