JDK-8273894 : ConcurrentModificationException raised every time ReferralsCache drops referral
  • Type: Bug
  • Component: security-libs
  • Sub-Component: org.ietf.jgss:krb5
  • Affected Version: 8,11,17,18
  • Priority: P4
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2021-09-14
  • Updated: 2022-12-12
  • Resolved: 2021-09-23
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 11 JDK 13 JDK 15 JDK 17 JDK 18
11.0.14Fixed 13.0.14Fixed 15.0.10Fixed 17.0.2Fixed 18 b17Fixed
Related Reports
Relates :  
JDK-8215032 - Support Kerberos cross-realm referrals (RFC 6806)
Description
A DESCRIPTION OF THE PROBLEM :
ConcurrentModificationException raised every time ReferralsCache drops referral. Method pruneExpired always raises exception is a referral TGT has expired.

Example trace:
Caused by: java.util.ConcurrentModificationException
	at java.util.HashMap$HashIterator.nextNode(HashMap.java:1493) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1526) ~[?:?]
	at java.util.HashMap$EntryIterator.next(HashMap.java:1524) ~[?:?]
	at sun.security.krb5.internal.ReferralsCache.pruneExpired(ReferralsCache.java:152) ~[java.security.jgss:?]
	at sun.security.krb5.internal.ReferralsCache.get(ReferralsCache.java:133) ~[java.security.jgss:?]
	at sun.security.krb5.internal.CredentialsUtil.serviceCredsReferrals(CredentialsUtil.java:365) ~[java.security.jgss:?]
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:333) ~[java.security.jgss:?]
	at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314) ~[java.security.jgss:?]
Comments
A pull request was submitted for review. URL: https://git.openjdk.org/jdk13u-dev/pull/442 Date: 2022-12-12 11:29:54 +0000
12-12-2022
A pull request was submitted for review. URL: https://git.openjdk.org/jdk15u-dev/pull/315 Date: 2022-12-12 11:01:29 +0000
12-12-2022
Fix request (13u, 15u) backporting for parity with LTS releases. Clean backport, security tests do pass (all but one failing because of JDK-8296485)
12-12-2022
verified
25-04-2022
Requested the submitter verify the fix by downloading the latest version of JDK 18 from https://jdk.java.net/18/
10-10-2021
Fix Request (11u) Same reason as for 17u.
01-10-2021
Fix Request (17u) Fixes a bug in ReferralsCache. Seen in production by the original reporter. Patch applies cleanly, jdk_security tests pass.
01-10-2021
Changeset: 2166ed13 Author: Jaikiran Pai <jpai@openjdk.org> Date: 2021-09-23 05:37:19 +0000 URL: https://git.openjdk.java.net/jdk/commit/2166ed136917bb68f8155a25e4f4a6c5c7115566
23-09-2021
The source code caused the problem can be found at: https://hg.openjdk.java.net/jdk8u/jdk8u/jdk/file/ece8c902f3e0/src/share/classes/sun/security/krb5/internal/ReferralsCache.java#l151 https://hg.openjdk.java.net/jdk/jdk/file/643978a35f6e/src/java.security.jgss/share/classes/sun/security/krb5/internal/ReferralsCache.java#l151 https://github.com/openjdk/jdk/blob/master/src/java.security.jgss/share/classes/sun/security/krb5/internal/ReferralsCache.java#L177
16-09-2021
Additional information from the submitter: I am not able to reproduce it in our testing environment. The issue happens all the time in one of our customer's environments. They have a multi-domain environment where kerberos authentication stops working after 10 hours of running the server. They have provided the exception trace where you can see that the exception is raised in method ReferralsCache.pruneExpired. This class seems to cache TGTs when a referral is transitioned during the kerberos negotiation. Once the TGT expires, it will try to remove it from the cache, raising the exception. Caused by: java.util.ConcurrentModificationException at java.util.HashMap$HashIterator.nextNode(HashMap.java:1493) ~[?:?] at java.util.HashMap$EntryIterator.next(HashMap.java:1526) ~[?:?] at java.util.HashMap$EntryIterator.next(HashMap.java:1524) ~[?:?] at sun.security.krb5.internal.ReferralsCache.pruneExpired(ReferralsCache.java:152) ~[java.security.jgss:?] at sun.security.krb5.internal.ReferralsCache.get(ReferralsCache.java:133) ~[java.security.jgss:?] at sun.security.krb5.internal.CredentialsUtil.serviceCredsReferrals(CredentialsUtil.java:365) ~[java.security.jgss:?] at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:333) ~[java.security.jgss:?] at sun.security.krb5.internal.CredentialsUtil.serviceCreds(CredentialsUtil.java:314) ~[java.security.jgss:?] at sun.security.krb5.internal.CredentialsUtil.acquireServiceCreds(CredentialsUtil.java:169) ~[java.security.jgss:?] at sun.security.krb5.Credentials.acquireServiceCreds(Credentials.java:490) ~[java.security.jgss:?] at sun.security.jgss.krb5.Krb5Context.initSecContext(Krb5Context.java:697) ~[java.security.jgss:?] Then I looked at this method implementation, on JDK 11.0.12.7.hotspot from AdoptOpenJDK and I saw this: for (Entry<String, ReferralCacheEntry> mapEntry : entries.entrySet()) { if (mapEntry.getValue().getCreds().getEndTime().before(now)) { entries.remove(mapEntry.getKey()); } } If execution enters the IF statement, the remove call will always throw ConcurrentModificationException. The correct way would be to obtain an iterator and call the iterator's remove method.
16-09-2021
Requested a reproducer from the submitter.
16-09-2021