ADDITIONAL SYSTEM INFORMATION :
$ cat /etc/lsb-release
DISTRIB_ID=Ubuntu
DISTRIB_RELEASE=18.04
DISTRIB_CODENAME=bionic
DISTRIB_DESCRIPTION="Ubuntu 18.04.4 LTS"
$ java -version
openjdk version "1.8.0_242"
OpenJDK Runtime Environment (build 1.8.0_242-8u242-b08-0ubuntu3~18.04-b08)
OpenJDK 64-Bit Server VM (build 25.242-b08, mixed mode)
A DESCRIPTION OF THE PROBLEM :
Up until recently we've used a JAAS configuration to allow Kafka to authenticate against a Kerberos secured Zookeeper server using this JAAS config:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/fqdn.redacted@EXAMPLE.COM";
};
With OpenJDK 8 Update 242 (in Ubuntu Bionic: 8u242-b08-0ubuntu3~18.04) this configuration fails to work in our environment under the following conditions:
- In the JAAS config, the service principal is referenced to using its AD servicePrincipalName kafka/fqdn.redacted.
- The sAMAccountName differs, it is actually $48M300-TK9QSTNPCT80.
- Kafka gets itself a Kerberos ticket using the JAAS config above.
Turns out the principal is authenticated as $48M300-TK9QSTNPCT80@EXAMPLE.COM instead of kafka/fqdn.redacted@EXAMPLE.COM like it was before Update 242. Krb5LoginModule seemed to pick "client_alias" before and "client" after the upgrade (see debug output in section "Actual Result" of this bug report.
However, the problem does *not* appear when JAAS is configured to use a Kerberos ticket in the ticket cache for the same principal using this JAAS config:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=false
useTicketCache=true;
};
Here, the principal is still authenticated correctly as kafka/fqdn.redacted@EXAMPLE.COM.
REGRESSION : Last worked in version 8
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
- In AD there is a service principal with servicePrincipalName=kafka/fqdn.redacted and sAMAccountName=$48M300-TK9QSTNPCT80
- JAAS config is as follows:
Client {
com.sun.security.auth.module.Krb5LoginModule required
useKeyTab=true
keyTab="/etc/security/keytabs/kafka.service.keytab"
storeKey=true
useTicketCache=false
serviceName="zookeeper"
principal="kafka/fqdn.redacted@EXAMPLE.COM";
};
- Zookeeper client connects to Zookeeper the JAAS config above: KAFKA_OPTS="-Djava.security.auth.login.config=/opt/kafka/kafka/config/kafka_jaas.conf -Dsun.security.krb5.debug=true" bin/zookeeper-shell.sh fqdn.redacted:2181
- As I'm no developer I'm sorry for not being able to present steps to reproduce the bug in a non-Kafka-Environment, i.e. a minimal working example.
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Zookeeper authenticates kafka/fqdn.redacted@EXAMPLE.COM as it was before the OpenJDK 8 Update 242 upgrade.
ACTUAL -
Zookeeper authenticates $48M300-TK9QSTNPCT80@EXAMPLE.COM
See this log:
Connecting to fqdn.redacted:2181
Welcome to ZooKeeper!
JLine support is disabled
>>> KeyTabInputStream, readName(): EXAMPLE.COM
>>> KeyTabInputStream, readName(): kafka
>>> KeyTabInputStream, readName(): fqdn.redacted
>>> KeyTab: load() entry length: 99; type: 18
Looking for keys for: kafka/fqdn.redacted@EXAMPLE.COM
Java config name: null
Native config name: /etc/krb5.conf
Loaded from native config
Added key: 18version: 1
>>> KdcAccessibility: reset
Looking for keys for: kafka/fqdn.redacted@EXAMPLE.COM
Added key: 18version: 1
default etypes for default_tkt_enctypes: 18 17 16 23 18 17 16 23.
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=example.com UDP:88, timeout=30000, number of retries =3, #bytes=197
>>> KDCCommunication: kdc=example.com UDP:88, timeout=30000,Attempt =1, #bytes=197
>>> KrbKdcReq send: #bytes read=195
>>>Pre-Authentication Data:
PA-DATA type = 19
PA-ETYPE-INFO2 etype = 18, salt = EXAMPLE.COMkafkafqdn.redacted, s2kparams = null
>>>Pre-Authentication Data:
PA-DATA type = 2
PA-ENC-TIMESTAMP
>>>Pre-Authentication Data:
PA-DATA type = 16
>>>Pre-Authentication Data:
PA-DATA type = 15
>>> KdcAccessibility: remove example.com
>>> KDCRep: init() encoding tag is 126 req type is 11
>>>KRBError:
sTime is Fri Jan 31 11:03:30 CET 2020 1580465010000
suSec is 347467
error code is 25
error Message is Additional pre-authentication required
sname is krbtgt/EXAMPLE.COM@EXAMPLE.COM
eData provided.
msgType is 30
>>>Pre-Authentication Data:
PA-DATA type = 19
PA-ETYPE-INFO2 etype = 18, salt = EXAMPLE.COMkafkafqdn.redacted, s2kparams = null
>>>Pre-Authentication Data:
PA-DATA type = 2
PA-ENC-TIMESTAMP
>>>Pre-Authentication Data:
PA-DATA type = 16
>>>Pre-Authentication Data:
PA-DATA type = 15
KrbAsReqBuilder: PREAUTH FAILED/REQ, re-send AS-REQ
default etypes for default_tkt_enctypes: 18 17 16 23 18 17 16 23.
Looking for keys for: kafka/fqdn.redacted@EXAMPLE.COM
Added key: 18version: 1
Looking for keys for: kafka/fqdn.redacted@EXAMPLE.COM
Added key: 18version: 1
default etypes for default_tkt_enctypes: 18 17 16 23 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbAsReq creating message
>>> KrbKdcReq send: kdc=example.com UDP:88, timeout=30000, number of retries =3, #bytes=282
>>> KDCCommunication: kdc=example.com UDP:88, timeout=30000,Attempt =1, #bytes=282
>>> KrbKdcReq send: #bytes read=88
>>> KrbKdcReq send: kdc=example.com TCP:88, timeout=30000, number of retries =3, #bytes=282
>>> KDCCommunication: kdc=example.com TCP:88, timeout=30000,Attempt =1, #bytes=282
>>>DEBUG: TCPClient reading 1617 bytes
>>> KrbKdcReq send: #bytes read=1617
>>> KdcAccessibility: remove example.com
Looking for keys for: kafka/fqdn.redacted@EXAMPLE.COM
Added key: 18version: 1
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> KrbAsRep cons in KrbAsReq.getReply kafka/fqdn.redacted
WATCHER::
WatchedEvent state:SyncConnected type:None path:null
Found ticket for $48M300-TK9QSTNPCT80@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Fri Jan 31 21:03:30 CET 2020
Entered Krb5Context.initSecContext with state=STATE_NEW
Found ticket for $48M300-TK9QSTNPCT80@EXAMPLE.COM to go to krbtgt/EXAMPLE.COM@EXAMPLE.COM expiring on Fri Jan 31 21:03:30 CET 2020
Service ticket not found in the subject
>>> Credentials serviceCredsSingle: same realm
default etypes for default_tgs_enctypes: 18 17 16 23 18 17 16 23.
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> CksumType: sun.security.krb5.internal.crypto.HmacSha1Aes256CksumType
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> KrbKdcReq send: kdc=example.com TCP:88, timeout=30000, number of retries =3, #bytes=1494
>>> KDCCommunication: kdc=example.com TCP:88, timeout=30000,Attempt =1, #bytes=1494
>>>DEBUG: TCPClient reading 1496 bytes
>>> KrbKdcReq send: #bytes read=1496
>>> KdcAccessibility: remove example.com
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
>>> TGS credentials serviceCredsSingle:
>>> DEBUG: ----Credentials----
client: $48M300-TK9QSTNPCT80@EXAMPLE.COM
client alias: kafka/fqdn.redacted@EXAMPLE.COM
server: zookeeper/fqdn.redacted@EXAMPLE.COM
ticket: sname: zookeeper/fqdn.redacted@EXAMPLE.COM
startTime: 1580465010000
endTime: 1580501010000
----Credentials end----
>>> KrbApReq: APOptions are 00000000 00000000 00000000 00000000
>>> EType: sun.security.krb5.internal.crypto.Aes256CtsHmacSha1EType
Krb5Context setting mySeqNumber to: 94571730
Krb5Context setting peerSeqNumber to: 94571730
---------- BEGIN SOURCE ----------
unfortunately none
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
There are two workarounds:
1. Changing the JAAS config to use a ticket in the ticket cache created upfront using kinit. However this workaround can't be used in automated environments where clients authenticate themselves using keytabs.
2. Downgrade from OpenJDK 8 Update 242 to Update 232. Using the same configuration, the principal is authenticated as kafka/fqdn.redacted@EXAMPLE.COM. However, downgrading a package can only be a temporary solution.
FREQUENCY : always