ADDITIONAL SYSTEM INFORMATION :
OS:
>uname -a
Linux 3.10.0-1160.2.2.el7.x86_64 #1 SMP Sat Oct 17 05:06:47 UTC 2020 x86_64 x86_64 x86_64 GNU/Linux
>hostnamectl
Static hostname: ..................
Icon name: computer-vm
Chassis: vm
Machine ID: ...................
Boot ID: ...................
Virtualization: vmware
Operating System: Red Hat Enterprise Linux
CPE OS Name: cpe:/o:redhat:enterprise_linux:7.9:GA:server
Kernel: Linux 3.10.0-1160.2.2.el7.x86_64
Architecture: x86-64
Java Runtime :
>java -version
openjdk version "1.8.0_272"
OpenJDK Runtime Environment (build 1.8.0_272-b10)
OpenJDK 64-Bit Server VM (build 25.272-b10, mixed mode)
A DESCRIPTION OF THE PROBLEM :
Our applications use JMX connections over TLSv1.2 with authentication enabled and run on RedHat Enterprise v7.x. These applications run on the same linux node and hence share the same Java Runtime Environment.
These applications were running properly on the JDK/JRE to Java 8 build 262.
After upgrading to Java 8 build 272, the JMX interface broke. The SSL server throws Null Pointer Exception during SSL Handshake.
Note: The Java release selection in the "Java Release" drop down does not list Java 8 update 272 and only lists build 271 as the latest update, so 271 is selected. We use build 272.
Following are observations between when application was running successfully (Java 8 build 262) and when it broke (Java 8 build 272):
1. On build 262 the negotiated cipherSuite is "TLS_ECDHE_RSA_WITH_AES_256_CBC_SHA384".
2. On build 272 the negotiated cipherSuite is "TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384".
3. 2. Importantly, on Java 8 build 262 when the JMX Server is restricted to use the given GCM cipher suite (which fails on build 272), it works.
Similar issue was reported on Java 8 through ticket JDK-8238063 and seems to be fixed in build 8u261. We do not see this issue on 8u262, but the issue is visibile on 8u272.
STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
#The following steps are written for Linux environment.
#Create a directory
mkdir -p /tmp/jmx_ssl_test;
#Change the directory to /tmp/jmx_ssl_test;
cd /tmp/jmx_ssl_test;
# Download the following required Bouncy Cycle libraries bcprov-jdk15on-1.60.jar and bcpkix-jdk15on-1.56.jar from the Maven repository
https://repo1.maven.org/maven2/org/bouncycastle/bcprov-jdk15on/1.60/
https://repo1.maven.org/maven2/org/bouncycastle/bcpkix-jdk15on/1.56/
# Copy bcprov-jdk15on-1.60.jar and bcpkix-jdk15on-1.56.jar to the /tmp/jmx_ssl_test directory
# Copy the Main.java and Client.java files to the /tmp/jmx_ssl_test directory.
# Create the java package directory structure
mkdir -p com/example;
# Compile the Main and Client Java classes
javac -cp bcprov-jdk15on-1.60.jar:bcpkix-jdk15on-1.56.jar -d ./ *.java
#Generate Key pair using the following command and entering required values (can pass literal string 'test' name, OU etc.)
keytool -genkeypair -keyalg RSA -alias testkey -keysize 2048 -keypass jmxssltest -validity 180 -keystore test_ks.jks -storepass jmxssltest
#export the cert and when prompted for the keystore password enter jmxssltest as the password
keytool -export -alias testkey -keystore test_ks.jks -file jmxssltest_cert.cer
# import the cert in truststore and enter jmxssltest as the truststore password
keytool -import -trustcacerts -file jmxssltest_cert.cer -keystore test_ts.jks
# Make sure that the keystore, truststore and the bouncy castle jars are in the /tmp/jmx_ssl_test directory.
#Create a new file named as jmxremote.access (using vi or another text editor) and add the following userId and permission and save it.
admin readwrite
#Create a new file named as jmxremote.password (using vi or another text editor) and add the following userId and password and save it.
admin nimda
# Change the permission of the jmxremote.passowrd file to be 0400
chmod 400 jmxremote.password;
#Important: This example uses port 9010 as JMX port.
#If you prefer another port please change the port accordingly (it is passed as command line argument) when invoking Main and Client applications
#Open a new Linux shell/prompt and change directory to /tmp/jmx_ssl_test
cd /tmp/jmx_ssl_test;
# Start the Main (JMX Server) application using the following command:
java -Dcom.sun.management.jmxremote -Dcom.sun.management.jmxremote.port=9010 -Dcom.sun.management.jmxremote.rmi.port=9010 -Dcom.sun.management.jmxremote.authticate=true -Dcom.sun.management.jmxremote.password.file=jmxremote.password -Dcom.sun.management.jmxremote.access.file=jmxremote.access -Dcom.sun.management.jmxremote.ssl=true -Dcom.sun.management.jmxremote.registry.ssl=true -Dcom.sun.management.jmxremote.ssl.enabled.protocols=TLSv1.2 -Djavax.net.ssl.keyStore=./test_ks.jks -Djavax.net.ssl.keyStorePassword=jmxssltest -Djavax.net.ssl.trustStore=./test_ts.jks -Djavax.net.ssl.trustStorePassword=jmxssltest -cp .:bcprov-jdk15on-1.60.jar:bcpkix-jdk15on-1.56.jar: -Djavax.net.debug=ssl,handshake com.example.Main
#Open a new Linux shell/prompt and change directory to /tmp/jmx_ssl_test
cd /tmp/jmx_ssl_test;
# Start the Client (JMX client) application using the following command:
java -Djavax.net.ssl.trustStore=./test_ts.jks -Djavax.net.ssl.trustStorePassword=jmxssltest -Dcomun.management.jmxremote.ssl.enabled.otocols=TLSv1.2 -Djdk.tls.client.protocols="TLSv1.2" -cp .:bcprov-jdk15on-1.60.jar:bcpkix-jdk15on-1.56.jar -Djmx.ssl.test.server.port=9010 -Djavax.net.debug=ssl,handshake com.example.Client
EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
Following are the expected results (as observed when run on Java 8 build 262):
1. The Client application properly connects to the Main (Jmx server) application.
2. The Client application displays the following text:
Create an RMI connector client and connect it to the RMI connector server
Get an MBeanServerConnection
Press <Enter> to continue...
3. When user clicks/presses "Enter" the client shows the Domains from the Mbean server, something similar to the following:
Domains:
Domain = JMImplementation
Domain = com.sun.management
Domain = java.lang
Domain = java.nio
Domain = java.util.logging
Domain = jdk.management.jfr
ACTUAL -
The JMX client receives the following exception during SSL Handshake:
javax.net.ssl|FINE|01|main|2020-11-11 18:15:13.308 EST|Logger.java:765|close the SSL connection (initiative)
Exception in thread "main" java.io.IOException: Failed to retrieve RMIServer stub: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLException: Received fatal alert: internal_error]
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:369)
at javax.management.remote.JMXConnectorFactory.connect(JMXConnectorFactory.java:270)
at com.example.Client.main(Client.java:77)
Caused by: javax.naming.CommunicationException [Root exception is java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLException: Received fatal alert: internal_error]
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:136)
at com.sun.jndi.toolkit.url.GenericURLContext.lookup(GenericURLContext.java:205)
at javax.naming.InitialContext.lookup(InitialContext.java:417)
at javax.management.remote.rmi.RMIConnector.findRMIServerJNDI(RMIConnector.java:1955)
at javax.management.remote.rmi.RMIConnector.findRMIServer(RMIConnector.java:1922)
at javax.management.remote.rmi.RMIConnector.connect(RMIConnector.java:287)
... 2 more
Caused by: java.rmi.ConnectIOException: error during JRMP connection establishment; nested exception is:
javax.net.ssl.SSLException: Received fatal alert: internal_error
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:307)
at sun.rmi.transport.tcp.TCPChannel.newConnection(TCPChannel.java:202)
at sun.rmi.server.UnicastRef.newCall(UnicastRef.java:343)
at sun.rmi.registry.RegistryImpl_Stub.lookup(RegistryImpl_Stub.java:116)
at com.sun.jndi.rmi.registry.RegistryContext.lookup(RegistryContext.java:132)
... 7 more
Caused by: javax.net.ssl.SSLException: Received fatal alert: internal_error
at sun.security.ssl.Alert.createSSLException(Alert.java:133)
at sun.security.ssl.Alert.createSSLException(Alert.java:117)
at sun.security.ssl.TransportContext.fatal(TransportContext.java:311)
at sun.security.ssl.Alert$AlertConsumer.consume(Alert.java:293)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:185)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
at sun.security.ssl.SSLSocketImpl$AppOutputStream.write(SSLSocketImpl.java:961)
at java.io.BufferedOutputStream.flushBuffer(BufferedOutputStream.java:82)
at java.io.BufferedOutputStream.flush(BufferedOutputStream.java:140)
at java.io.DataOutputStream.flush(DataOutputStream.java:123)
at sun.rmi.transport.tcp.TCPChannel.createConnection(TCPChannel.java:229)
... 11 more
While the JMX Server shows the following exception:
javax.net.ssl|SEVERE|0C|RMI TCP Connection(1)-10.22.0.186|2020-11-11 18:15:13.302 EST|Logger.java:765|Fatal (INTERNAL_ERROR): null (
"throwable" : {
java.lang.NullPointerException
at org.bouncycastle.crypto.signers.PSSSigner.generateSignature(Unknown Source)
at org.bouncycastle.jcajce.provider.asymmetric.rsa.PSSSignatureSpi.engineSign(Unknown Source)
at java.security.Signature$Delegate.engineSign(Signature.java:1382)
at java.security.Signature.sign(Signature.java:698)
at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeMessage.<init>(ECDHServerKeyExchange.java:181)
at sun.security.ssl.ECDHServerKeyExchange$ECDHServerKeyExchangeProducer.produce(ECDHServerKeyExchange.java:499)
at sun.security.ssl.ClientHello$T12ClientHelloConsumer.consume(ClientHello.java:1020)
at sun.security.ssl.ClientHello$ClientHelloConsumer.onClientHello(ClientHello.java:727)
at sun.security.ssl.ClientHello$ClientHelloConsumer.consume(ClientHello.java:693)
at sun.security.ssl.SSLHandshake.consume(SSLHandshake.java:377)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:444)
at sun.security.ssl.HandshakeContext.dispatch(HandshakeContext.java:422)
at sun.security.ssl.TransportContext.dispatch(TransportContext.java:182)
at sun.security.ssl.SSLTransport.decode(SSLTransport.java:149)
at sun.security.ssl.SSLSocketImpl.decode(SSLSocketImpl.java:1143)
at sun.security.ssl.SSLSocketImpl.readHandshakeRecord(SSLSocketImpl.java:1054)
at sun.security.ssl.SSLSocketImpl.startHandshake(SSLSocketImpl.java:394)
at sun.security.ssl.SSLSocketImpl.ensureNegotiated(SSLSocketImpl.java:708)
at sun.security.ssl.SSLSocketImpl.access$100(SSLSocketImpl.java:72)
at sun.security.ssl.SSLSocketImpl$AppInputStream.read(SSLSocketImpl.java:791)
at java.io.BufferedInputStream.fill(BufferedInputStream.java:246)
at java.io.BufferedInputStream.read(BufferedInputStream.java:265)
at java.io.DataInputStream.readInt(DataInputStream.java:387)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run0(TCPTransport.java:727)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.lambda$run$0(TCPTransport.java:688)
at java.security.AccessController.doPrivileged(Native Method)
at sun.rmi.transport.tcp.TCPTransport$ConnectionHandler.run(TCPTransport.java:687)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at java.lang.Thread.run(Thread.java:748)}
---------- BEGIN SOURCE ----------
Main.Java class:
/*
* Main.java - main class registers with the MBean Server.
*/
package com.example;
import javax.management.MBeanServer;
import java.lang.management.ManagementFactory;
import java.security.Security;
public class Main {
/* For simplicity, we declare "throws Exception".
Real programs will usually want finer-grained exception handling. */
public static void main(String[] args) throws Exception {
// Security.addProvider(new BouncyCastleProvider());
Security.insertProviderAt(new org.bouncycastle.jce.provider.BouncyCastleProvider(), 2);
// Get the Platform MBean Server
MBeanServer mbs = ManagementFactory.getPlatformMBeanServer();
// Wait forever
System.out.println("Waiting for incoming requests...");
Thread.sleep(Long.MAX_VALUE);
}
}
Client.java class:
/*
* Client.java - JMX client that interacts with the JMX agent.
*/
package com.example;
import javax.management.AttributeChangeNotification;
import javax.management.MBeanServerConnection;
import javax.management.Notification;
import javax.management.NotificationListener;
import javax.management.ObjectName;
import javax.management.remote.JMXConnector;
import javax.management.remote.JMXConnectorFactory;
import javax.management.remote.JMXServiceURL;
import javax.rmi.ssl.SslRMIClientSocketFactory;
import java.io.IOException;
import java.security.Security;
import java.util.Arrays;
import java.util.HashMap;
import java.util.Set;
import java.util.TreeSet;
public class Client {
/**
* Inner class that will handle the notifications.
*/
public static class ClientListener implements NotificationListener {
public void handleNotification(Notification notification,
Object handback) {
echo("\nReceived notification:");
echo("\tClassName: " + notification.getClass().getName());
echo("\tSource: " + notification.getSource());
echo("\tType: " + notification.getType());
echo("\tMessage: " + notification.getMessage());
if (notification instanceof AttributeChangeNotification) {
AttributeChangeNotification acn =
(AttributeChangeNotification) notification;
echo("\tAttributeName: " + acn.getAttributeName());
echo("\tAttributeType: " + acn.getAttributeType());
echo("\tNewValue: " + acn.getNewValue());
echo("\tOldValue: " + acn.getOldValue());
}
}
}
/* For simplicity, we declare "throws Exception".
Real programs will usually want finer-grained exception handling. */
public static void main(String[] args) throws Exception {
//Security.addProvider(new BouncyCastleProvider());
Security.insertProviderAt(new org.bouncycastle.jce.provider.BouncyCastleProvider(), 2);
// Create an RMI connector client and
// connect it to the RMI connector server
//
String jmxServerPort = System.getProperty("jmx.ssl.test.server.port");
if ((jmxServerPort == null) || jmxServerPort.isEmpty()) {
throw new IllegalStateException("System property 'jmx.ssl.test.server.port' must be set (numeric)");
}
echo("\nCreate an RMI connector client and " +
"connect it to the RMI connector server");
JMXServiceURL url =
//new JMXServiceURL("service:jmx:rmi:///jndi/rmi://:9010/jmxrmi");
new JMXServiceURL("service:jmx:rmi:///jndi/rmi://:" + jmxServerPort + "/jmxrmi");
HashMap environment = new HashMap();
String[] credentials = new String[]
{
"admin", "nimda"
};
environment.put(JMXConnector.CREDENTIALS, credentials);
environment.put("com.sun.jndi.rmi.factory.socket", new SslRMIClientSocketFactory());
JMXConnector jmxc = JMXConnectorFactory.connect(url, environment);
// Create listener
//
ClientListener listener = new ClientListener();
// Get an MBeanServerConnection
//
echo("\nGet an MBeanServerConnection");
MBeanServerConnection mbsc = jmxc.getMBeanServerConnection();
waitForEnterPressed();
// Get domains from MBeanServer
//
echo("\nDomains:");
String domains[] = mbsc.getDomains();
Arrays.sort(domains);
for (String domain : domains) {
echo("\tDomain = " + domain);
}
waitForEnterPressed();
// Get MBeanServer's default domain
//
echo("\nMBeanServer default domain = " + mbsc.getDefaultDomain());
// Get MBean count
//
echo("\nMBean count = " + mbsc.getMBeanCount());
// Query MBean names
//
echo("\nQuery MBeanServer MBeans:");
Set<ObjectName> names =
new TreeSet<ObjectName>(mbsc.queryNames(null, null));
for (ObjectName name : names) {
echo("\tObjectName = " + name);
}
waitForEnterPressed();
// Close MBeanServer connection
//
echo("\nClose the connection to the server");
jmxc.close();
echo("\nBye! Bye!");
}
private static void echo(String msg) {
System.out.println(msg);
}
private static void waitForEnterPressed() {
try {
echo("\nPress <Enter> to continue...");
System.in.read();
} catch (IOException e) {
e.printStackTrace();
}
}
}
---------- END SOURCE ----------
CUSTOMER SUBMITTED WORKAROUND :
None found. We had to revert the Java Runtime Environment to Java 8 build 262.