JDK-8173926 : ServerSocketChannel.bind(address) throw exception "java.net.SocketException: Operation not supported on transport endpoint" intermittently on solaris
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio
  • Affected Version: 9,10,12
  • Priority: P4
  • Status: Open
  • Resolution: Unresolved
  • OS: solaris
  • Submitted: 2017-02-04
  • Updated: 2020-04-25
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Relates :  
Description
This issue is exposed by JDK-8169363.
The issue should be a production issue or system issue on Solaris.
The issue was reproduced on Oracle.Solaris.11.2.X86:

The java code to reproduce the issue is:


public class ServerSocketChannelTestInfinite {
    
    static private ServerSocketChannel channel;
    
    public static void main(String args[]) throws Exception {

        for (int i = 0; i < 100000; i++) {
            // Create and bind a new server socket channel
            channel = ServerSocketChannel.open();
    
            // Enable SO_REUSEADDR before binding
            channel.setOption(SO_REUSEADDR, true);
    
            // Enable SO_REUSEPORT, if supported, before binding
            if (channel.supportedOptions().contains(SO_REUSEPORT)) {
                channel.setOption(SO_REUSEPORT, true);
            }

            try {    
                channel.bind(new InetSocketAddress(0));
            } catch (Exception ex) {
                System.err.println("loop, " + i);
                throw ex;
            }
            Thread.sleep(3);
            channel.close();
        }

    }
}


stack trace as below:
$ java ServerSocketChannelTestInfinite
loop, 8290
Exception in thread "main" java.net.SocketException: Operation not supported on transport endpoint
        at java.base/sun.nio.ch.Net.listen(Native Method)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:226)
        at java.base/java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:157)
        at ServerSocketChannelTestInfinite.main(ServerSocketChannelTestInfinite.java:39)

loop, 1143
Exception in thread "main" java.net.SocketException: Operation not supported on transport endpoint
        at java.base/sun.nio.ch.Net.listen(Native Method)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:226)
        at java.base/java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:157)
        at ServerSocketChannelTestInfinite.main(ServerSocketChannelTestInfinite.java:39)


loop, 14039
Exception in thread "main" java.net.SocketException: Operation not supported on transport endpoint
        at java.base/sun.nio.ch.Net.listen(Native Method)
        at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:226)
        at java.base/java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:157)
        at ServerSocketChannelTestInfinite.main(ServerSocketChannelTestInfinite.java:39)

Comments
This is most likely a Solaris bug, it's not clear that we can do anything in JDK to workaround this.
23-08-2017

Observed on Mach 5 system 9-dev build: http://java.se.oracle.com/mach5/view/JDK%209/view/9-dev/job/9-dev/2198/artifact/logs/tier3-solaris-x64.log/*view*/ Log snippet: RMID: Starting rmid on port 0, at 02:13:48.194316 JAVAVM: command = [/scratch/jenkins/workspace/9-dev/build/jdk/bin/java, -ea, -esa, -Xmx512m, -Djava.security.policy=/scratch/jenkins/workspace/9-dev/jdk/test/java/rmi/activation/ActivationSystem/activeGroup/rmid.security.policy, -Dsun.rmi.server.activation.debugExec=true, -Dtest.src=/scratch/jenkins/workspace/9-dev/jdk/test/java/rmi/activation/ActivationSystem/activeGroup, -Dtest.classes=/scratch/jenkins/workspace/9-dev/testresults/solaris-x64/jdk_test.tier3/jtreg/JTwork/classes/5/java/rmi/activation/ActivationSystem/activeGroup, -Djava.rmi.server.logLevel=v, -Dsun.rmi.activation.execTimeout=60000, -Dsun.rmi.transport.tcp.handshakeTimeout=5000, -classpath, /scratch/jenkins/workspace/9-dev/testresults/solaris-x64/jdk_test.tier3/jtreg/JTwork/classes/5/java/rmi/activation/ActivationSystem/activeGroup:/scratch/jenkins/workspace/9-dev/testresults/solaris-x64/jdk_test.tier3/jtreg/JTwork/classes/5/java/rmi/testlibrary, --add-exports=java.base/sun.nio.ch=ALL-UNNAMED, -Djava.nio.channels.spi.SelectorProvider=RMIDSelectorProvider, -Dtest.java.rmi.testlibrary.RMIDSelectorProvider.port=0, -Dtest.java.rmi.testlibrary.RMIDSelectorProvider.timeout=0, -Dsun.rmi.server.activation.disableErrRedirect=true, sun.rmi.server.Activation, -log, /scratch/jenkins/workspace/9-dev/testresults/solaris-x64/jdk_test.tier3/jtreg/JTwork/scratch/5/./log, -C-Dtest.src=/scratch/jenkins/workspace/9-dev/jdk/test/java/rmi/activation/ActivationSystem/activeGroup, -C-Dtest.classes=/scratch/jenkins/workspace/9-dev/testresults/solaris-x64/jdk_test.tier3/jtreg/JTwork/classes/5/java/rmi/activation/ActivationSystem/activeGroup, -C-ea, -C-esa, -C-Xmx512m, -C-Djava.rmi.server.useCodebaseOnly=false] Activation.main: an exception occurred: java.security.PrivilegedActionException: java.net.SocketException: Operation not supported on transport endpoint java.security.PrivilegedActionException: java.net.SocketException: Operation not supported on transport endpoint at java.base/java.security.AccessController.doPrivileged(Native Method) at java.rmi/sun.rmi.server.Activation.main(Activation.java:1957) Caused by: java.net.SocketException: Operation not supported on transport endpoint at java.base/sun.nio.ch.Net.listen(Native Method) at java.base/sun.nio.ch.ServerSocketChannelImpl.bind(ServerSocketChannelImpl.java:226) at java.base/java.nio.channels.ServerSocketChannel.bind(ServerSocketChannel.java:157) at RMIDSelectorProvider.inheritedChannel(RMIDSelectorProvider.java:113) at java.base/java.lang.System.inheritedChannel(System.java:277) at java.rmi/sun.rmi.server.Activation$2.run(Activation.java:1960) at java.rmi/sun.rmi.server.Activation$2.run(Activation.java:1958) ... 2 more RMID: Started rmid on port 0, at 02:13:48.589649 TEST FAILED: Rmid process exited with status 1 after 102ms. TEST FAILED: Test failed with: TEST FAILED: Rmid process exited with status 1 after 102ms. TestFailedException: TEST FAILED: Rmid process exited with status 1 after 102ms. at TestLibrary.bomb(TestLibrary.java:121) at TestLibrary.bomb(TestLibrary.java:124) at RMID.start(RMID.java:366) at RMID.start(RMID.java:325) at IdempotentActiveGroup.main(IdempotentActiveGroup.java:68) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:563) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110) at java.base/java.lang.Thread.run(Thread.java:844) RMID: shutdown() RMID: lookupSystem() returned null after 155ms. RMID: Destroying RMID process. RMID: Destroy successful after 159ms. TestFailedException: TEST FAILED: ; nested exception is: TestFailedException: TEST FAILED: Rmid process exited with status 1 after 102ms. TestFailedException: TEST FAILED: Rmid process exited with status 1 after 102ms. at TestLibrary.bomb(TestLibrary.java:121) at TestLibrary.bomb(TestLibrary.java:124) at RMID.start(RMID.java:366) at RMID.start(RMID.java:325) at IdempotentActiveGroup.main(IdempotentActiveGroup.java:68) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method) at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62) at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43) at java.base/java.lang.reflect.Method.invoke(Method.java:563) at com.sun.javatest.regtest.agent.MainWrapper$MainThread.run(MainWrapper.java:110) at java.base/java.lang.Thread.run(Thread.java:844) JavaTest Message: Test threw exception: TestFailedException: TEST FAILED: ; nested exception is: TestFailedException: TEST FAILED: Rmid process exited with status 1 after 102ms. JavaTest Message: shutting down test STATUS:Failed.`main' threw exception: TestFailedException: TEST FAILED: ; nested exception is: TestFailedException: TEST FAILED: Rmid process exited with status 1 after 102ms.
04-04-2017

just finished 50 rounds of the test, all passed. I did not change configuration.
08-02-2017

This is almost certainly a Solaris configuration issue. Nothing jumps out from the netstat logs, but considering there is only one socket open at a time, that might be expected. One thing worth trying could be to use ndd to change the ephemeral port range. If the range is reduced and the problem occurs more frequently or sooner then that could be the source of the problem.
07-02-2017

netstat.log.s.before.run and netstat.log.s.after.run are "netstat -a" output before and after running the test code. netstat.log.s.1/2/3 are "netstat -a" output during running the test code.
07-02-2017

Can you look at the `netstat -a` output while the test runs. My guess is that we are running into an Solaris OS issue, maybe an issue with the ephemeral port range where I assume Solaris needs to bind to same IPv4 and IPv6 port.
06-02-2017