We have a problem with the way that the java.net implements Socket.
This issue is quite complicated but I will try to describe why this happens.
We have a blade system with 40 processors and they are treated as one entity when it comes to IP connectivity.
The problem is related with the bind command on socket.
A bind is treated as a cluster wide resource and should be only used if the endpoint wants to be able to accept connections on that port.
Now all constructors to java.net.Socket except of the empty constructor does an automatic bind on the socket.
In the majority of cases this is not what we want because the developer does not even specify a port in the bind.
Then bind(0) is called. In that pool we have only 4096 ports available in the cluster. This means that each board will only be able to have around 100 connections.
For our own software we can avoid it by using the empty constructor and then call connect. But for our third party software like JDBC drivers, RMI and others using Socket we can not influence it since the code is already written.
So it has happened that we end up without available ports and since this is a cluster wide resource the connecting takes longer time since it needs to be coordinated over the cluster.
We would like to get an advice from you how we can work around this problem.
We are getting into acceptance testing at customer site by the end of this week so we have limited time to make it work.
It is not blocking issue since it only happens sporadically but it prevents us from going live since we can not guarantee that it will not disturb traffic.
Right now we are running a standard Java5 VM on SUSE Linux 9.
Here are the options we have come up with on our own:
1) Rewrite the code in java.net.Socket (Not good, maybe legal issues since this is not our JVM)
2) Extend SocketImplFactory and implement SocketImpl
To implement SocketImpl is not trivial, PlainSocketImpl could be used but it is package scoped. Then there are two possibilities to write our class sub classing PlainSocketImpl and only override bind() or use java.lang.reflect to work around the package permissions and override the package protection. Non seem to be a clean approach, there is some reference to a API in this bug from your homepage but I guess that it was never implemented in Tiger or? http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4245730 (look at the evaluation answer)
3) Use some kind of AOP framework (It would make the socket creation much slower)
4) Use our own OS and JVM (not good since it requires major rewriting)
So no clean and easy way forward.
Can you advice what we should do in this matter!?