JDK-8075679 : Corba server load failure post internal warnings & subsequent timeout
  • Type: Bug
  • Component: other-libs
  • Sub-Component: corba
  • Affected Version: 7u60
  • Priority: P3
  • Status: Closed
  • Resolution: Not an Issue
  • Submitted: 2015-03-23
  • Updated: 2015-04-23
  • Resolved: 2015-04-23
Related Reports
Relates :  
JDK-8017195 - Introduce option to setKeepAlive parameter on CORBA sockets
Relates :  
JDK-6873605 - Missing finishedDispatch() call in ORBImpl causes test failures after 5u20 b04
Description
EJB application deployed on IBM Websphere application server fails to 
invoke EJB after an idle period of about 30 mins. 

WARNING: "IOP00710261: (INTERNAL) Number of invocations is already zero, but 
another invocation has completed" 
org.omg.CORBA.INTERNAL:   vmcid: SUN  minor code: 261  completed: Yes 
       at com.sun.corba.se.impl.logging.ORBUtilSystemException.numInvocationsAlreadyZero (Unknown Source) 
       at  com.sun.corba.se.impl.logging.ORBUtilSystemException.numInvocationsAlreadyZero (Unknown Source) 
       at com.sun.corba.se.impl.orb.ORBImpl.finishedDispatch(Unknown Source) 
       at com.sun.corba.se.impl.orb.ORBImpl.releaseOrDecrementInvocationInfo(Unknown Source) 
       at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.releaseReply(Unknown Source) 
       at org.omg.CORBA.portable.ObjectImpl._releaseReply(Unknown Source) 
[...]
Comments
Submitter got back to confirm that the keep-alive CORBA property worked for them: === After using CORBA Parameter the unexpected and undesired behaviour is no longer reproducible. The JNLP client was modified with the following property in its jnlp file: <property name="jnlp.com.sun.CORBA.transport.enableTcpKeepAlive" value="true"/> === I'm going to close this one out.
23-04-2015
JDK-6763340 was a CORBA memory leak issue fixed in 6u18. I suspect it's a factor in the issue being seen. The "numInvocationsAlreadyZero" monitoring was introduced as a result of JDK-6763340 also but could be a result of a deeper problem in the corba code or Cu code. When the memory leak existed, the socket resources may not have been cleaned up correctly and were kept alive. The 30 mins of idle time is an interesting observation here. Is the server/client socket connection being closed out after a set amount of idle time? Has any logging been captured to that effect ? The default corba socket factory does not have keep-alive setting. Ideally, a corba app should implement some sort of ping activity on such a socket to keep it from being closed. Is is possible that a firewall or other software is monitoring idle sockets and closing them out ? https://bugs.openjdk.java.net/browse/JDK-8017195 introduced a private property whereby the com.sun.corba.se.impl.transport.DefaultSocketFactoryImpl implementation set keep alive on sockets created by the corba factory. The keep alive settings from the solaris kernel would then dertermine how long idle sockets were kept open. The property is activated by -Dcom.sun.CORBA.transport.enableTcpKeepAlive=true - I think it's a valid test to set this on the client VM end to see if socket keep alive settings is indeed a factor for this issue. Please have the submitter try this and report back on results.
24-03-2015
Most likely related to the JDK-6873605 fix.
23-03-2015