United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
Bug ID: JDK-6553303 Corba application fails w/ org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No
JDK-6553303 : Corba application fails w/ org.omg.CORBA.COMM_FAILURE: vmcid: SUN minor code: 203 completed: No

Details
Type:
Bug
Submit Date:
2007-05-03
Status:
Resolved
Updated Date:
2010-07-29
Project Name:
JDK
Resolved Date:
2007-09-26
Component:
other-libs
OS:
solaris_9
Sub-Component:
corba:orb
CPU:
sparc
Priority:
P1
Resolution:
Fixed
Affected Versions:
5.0u11
Fixed Versions:
5.0u14 (b02)

Related Reports
Backport:
Backport:

Sub Tasks

Description
Beginning with Java 1.5.0 customers report to run into CORBA exceptions of the 
following kind:

Mar 30, 2007 1:12:12 PM com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl <init>
WARNING: "IOP00410201: (COMM_FAILURE) Connection failure: socketType: IIOP_CLEAR_TEXT; hostname: 47.142.120.39; port: 37220"
org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 201  completed: No
        at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2172)
        at com.sun.corba.se.impl.logging.ORBUtilSystemException.connectFailure(ORBUtilSystemException.java:2193)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:205)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:218)
        at com.sun.corba.se.impl.transport.SocketOrChannelContactInfoImpl.createConnection(SocketOrChannelContactInfoImpl.java:101)
        at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.beginRequest(CorbaClientRequestDispatcherImpl.java:152)
        at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.request(CorbaClientDelegateImpl.java:118)
        at org.omg.CORBA.portable.ObjectImpl._request(ObjectImpl.java:431)
        at com.nortelnetworks.fwcomp_if.common.coseventcomm._PushConsumerStub.push(_PushConsumerStub.java:18)
        at com.nortelnetworks.fwcomp.server.corbabase.es.ProxyPushSupplierImpl.forwardEvent(ProxyPushSupplierImpl.java:231)
        at com.nortelnetworks.fwcomp.common.events.notification.EFQ.run(EFQ.java:181)
 Caused by: java.net.ConnectException: Connection timed out
        at sun.nio.ch.Net.connect(Native Method)
        at sun.nio.ch.SocketChannelImpl.connect(SocketChannelImpl.java:464)
        at java.nio.channels.SocketChannel.open(SocketChannel.java:146)
        at com.sun.corba.se.impl.transport.DefaultSocketFactoryImpl.createSocket(DefaultSocketFactoryImpl.java:60)
        at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.<init>(SocketOrChannelConnectionImpl.java:188)
        ... 8 more


 After some repetitions of the warning, the application sees the following error:

Mar 30 13:30:56 
 org.omg.CORBA.COMM_FAILURE:   vmcid: SUN  minor code: 203  completed: No
 at com.sun.corba.se.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2231)
 at com.sun.corba.se.impl.logging.ORBUtilSystemException.writeErrorSend(ORBUtilSystemException.java:2249)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.sendWithoutLock(SocketOrChannelConnectionImpl.java:1042)
 at com.sun.corba.se.impl.encoding.BufferManagerWriteStream.sendFragment(BufferManagerWriteStream.java:82)
 at com.sun.corba.se.impl.encoding.BufferManagerWriteStream.sendMessage(BufferManagerWriteStream.java:96)
 at com.sun.corba.se.impl.encoding.CDROutputObject.finishSendingMessage(CDROutputObject.java:144)
 at com.sun.corba.se.impl.protocol.CorbaMessageMediatorImpl.finishSendingRequest(CorbaMessageMediatorImpl.java:247)
 at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete1(CorbaClientRequestDispatcherImpl.java:342)
 at com.sun.corba.se.impl.protocol.CorbaClientRequestDispatcherImpl.marshalingComplete(CorbaClientRequestDispatcherImpl.java:323)
 at com.sun.corba.se.impl.protocol.CorbaClientDelegateImpl.invoke(CorbaClientDelegateImpl.java:129)
 at org.omg.CORBA.portable.ObjectImpl._invoke(ObjectImpl.java:457)
 at com.nortelnetworks.fwmgmt_if.common.SnmpIf._AccessorStub.get(_AccessorStub.java:218)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession$1.sendRequest(SNMPSession.java:544)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.sendRequest(SNMPSession.java:1269)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.get(SNMPSession.java:550)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPSession.get(SNMPSession.java:410)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:206)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1326)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:188)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1311)
 at com.nortelnetworks.fwmgmt.server.adaptation.SNMP.SNMPAdaptor.get(SNMPAdaptor.java:167)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000Adaptor.get(Mg5000Adaptor.java:1297)
 at com.nortelnetworks.mg9kem.server.adaptation.base.Mg5000NeAdaptor.getEntPhysicalEntry(Mg5000NeAdaptor.java:2348)
 at com.nortelnetworks.mg9kem.server.em.logicalequipment.Mg5kNe.getEntPhysicalEntry(Mg5kNe.java:4484)
 at com.nortelnetworks.mg9kem.server.em.cards.CardOverloadIface.auditData(CardOverloadIface.java:757)
 at com.nortelnetworks.mg9kem.server.em.physicalequipment.Mg5kIntelligentCard$CardAuditNoRetry.execute(Mg5kIntelligentCard.java:7135)
 at com.nortelnetworks.fwmgmt.server.dataaudit.task.AuditTask$PausableThread.run(AuditTask.java:177)
 Caused by: java.nio.channels.ClosedChannelException
 at sun.nio.ch.SocketChannelImpl.ensureWriteOpen(SocketChannelImpl.java:125)
 at sun.nio.ch.SocketChannelImpl.write(SocketChannelImpl.java:294)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.write(SocketOrChannelConnectionImpl.java:715)
 at com.sun.corba.se.impl.encoding.CDROutputObject.writeTo(CDROutputObject.java:174)
 at com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.sendWithoutLock(SocketOrChannelConnectionImpl.java:992)
 ... 24 more
 

 The application needs to be restarted in order to cure the problem.

 This behaviour of the application was never seen with 1.4.2.


 Customer found out the following:

 Class com.sun.corba.se.impl.transport.SocketOrChannelConnectionImpl.java 
 uses a different send method depending on type of Socket ( nio vs the 
 old type)

 (i) Customer wrote two custom socket factories, one based on new IO,
     the other on the old socket mechanism (i.e without SocketChannels).
     Customer added a way to get a handle to all the client sockets that 
     were created by these factories, so that they could be closed at 
     will.
 (ii) Customer used each of these factories in their application using 
     -Dcom.sun.CORBA.transport.ORBSocketFactoryClass property and then 
     closed the sockets after some time. Customer found the following :
     (a) With nio sockets, the ORB keeps attempting to reuse the old 
         (closed) socket ad infinitum and throws the COMM_FAILURE with 
         minor code 203 -  this is the behavior seen in their application.
     (b) With the old sockets, the COMM_FAILURE happens once after the 
         socket is closed, and then a new one is created to replace it, 
         and subsequent calls are successful.

 In JDK 1.5, the default socket factory is nio based, so this would 
 explain why dropped/closed TCP connections have not given us any 
 trouble in the past.

 Customer provided the following testcase:

 Please find attached the custom socket factory that customer used to 
 demonstrate the behavior: NortelSocketFactory.java

 It can be used with any corba client (no third party software required).
 Customer ran their corba client with the following commandline switches
 (i)  
   -Dcom.sun.CORBA.transport.ORBSocketFactoryClass=NortelSocketFactory  
   -DUseNio=true
 In (i), the client never recovers.
 (ii) 
   -Dcom.sun.CORBA.transport.ORBSocketFactoryClass=NortelSocketFactory  
   -DUseNio=false
 In (ii) , the client recovers automatically.

                                    

Comments
EVALUATION

One part of this problem is clear: if the ORB successfully opens a connection,
but sometime later the connection is no longer usable, it gets stuck in the cache.
In particular, there is no code in the BufferManagerWriteStream class (where all
of the client side writes take place) to remove a connection from the cache
if a failure occurs in SocketOrChannelConnectionImpl.sendWithoutLock.  This 
clearly needs to be added.

I wonder if the first observed error has anything to do with the second.
The first case is what normally happens when attempting to connect to an
address that is not reachable for some reason: we handle the exception, and
no connection is stored in the cache.  Perhaps the failure of the connection occurs
later in some cases than others?
                                     
2007-06-29
EVALUATION

The problem is caused by the lack of a call to purgeCalls in the
exception handler of the sendWithoutLock method.
                                     
2007-07-10



Hardware and Software, Engineered to Work Together