United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-5083450 : (se) Temporary selectors not closed upon thread exit

Details
Type:
Bug
Submit Date:
2004-08-05
Status:
Resolved
Updated Date:
2005-04-16
Project Name:
JDK
Resolved Date:
2005-04-16
Component:
core-libs
OS:
windows_xp,windows_2000
Sub-Component:
java.nio
CPU:
x86
Priority:
P4
Resolution:
Fixed
Affected Versions:
1.4.2,1.4.2_09
Fixed Versions:

Related Reports
Backport:
Backport:
Relates:

Sub Tasks

Description
Name: rmT116609			Date: 08/05/2004


FULL PRODUCT VERSION :
java version "1.5.0-beta2"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.5.0-beta2-b51)
Java HotSpot(TM) Client VM (build 1.5.0-beta2-b51, mixed mode, sharing)

java version "1.4.2_05"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.2_05-b04)
Java HotSpot(TM) Client VM (build 1.4.2_05-b04, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Microsoft Windows XP [Version 5.1.2600]


A DESCRIPTION OF THE PROBLEM :
When using a SocketChannel in blocking mode,  sun.nio.ch.SocketAdaptor (via the innerclass SocketInputStream) creates a temporary selector (via sun.nio.ch.Util) for imitating stream-like read functionality when a timeout is specified on the socket.  On Windows, this selector is of type sun.nio.ch.WindowsSelectorImpl.  WindowsSelectorImpl uses a Pipe class for wakeup monitoring.  This Pipe, which has one socket allocated each for source and sink, is only closed properly when implClose is executed on WindowsSelectorImpl (i.e. if selector.close() were called at some point).  However, when SocketAdaptor is done with the temporarily created selector, it never explicitly calls selector.close(), which causes the Pipe-related sockets to be orphaned and left in an ESTABLISHED state until process death.  Over time, this exhausts the number of available sockets and degrades both application and operating system performance. Pipe-related sockets are only cleaned up when the owning process is killed.

  To be even more clear, it seems that by setting a timeout on the underlying socket of the SocketChannel causes this temporary selector and it's associated pipe sockets to never be cleaned up properly.

This is also broken in J2SE 1.4.2_05 &  b59 of 1.5.0_beta3.


STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Run the MiniServer program included below, specifying a port (e.g. java MiniServer 17605)
2. In a second command window, run "netstat -n -o -p tcp".  Take note of the number of sockets (and their port numbers) associated with MiniServer process.
3. In a third command, run the Client program included below, specifying the server's port (e.g. java Client 17605). Wait for completion.
4. In the second command window, run "netstat -n -o -p tcp" and note there are two more sockets left established that are associated with the MiniServer process.
5. Repeat steps 3 & 4, noting how there are two more sockets left open for each time Client is run.

(See source code example for programs)



EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
After each run of the Client program completes, I'd expect to see that all related sockets are in a TIME_WAIT state.
ACTUAL -
After each connection, the source and sink sockets are left in an ESTABLISHED state.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
When running netstat -n -o -p tcp after several runs of the Client program, you see results like the following (PID of server is 1936).

C:\Documents and Settings\tjohnson>netstat -n -o -p tcp
Active Connections

  Proto  Local Address          Foreign Address        State           PID
  TCP    127.0.0.1:3131         127.0.0.1:3132         ESTABLISHED     1936
  TCP    127.0.0.1:3132         127.0.0.1:3131         ESTABLISHED     1936
  TCP    127.0.0.1:3134         127.0.0.1:3136         ESTABLISHED     1936
  TCP    127.0.0.1:3136         127.0.0.1:3134         ESTABLISHED     1936
  TCP    127.0.0.1:3139         127.0.0.1:3140         ESTABLISHED     1936
  TCP    127.0.0.1:3140         127.0.0.1:3139         ESTABLISHED     1936
  TCP    127.0.0.1:3144         127.0.0.1:3146         ESTABLISHED     1936
  TCP    127.0.0.1:3146         127.0.0.1:3144         ESTABLISHED     1936
  TCP    127.0.0.1:3149         127.0.0.1:3150         ESTABLISHED     1936
  TCP    127.0.0.1:3150         127.0.0.1:3149         ESTABLISHED     1936
  TCP    127.0.0.1:3154         127.0.0.1:3155         ESTABLISHED     1936
  TCP    127.0.0.1:3155         127.0.0.1:3154         ESTABLISHED     1936
  TCP    127.0.0.1:3159         127.0.0.1:3160         ESTABLISHED     1936
  TCP    127.0.0.1:3160         127.0.0.1:3159         ESTABLISHED     1936
  TCP    127.0.0.1:3164         127.0.0.1:3165         ESTABLISHED     1936
  TCP    127.0.0.1:3165         127.0.0.1:3164         ESTABLISHED     1936


REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
[MiniServer Program]
import java.io.*;
import java.net.*;
import java.nio.*;
import java.nio.channels.*;
import java.util.*;

public class MiniServer
{
  private int port;
  private ByteBuffer echoBuffer = ByteBuffer.allocate(1024);

  public MiniServer(int port) throws IOException
  {
    this.port = port;
    go();
  }

  static public void main(String args[]) throws Exception
  {
    if(args.length<=0)
    {
      System.err.println("Usage: java MiniServer port");
      System.exit(1);
    }

    int port = Integer.parseInt(args[0]);
    try
    {
      new MiniServer(port);
    }
    catch(Exception e)
    {
      e.printStackTrace();
      System.exit(1);
    }
  }

  private void go() throws IOException
  {
    Selector selector = Selector.open();
    ServerSocketChannel ssc = ServerSocketChannel.open();
    ssc.configureBlocking( false );
    ServerSocket ss = ssc.socket();
    InetSocketAddress address = new InetSocketAddress(this.port);
    ss.bind( address );

    ssc.register( selector, SelectionKey.OP_ACCEPT );
    System.out.println("Going to listen on " + port);

    while(true)
    {
      int num = selector.select();
      Set selectedKeys = selector.selectedKeys();
      Iterator it = selectedKeys.iterator();

      while(it.hasNext())
      {
        SelectionKey key = (SelectionKey)it.next();
        if((key.readyOps() & SelectionKey.OP_ACCEPT) == SelectionKey.OP_ACCEPT)
        {
          ssc = (ServerSocketChannel)key.channel();
          SocketChannel sc = ssc.accept();
          sc.configureBlocking(true);
          ClientProcessor cp = new ClientProcessor(sc.socket());
          it.remove();
          System.out.println("Got connection from " + sc);
          cp.start();
        }
      }
    }
  }

  public class ClientProcessor extends java.lang.Thread
  {
    private Socket socket;
    public ClientProcessor(Socket s)
    {
      socket = s;
    }

    public void run()
    {
      try
      {
	  //NOTE:  If you comment out the following line, the bug goes away!
        this.socket.setSoTimeout(60000);
        copy(this.socket.getInputStream(), this.socket.getOutputStream(), 1024);
      }
      catch(Exception e)
      {
        e.printStackTrace();
      }
      finally
      {
        try
        {
          socket.shutdownInput();
          socket.shutdownOutput();
          socket.close();
        }
        catch(Exception e)
        {
          e.printStackTrace();
        }
      }
    }
    public long copy(final java.io.InputStream inputStream, final java.io.OutputStream outputStream, final int bufferSize)
    throws java.io.IOException
    {
      long totalBytesReadCount = 0;
      final byte[] byteBuf = new byte[bufferSize];
      for(;;)
      {
        final int bytesReadCount = inputStream.read(byteBuf);
        if(bytesReadCount == -1)
          break;
        outputStream.write(byteBuf, 0, bytesReadCount);
        totalBytesReadCount += bytesReadCount;
      }
      return(totalBytesReadCount);
    }
  }
}

[CLIENT PROGRAM]
import java.nio.channels.SocketChannel;
import java.nio.ByteBuffer;
import java.nio.channels.Selector;
import java.nio.channels.SelectionKey;

public class Client
{

  private String text = "echo text";
  private int port;

  public Client(int port)
  {
    this.port = port;
  }

  public void run()
   throws Exception
  {
    SocketChannel sc = SocketChannel.open();
    sc.connect(new java.net.InetSocketAddress("localhost", port));
    sc.configureBlocking(false);
    while(!sc.finishConnect())
    {
      //spin until connected
    }

    int bytesWritten = 0;
    ByteBuffer wb = ByteBuffer.wrap(text.getBytes());

    bytesWritten = sc.write(wb);

    Selector selector = Selector.open();
    sc.register(selector, SelectionKey.OP_READ);

    ByteBuffer rb = ByteBuffer.allocate(128);

    selector.select();

    int bytesRead = 0;

    while(bytesRead != -1 && bytesWritten != bytesRead)
    {
      rb.clear();
      bytesRead = sc.read(rb);
      rb.flip();
      System.out.println("Returned text:  " + new String(rb.array()));
    }
    selector.close();
    sc.close();
  }

  static public void main( String args[] ) throws Exception
  {
    if(args.length<=0)
    {
      System.err.println( "Usage: java Client port" );
      System.exit( 1 );
    }

    Client client = new Client(Integer.parseInt(args[0]));
    try
    {
      client.run();
    }
    catch(Exception e)
    {
      e.printStackTrace();
      System.exit(1);

    }
  }
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
By adding a check for the selector in the finally clause of SocketInputStream in sun.nio.ch.SocketAdaptor and calling close if the selector not null, the pipe-related sockets are properly cleaned up up.

finally
{
  //... other code that goes here
if(selector != null)
  selector.close();
}
(Incident Review ID: 295699) 
======================================================================

                                    

Comments
EVALUATION

There's definitely a bug in our code here.  Temporary selectors are cached on a
per-thread basis, but when a thread goes away nothing is done to reclaim the
resources used by that thread's selector.  This bug can (and should) be fixed
by adding a small amount of GC-triggered cleanup code.

The suggested fix, which would have the SocketAdaptor close the selector when
the I/O operation is finished, would negate the benefits of this cache -- which
are nontrivial on Windows, less so on other platforms.

A workaround that you could try is to not allocate a new thread for every
single client request.  An easy way to do this, while still being able to
service requests in parallel, is to use the thread pool facilities in Doug
Lea's util.concurrent package.  That package, slightly changed, is now
java.util.concurrent in JDK 5, but if you need to run on 1.4.2 you can get
Doug's package (it's public-domain code) from his web page (just google "Doug
Lea", his page shows up first).

-- ###@###.### 2004/8/25
                                     
191-12-08 0



Hardware and Software, Engineered to Work Together