FULL PRODUCT VERSION :
java version "1.6.0_20"
Java(TM) SE Runtime Environment (build 1.6.0_20-b02)
Java HotSpot(TM) 64-Bit Server VM (build 16.3-b01, mixed mode)

ADDITIONAL OS VERSION INFORMATION :
Linux dbnu099 2.6.16.60-0.42.4-smp #1 SMP Fri Aug 14 14:33:26 UTC 2009 x86_64 x86_64 x86_64 GNU/Linux

A DESCRIPTION OF THE PROBLEM :
Every request on webserver port which is not http conform for example just open und direclty close a socket on webserver produce memory dust.

it looks like a memory leak.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
start a

com.sun.net.httpserver.HttpServer;

Webserver with or without context. simply start any kind of these webserver.

open a telnet connection on the webserver port and directly close it in a while loop.




EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
when i fininish these test i expected a normally running webserver.
ACTUAL -
acutally there is a memory leak. Every open socket connection on webserver produce a object that never cleaned up in memory.

This only appears when the request is non http conform request. such from a port scanner or similar.

when i make a heap dump i saw a huge amount of:

sun.nio.ch.SelektionKeyImpl
and
sun.net.httpserver.HTTPConnection

Objects, which are not cleaned up by the garbage collector. so there still is a reference.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
FEIN: RMI TCP Connection(1863)-10.1.102.144: close connection
Exception in thread "RMI TCP Connection(idle)" java.lang.OutOfMemoryError: Java heap space

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------
package TestServer;
import java.io.IOException;
import java.io.OutputStream;
import java.net.InetSocketAddress;
import java.util.List;
import java.util.Map;
import java.util.Set;
import java.util.logging.Logger;

import com.sun.net.httpserver.Headers;
import com.sun.net.httpserver.HttpExchange;
import com.sun.net.httpserver.HttpHandler;
import com.sun.net.httpserver.HttpServer;


public class HTTPServer extends Thread implements HttpHandler {
	
	private static Logger logger = Logger.getLogger("Test");
	private InetSocketAddress address;
	private HttpServer server;
	private String hostname = "localhost";
	private int port=6686;
	
	private Object waiter = new Object();
	
	private boolean isRunning=false;
	
	//private Executor exec = ThreadPoolExecutor.;
	
	
	private boolean isRunning() {
		return isRunning;
	}

	private void setRunning(boolean isRunning) {
		this.isRunning = isRunning;
	}

	public HTTPServer() {
		
        try {
        	address = new InetSocketAddress(hostname, port);
			server = HttpServer.create(address, 10);
			
	        //server.createContex
	        
	        logger.info("starting http server("+address+")");
	        
	        server.createContext("/", this);
	        //server.setExecutor(Executors.newCachedThreadPool());
	        server.start();
	        
	        this.performAction();

		} catch (Exception e) {
			e.printStackTrace();
			logger.warning(e.toString());
		}
        
	}
	
	private void performAction() {
		setRunning(true);
        Runtime.getRuntime().addShutdownHook(this);
           	
    	while (isRunning()) {
    		try {
		    	synchronized (waiter) {
		    		waiter.wait(5000);
		    	}
    		} catch (InterruptedException ie) {
    		}
    	}
    	server.stop(3);
    	//TODO unpublishAllEndpoints();
	}

	@Override
	public void handle(HttpExchange xchg) throws IOException {
		
		Headers headers = xchg.getRequestHeaders();
		logger.info("receive request from: "+xchg.getRemoteAddress());
		Set<Map.Entry<String, List<String>>> entries = headers.entrySet();

		StringBuffer response = new StringBuffer();
		for (Map.Entry<String, List<String>> entry : entries)
			response.append(entry.toString() + "\n");

		xchg.sendResponseHeaders(200, response.length());
		OutputStream os = xchg.getResponseBody();
		os.write(response.toString().getBytes());
		os.close();
	}


	public void run() {
		logger.info("HTTP Test Server ShutdownHook has been aktivated.");
		
    	setRunning(false);
    	synchronized (waiter) {
    		waiter.notifyAll();
    	}
    }

}



package TestServer;
import java.util.logging.Logger;


public class MainTest {
	
	private static Logger logger = Logger.getLogger("Test");

	/**
	 * @param args
	 */
	public static void main(String[] args) {
		logger.info("create HTTPServer...");
		new HTTPServer();
	}

}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
dont send non http conform requests to server