FULL PRODUCT VERSION :
java version "1.7.0_04"
Java(TM) SE Runtime Environment (build 1.7.0_04-b20)
Java HotSpot(TM) 64-Bit Server VM (build 23.0-b21, mixed mode)


ADDITIONAL OS VERSION INFORMATION :
Linux 3.2.0-24-generic #38-Ubuntu SMP Tue May 1 16:18:50 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
Linux 3.0.0-17-server #30-Ubuntu SMP Thu Mar 8 22:15:30 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux

EXTRA RELEVANT SYSTEM CONFIGURATION :
cat /proc/cmdline
BOOT_IMAGE=/boot/vmlinuz-3.0.0-17-server root=UUID=XXXXXXXXXXXXX ro transparent_hugepage=always


A DESCRIPTION OF THE PROBLEM :
Going from jdk1.7.0_03 to jdk1.7.0_04 I discovered a memory leak in the Linux 64bit implementation of jdk1.7.0_04. This work correctly in jdk1.7.0_03.  I was getting following errors in our server logs:

java.lang.OutOfMemoryError.
Java HotSpot(TM) 64-Bit Server VM warning: CodeCache is full. Compiler has been disabled.
Java HotSpot(TM) 64-Bit Server VM warning: Try increasing the code cache size using -XX:ReservedCodeCacheSize=
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
mmap failed for CEN and END part of zip file

sudo cat /proc/$PID/maps | wc -l

Shows maxed out mmaps(65532).  Most of the entries are of the form:

7f4c5d4d9000-7f4c5d4da000 rw-s 00000000 00:12 469934 /run/shm/sem.ZEIYcO (deleted)

Looking at

http://stackoverflow.com/questions/10441276/jdk-1-7-too-many-open-files-due-to-posix-semaphores
http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=7166379

I thought I had found a workaround, but It seems my bug is different and the work around doesn't work.

What I finally discovered was that using ImageIO with certain JPEGs causes a the memory maps to grow unbounded in jdk1.7.0_04 but not in jdk1.7.0_03. Most JPEGs don't cause the memory leak.

I have a test case.  It uses one of the jpegs I found to cause the memory leak.  It is an image of Rx.
http://postimage.org/image/h6mmgn3ij/
http://s9.postimage.org/h6mmgn3ij/ico_rx_sm.jpg


REGRESSION.  Last worked in version 7

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1.  Download the Rx image from http://postimage.org/image/h6mmgn3ij/ or  http://s9.postimage.org/h6mmgn3ij/ico_rx_sm.jpg

2. Run:
jdk1.7.0_04/bin/java MMAPMemoryLeakTest 25000 ico_rx_sm.jpg
 

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
jdk1.7.0_03/bin/java MMAPMemoryLeakTest 25000 ico_rx_sm.jpg
Process ID is 17499
Iterating 25000 times
Examining /proc/17499/maps for leaks.
Image height, width is 17, 18
  Total mmaps is 211
(deleted) mmaps is 0
difference is 211
ACTUAL -
jdk1.7.0_04/bin/java MMAPMemoryLeakTest 25000 ico_rx_sm.jpg
Process ID is 17404
Iterating 25000 times
Examining /proc/17404/maps for leaks.
Image height, width is 17, 18
  Total mmaps is 65532
(deleted) mmaps is 65321
difference is 211
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to deallocate stack guard pages failed.
Java HotSpot(TM) 64-Bit Server VM warning: Attempt to allocate stack guard pages failed.

REPRODUCIBILITY :
This bug can be reproduced always.

---------- BEGIN SOURCE ----------

import java.io.BufferedReader;
import java.io.File;
import java.io.FileReader;
import java.io.IOException;
import java.lang.management.ManagementFactory;
import java.util.Iterator;
import javax.imageio.ImageIO;
import javax.imageio.ImageReader;
import javax.imageio.stream.ImageInputStream;
 
 
public class MMAPMemoryLeakTest {
 
	public static void main(String... args) throws IOException {
		final int iterations = Integer.parseInt(args[0]);
		final File file = new File(args[1]);
 
		String processId = ManagementFactory.getRuntimeMXBean().getName();
		processId = processId.substring(0, processId.indexOf('@'));
		System.out.println("Image is " + file);
		System.out.println("Process ID is " + processId);
		System.out.println("Iterating " + iterations + " times ");
		int height = 0;
		int width = 0;
		for (int i = 0; i < iterations; i++) {
			try (ImageInputStream iis = ImageIO.createImageInputStream(file)) {
				Iterator<ImageReader> iter = ImageIO.getImageReaders(iis);
				if (iter.hasNext()) {
					ImageReader reader = null;
					try {
						reader = iter.next();
						reader.setInput(iis);
						height = reader.getHeight(0);
						width = reader.getWidth(0);
					} finally {
						if (reader != null) {
							reader.dispose();
						}
					}
				}
			}
		}
		String mmapFile = "/proc/" + processId + "/maps";
		System.out.println("Examining " + mmapFile + " for leaks.");
		try (BufferedReader in = new BufferedReader(new FileReader(mmapFile))) {
			int totalMMaps = 0;
			int markedAsDeleted = 0;
			String ln = in.readLine();
			while (ln != null) {
				totalMMaps++;
				int index = ln.indexOf("(deleted)");
				if (index != -1) {
					markedAsDeleted++;
				}
				ln = in.readLine();
			}
			System.out.println("Image height, width is " + height + ", " + width);
			System.out.println("Total mmaps is " + totalMMaps);
			System.out.println("(deleted) mmaps is " + markedAsDeleted);
			System.out.println("difference is " + (totalMMaps - markedAsDeleted));
		}
	}
}

---------- END SOURCE ----------

CUSTOMER SUBMITTED WORKAROUND :
No workaround.