JDK-4724038 : (fs) Add unmap method to MappedByteBuffer
  • Type: Enhancement
  • Component: core-libs
  • Sub-Component: java.nio
  • Affected Version: 1.4.0,5.0,7
  • Priority: P4
  • Status: Closed
  • Resolution: Won't Fix
  • OS:
    generic,solaris_8,windows_2000,windows_vista generic,solaris_8,windows_2000,windows_vista
  • CPU: generic,x86
  • Submitted: 2002-07-31
  • Updated: 2023-12-07
  • Resolved: 2023-12-06
Related Reports
Duplicate :  
Duplicate :  
Duplicate :  
Duplicate :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Description
Name: nt126004			Date: 07/31/2002


FULL PRODUCT VERSION :
java version "1.4.0"
Java(TM) 2 Runtime Environment, Standard Edition (build 1.4.0-b92)
Java HotSpot(TM) Client VM (build 1.4.0-b92, mixed mode)

FULL OPERATING SYSTEM VERSION :
Microsoft Windows 2000 [Version 5.00.2195]

A DESCRIPTION OF THE PROBLEM :
Once a file has been mapped a number of operations on that
file will fail until the mapping has been released (e.g.
delete, truncating to a size less than the mapped area).
However the programmer can't control accurately the time at
which the unmapping takes place --- typically it depends on
the processing of finalization or a PhantomReference queue.

The danger associated with unmapping that is noted in the
JavaDoc could be avoided if when an area is unmapped, the
memory is not made available for reallocation but instead
is reserved (e.g. by using VirtualAlloc under Windows).
Then any unwanted reference to the previously mapped area
would result in an exception without incurring performance
penalties. The finalization process could release the
reservation when there were no remaining references to the
buffer.

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
1. Run the attached code.
2. A 1KB file will remain --- the File.delete call fails
unless the System.gc() is uncommented. However we can't
rely on the GC call actually doing anything.


EXPECTED VERSUS ACTUAL BEHAVIOR :
There is no way of ensuring that the delete will work even
though we are no longer interested in the mapping.
The deleteOnExit doesn't work either (with far less excuse)
and I have submitted a bug report for that.


REPRODUCIBILITY :
This bug can be reproduced often.

---------- BEGIN SOURCE ----------
import java.io.*;
import java.nio.MappedByteBuffer;
import java.nio.channels.FileChannel;

class TestMemoryMapping
{
	public static void main(String[] args)
	{
		try
		{
			File f = File.createTempFile("Test", null);
			f.deleteOnExit();
			RandomAccessFile raf = new RandomAccessFile(f, "rw");
			raf.setLength(1024);
			FileChannel channel = raf.getChannel();
			MappedByteBuffer buffer = channel.map
(FileChannel.MapMode.READ_WRITE, 0, 1024);
			channel.close();
			raf.close();
			buffer = null;
			// System.gc();
			if (f.delete())
				System.out.println("Temporary file
deleted: "+f);
			else
				System.out.println("Not yet deleted: "+f);
		}
		catch (IOException ex)
		{
			ex.printStackTrace();
		}
	}
}
---------- END SOURCE ----------
(Review ID: 153923) 
======================================================================

Comments
The functionality we need is in JEP 454: Foreign Function & Memory API
06-12-2023

Hi, in my opinion this is "solved" by the MemorySegment API as to be released for preview in JDK 19. We only want to give some options for people still using MappedByteBuffer and document those. It may also be good to add a warning to Javadocs of the old-style map method. There are 2 options: - Tell them to use the new API FileChannel#map(...,, MemorySession) which returns a MemorySegment. This memorySegment could be converted to a ByteBuffer using asByteBuffer(). To unmap and close people can just close the session. A close() method on the ByteBuffer is not needed - For easy usage, add some other (final) utility method to FileChannel as replacement to FileChannel#map() returning MappedByteBuffer that also takes a MemorySession but returns a (Mapped)ByteBuffer. Behind scenes it could do the above (map as MemorySegment and convert to buffer and return). The problem with 2nd approach is that you can't have 2 overloads with different return type. I was struggling on this when reviewing the original panama API.
30-05-2022

I'd like to second [~uschindler] suggestion. Are we any closer to seeing a write up on proposed solution ?
30-05-2022

Hi Andrew, hi Mark, Dalibor and Rory invited me to be part of the OpenJDK community, so I also finally got access to this bug tracker. I don't want to spam you, but I am still very interested in resolving this issue, before sun.misc.Unsafe finally goes away. The current workaround for this (already used by some opensource projects) is to call the Method sun.misc.Unsafe#invokeCleaner(ByteBuffer), which forcefully unmaps since Java 9. Is there anything I could do to get the ball running? Maybe write some proposal (unfortunately, I dn't fully understand Andrew's plans)? Thanks, Uwe
05-12-2018

Andrew: Be my guest, and I look forward to the result! This is precisely why I've left this bug open since 2005.
12-12-2016

It is possible to unmap a MappedByteBuffer safely with little effect on efficiency: my basic idea is to add an indirection which all callers (even those using derived buffers) have to use to access the memory, and then removing most uses of this indirection with a little compiler magic. A few will still remain, but these will be the minimum required to guarantee security and won't affect the speed of most operations. I have a plan to implement this, but it requires changes to the Java Memory Model and to the compilers. I didn't get it done in time for JDK 9 because of being distracted by a ton of other things, but I hope I'll get it done by JDK 10. I'm taking this bug; I hope that's OK.
12-12-2016

http://mail.openjdk.java.net/pipermail/core-libs-dev/2015-September/035055.html "I have two suggested fixes: "1. Have an unmap method that throws if a SecurityManager is present. Non-secure code that owns its own JVM (i.e. nearly all of it) would then be able to unmap files and delete them on Windows. Sandboxed code would still suffer, but OK, so be it. "2. Have an internal subclass or wrapper object around MappedByteBuffer that is only used when a SecurityManager is present, which does the check against the volatile field. The JVM should optimise out the virtual method call or inline the wrapper code, thus ensuring only sandboxed code pays the cost."
07-09-2015

EVALUATION -- (2005-05-24) In mustang b86 there is a small update that may provide some interim relief for cases where the map method currently fails due to a resource issue. The bugID for this work is 6417205.
24-05-2006

EVALUATION There is no unmap() method on mapped byte buffers because there is no known technique for providing one without running into insurmountable security and performance issues. Suppose that a thread operating on behalf of Alice maps a file into memory and then unmaps it. A second thread operating on behalf of Bob then maps some other file that the underlying operating system happens to assign to the same memory address. Now Alice's thread can read, and possibly even modify, the contents of Bob's file. Oops. This problem could be avoided by defining a private volatile field in a mapped buffer to indicate whether or not the mapping is still valid, and having every access to the buffer's content first check this field and throw an appropriate exception if the mapping is no longer valid. This solution would significantly slow down access, however, and fast access is the whole point of supporting mapped buffers in the first place. Another potential solution is to implement unmap() by unmapping the buffer's memory region and then immediately remapping it from /dev/null, setting the page protections so that any further access to the region would cause a SIGSEGV (or the equivalent) to be raised and then translated into an appropriate Java-level exception. The remapped region would need to remain valid for as long as the original buffer object is valid, eventually being released by the cleaner thread. This solution therefore wouldn't reduce virtual-memory footprint but it would at least allow the underlying file to be manipulated on those operating systems (e.g., Windows) that forbid certain file manipulations when valid mappings exist. To avoid the race condition between the unmap and remap operations we'd need to globally synchronize all mapping operations within the same VM; this could, theoretically, limit scalability on some higher-end systems. The downfall of this second approach is that while we can avoid this race condition within the VM we can't avoid it everywhere within the VM's process. Memory-mapped files are pretty widely used these days, by everything from C's stdio functions (on some systems) to dynamic linkers (on most Unix systems). If a mapping operation were initiated by one of these subsystems, or by native application code (perhaps unknowingly, via an intermediate library), at just the wrong moment, and it happened to overlap our soon-to-be-remapped region, then the remap operation would fail. We at Sun have given this problem a lot of thought, both during the original development of NIO and in the time since. We have yet to come up with a way to implement an unmap() method that's safe, efficient, and plausibly portable across operating systems. We've explored several other alternatives aside from the two described above, but all of them were even more problematic. We'd be thrilled if someone could come up with a workable solution, so we'll leave this bug open in the hope that it will attract attention from someone more clever than we are. With regard to the "workaround", described in the JDC comments, of (ab)using reflection in order to invoke a mapped buffer's cleaner object: This is highly inadvisable, to put it mildly. It is exceedingly dangerous to forcibly unmap a mapped byte buffer that's visible to Java code. Doing so risks both the security and stability of the system. ###@###.### 2005-03-23 02:41:14 GMT
23-03-2005