JDK-8340728 : Test vmTestbase/gc/memory/Nio/Nio.java is failing to allocate all direct buffer memory
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.nio
  • Affected Version: 24
  • Priority: P3
  • Status: Closed
  • Resolution: Not an Issue
  • Submitted: 2024-09-24
  • Updated: 2024-12-18
  • Resolved: 2024-12-02
Related Reports
Relates :  
JDK-8340946 - Add vmTestbase/gc/memory/Nio/Nio.java and java/nio/Buffer/LimitDirectMemory.java to problem list
Relates :  
JDK-8342850 - Change ProblemList to have LimitDirectMemory refer to JDK-8342849
Relates :  
JDK-8346576 - Remove vmTestbase/gc/memory/Nio/Nio.java from test/hotspot/jtreg/ProblemList.txt
Relates :  
JDK-8249549 - vmTestbase/gc/memory/Nio/Nio.java assume that no direct memory is allocated by JDK
Relates :  
JDK-8319332 - Security properties files inclusion
Relates :  
JDK-8344882 - (bf) Temporary direct buffers should not count against the upper limit on direct buffer memory
Relates :  
JDK-8342849 - (bf) java/nio/Buffer/LimitDirectMemory.java fails with OutOfMemoryError
Sub Tasks
JDK-8340946 :  
Add vmTestbase/gc/memory/Nio/Nio.java and java/nio/Buffer/LimitDirectMemory.java to problem list - Resolved
Description
Test: vmTestbase/gc/memory/Nio/Nio.java

----------System.out:(1/43)----------
Allocating all the direct memory: 52428800
----------System.err:(12/1058)----------
gc.memory.Nio.Nio$Fault: Unexpected OOME during the first allocation java.lang.OutOfMemoryError: Cannot reserve 52428800 bytes of direct buffer memory (allocated: 8192, limit: 52428800)
	at gc.memory.Nio.Nio.run(Nio.java:81)
	at gc.memory.Nio.Nio.main(Nio.java:61)
	at java.base/jdk.internal.reflect.DirectMethodHandleAccessor.invoke(DirectMethodHandleAccessor.java:104)
	at java.base/java.lang.reflect.Method.invoke(Method.java:573)
	at com.sun.javatest.regtest.agent.MainWrapper$MainTask.run(MainWrapper.java:138)
	at java.base/java.lang.Thread.run(Thread.java:1576)

It looks like a recent change has caused 8K to already be allocated.
Comments
[~jpai] Yes, that's an oversight by me. I will fix it. Thanks for catching it.
18-12-2024
Hello Brian [~bpb], it looks like the vmTestbase/gc/memory/Nio/Nio.java test is still problem listed in test/hotspot/jtreg/ProblemList.txt. Is that an oversight?
18-12-2024
This issue is irrelevant now that JDK-8344882 has been integrated.
27-11-2024
This will also become a non-issue if the change proposed for JDK-8344882 is integrated.
25-11-2024
A pull request was submitted for review. Branch: master URL: https://git.openjdk.org/jdk/pull/21623 Date: 2024-10-21 18:22:03 +0000
21-10-2024
With this change (excerpt thereof), the test passed in the CI on all platforms, viz., {linux,macosx}-{aarch64,x64} and windows-x64. --- a/test/hotspot/jtreg/vmTestbase/gc/memory/Nio/Nio.java +++ b/test/hotspot/jtreg/vmTestbase/gc/memory/Nio/Nio.java + max_size = MAX_VM_SIZE; + while (true) { + try { + bb = ByteBuffer.allocateDirect((int)max_size); + System.out.println("... success"); + } catch (OutOfMemoryError oom) { + max_size -= 1024; + continue; + } + break; } The result in all cases was max_size = 52420608 which is 8192 less than the VM max memory. Curiously when run on my local x64 Mac, the result was 16384 less than the max memory (but it still passed).
18-10-2024
The 8K of direct memory is only allocated by the first use of file I/O, any use of FileChannel, Files and other APIs will allocate a direct buffer and that is cached for later use. So it's nothing specific to Properties.LineReader. The old use used a less efficient malloc/free per file I/O. Once we re-implement java.io on NIO then this old code will go away.
17-10-2024
From [~dholmes] above: "The test can be made more robust by scaling back from maxDirectMemory until the allocation succeeds without OOME, but it will still need to use all remaining direct memory for the next part of the test to pass (by failing to allocate a single byte). " This is exactly what I was thinking and how I found the 16K value I tested, but it needs to be done byte by byte. I also agree however that Properties.LineReader should be investigated.
17-10-2024
Interesting how the initial apparent 8K usage has now become 15K+. ?? The test can be made more robust by scaling back from maxDirectMemory until the allocation succeeds without OOME, but it will still need to use all remaining direct memory for the next part of the test to pass (by failing to allocate a single byte). But I'm more concerned that the code in Properties.LineReader is using this 8K of direct memory as that can impact real world applications that are operating at, or close to, the maximum usage of direct memory.
17-10-2024
If this change is made --- a/test/hotspot/jtreg/vmTestbase/gc/memory/Nio/Nio.java +++ b/test/hotspot/jtreg/vmTestbase/gc/memory/Nio/Nio.java @@ -55,7 +55,7 @@ */ public class Nio { - static final int MAX_SIZE = (int)VM.maxDirectMemory(); + static final int MAX_SIZE = (int)VM.maxDirectMemory() - 16*1024; then no OOME is thrown, but it still fails as there is no OOME thrown either when a subsequent attempt is made to allocate 1 byte. The behavior is the same on all five tested platforms. If 15*1024 is instead subtracted, the initial OOME is thrown.
16-10-2024
Regarding Nio.java, it looks like [~jpai]'s comment on the change at https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/security/Security.java#L278 being responsible for the new failure is spot on. If in Nio.java I subtract the value of java.nio.Bits.BUFFER_POOL.getMemoryUsed() from MAX_SIZE, then the test passes.
25-09-2024
It looks like as part of JDK-8319332, the java.security.Security class which loads properties file was changed to use NIO backed Files.newInputStream(...) to load and read properties files https://github.com/openjdk/jdk/blob/master/src/java.base/share/classes/java/security/Security.java#L278. Previously, before that change, the loading and reading of the properties file was using java.io backed FileInputStream as follows: private static boolean loadProps(File masterFile, String extraPropFile, boolean overrideAll) { InputStream is = null; try { if (masterFile != null && masterFile.exists()) { is = new FileInputStream(masterFile); } ... props.load(is); The use of NIO in this code appears to have introduced the dependency on the direct buffers: at java.base/java.nio.Bits.reserveMemory(Bits.java:178) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:105) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:363) at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:304) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:284) at java.base/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:251) at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:231) at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74) at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.base/java.io.InputStream.read(InputStream.java:220) at java.base/java.util.Properties$LineReader.readLine(Properties.java:505) at java.base/java.util.Properties.load0(Properties.java:421) at java.base/java.util.Properties.load(Properties.java:410) at java.base/java.security.Security$SecPropLoader.loadFromPath(Security.java:285) at java.base/java.security.Security$SecPropLoader.loadMaster(Security.java:139) at java.base/java.security.Security$SecPropLoader.loadAll(Security.java:120) at java.base/java.security.Security.initialize(Security.java:333)
25-09-2024
This test also fail by ours CI on linux.
25-09-2024
java/nio/Buffer/LimitDirectMemory.java is a test for the VM option, you can't run anything real with a limit of 10 bytes. In the failure, Properties.LineReader uses an 8K buffer, which requires >= 8K direct memory to do the I/O op so need the direct memory limit to be at least 8k to allow that. This test issues should have been caught before the change was integrated, don't know what testing was done.
24-09-2024
Yes but on the flip side any code that is limiting direct memory will be impacted if 8K is now being claimed where it was not before. I suspect we will have to adjust the tests, but there may be some mitigation possible on the libs side.
24-09-2024
Tests that limit direct memory size to small limits are very fragile, any changes to code do file or network I/O can change the amount of direct memory that is used.
24-09-2024
We have another failing test which sheds some light on the problem: Test: java/nio/Buffer/LimitDirectMemory.java Exception in thread "main" java.lang.OutOfMemoryError: Cannot reserve 8192 bytes of direct buffer memory (allocated: 0, limit: 10) at java.base/java.nio.Bits.reserveMemory(Bits.java:178) at java.base/java.nio.DirectByteBuffer.<init>(DirectByteBuffer.java:105) at java.base/java.nio.ByteBuffer.allocateDirect(ByteBuffer.java:363) at java.base/sun.nio.ch.Util.getTemporaryDirectBuffer(Util.java:242) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:304) at java.base/sun.nio.ch.IOUtil.read(IOUtil.java:284) at java.base/sun.nio.ch.FileChannelImpl.implRead(FileChannelImpl.java:251) at java.base/sun.nio.ch.FileChannelImpl.read(FileChannelImpl.java:231) at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:74) at java.base/sun.nio.ch.ChannelInputStream.read(ChannelInputStream.java:103) at java.base/java.io.InputStream.read(InputStream.java:220) at java.base/java.util.Properties$LineReader.readLine(Properties.java:505) at java.base/java.util.Properties.load0(Properties.java:421) at java.base/java.util.Properties.load(Properties.java:410) This test deliberately sets -XX:MaxDirectMemorySize=10 and this failure shows that the core libs code is now requiring at least 8K of direct memory buffers where previously it did not. It looks like JDK-8319332 is the cause of this so re-assigning to security-libs.
24-09-2024