JDK-8162520 : (fs) FileStore should support file stores with > Long.MAX_VALUE capacity
  • Type: Enhancement
  • Component: core-libs
  • Sub-Component: java.nio
  • Priority: P3
  • Status: Resolved
  • Resolution: Fixed
  • OS: generic
  • CPU: generic
  • Submitted: 2016-07-26
  • Updated: 2021-03-05
  • Resolved: 2019-11-01
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 14
14 b22Fixed
Related Reports
CSR :  
Relates :  
Relates :  
Description
FULL PRODUCT VERSION :
All java versions.

ADDITIONAL OS VERSION INFORMATION :
Problem exists on all platforms.

EXTRA RELEVANT SYSTEM CONFIGURATION :
Amazon EFS mount point. This is an NFS filesystem where any user can mount a problematic 8 exabyte filesystem.

A DESCRIPTION OF THE PROBLEM :
Java's FileStore methods (getTotalSpace/getUnallocatedSpace/getUsableSpace) are limited to reporting bytes as signed long (2^63-1), if the filesystem is any bigger, then the size overflows.

For example, any user of the Amazon EFS filesystem can easily mount 2^63 bytes, which is exactly one byte too big, and all methods return negative numbers (Long.MIN_VALUE). This is the problem, because its no longer a theoretical issue, but a practical one for anyone who uses that platform.

On Unix systems, the underlying values are represented as two 64-bit unsigned integers: blocksize and block counts. This is enough to support enormous filesystems. The problem is that FileStore multiplies these together into one 64-bit signed long, which overflows.

On Windows systems, the method being called actually returns a 64-bit unsigned long, but its treated by java as if it were signed, so it is currently not quite right either. This seems to be the maximum size supported by windows.

See email discussion on nio-dev@ 
http://mail.openjdk.java.net/pipermail/nio-dev/2016-July/003784.html

STEPS TO FOLLOW TO REPRODUCE THE PROBLEM :
Mount a filesystem larger than 2^63-1 bytes and inspect values reported by FileStore.

EXPECTED VERSUS ACTUAL BEHAVIOR :
EXPECTED -
FileStore should have new methods that support large filesystems. For example ZFS can support up to 2^78 bytes.

ACTUAL -
Disk usage goes negative.

ERROR MESSAGES/STACK TRACES THAT OCCUR :
This breaks our application (Elasticsearch), because it is a distributed system, it needs some basic knowledge of disk usage in order to allocate data...

REPRODUCIBILITY :
This bug can be reproduced always.


Comments
URL: https://hg.openjdk.java.net/jdk/jdk/rev/76638c631869 User: bpb Date: 2019-11-01 20:17:33 +0000
01-11-2019

Alternatives: - Look at Posix apis - Consider high level units than bytes (Gb or Mb) - BigInteger
09-02-2018

This will need an API change, probably new methods to get the actual sizes, and maybe change the existing methods to return Long.MAX_VALUE when they are too large for a long.
26-07-2016