United StatesChange Country, Oracle Worldwide Web Sites Communities I am a... I want to...
JDK-8035897 : Better memory allocation for file descriptors greater than 1024 on macosx

Details
Type:
Bug
Submit Date:
2014-02-26
Status:
Closed
Updated Date:
2014-09-04
Project Name:
JDK
Resolved Date:
2014-03-02
Component:
core-libs
OS:
os_x
Sub-Component:
java.net
CPU:
generic
Priority:
P1
Resolution:
Fixed
Affected Versions:
7u25,8,9
Fixed Versions:

Related Reports
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Backport:
Duplicate:
Duplicate:
Relates:
Relates:

Sub Tasks

Description
JDK-8021820 adds -D_DARWIN_UNLIMITED_SELECT to the build, but the fd_set struct is still limited to 1024.

Snippet from man select(2):

   "Although the provision of getdtablesize(2) was intended to allow user programs to be written
     independent of the kernel limit on the number of open files, the dimension of a sufficiently
     large bit field for select remains a problem.  The default size FD_SETSIZE (currently 1024) is
     somewhat smaller than the current kernel limit to the number of open files.  However, in order
     to accommodate programs which might potentially use a larger number of open files with select,
     it is possible to increase this size within a program by providing a larger definition of
     FD_SETSIZE before the inclusion of <sys/types.h>.

Either:
 1) FD_SETSIZE needs to be set to a larger value, but what value, the kernel limit, or other?
     This is wasteful for most typical apps that don't use large numbers of file descriptors. Or,
 2) If fd is greater than 1024, then an appropriate amount of memory could be allocated and cast
     to an fd_set. The FD_SET macro will write past FD_SETSIZE.

Option 2 above seems most reasonable. Will proceed with making the appropriate changes.
                                    

Comments
Related Regression test:  test/java/net/ServerSocket/AnotherSelectFdsLimit.java 
Platform for Verification: sc22bk02.us.oracle.com MacOSX 10.8.4

Testing Results:

Run with Unfixed version of java (jdk9/b03)
22bk02:JDK-8035897 aurora$ /net/sqenfs-1/export1/jdk/re/9/promoted/all/b03/binaries/macosx-amd64/bin/java AnotherSelectFdsLimit 1025
Using 1025 fds.
Exception in thread "Thread-1023" Exception in thread "Thread-1024" java.io.UncheckedIOException: java.net.SocketException: Socket closed
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:74)
Caused by: java.net.SocketException: Socket closed
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:70)
java.io.UncheckedIOException: java.net.SocketException: Socket closed
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:74)
Caused by: java.net.SocketException: Socket closed
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:70)
sc22bk02:JDK-8035897 aurora$ /net/sqenfs-1/export1/jdk/re/9/promoted/all/b03/binaries/macosx-amd64/bin/java AnotherSelectFdsLimit
Using 1600 fds.
Exception in thread "Thread-1271" Exception in thread "Thread-1279" Exception in thread "Thread-1283" Exception in thread "Thread-1285" java.io.UncheckedIOException: java.net.SocketException: Invalid argument
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:74)
Caused by: java.net.SocketException: Invalid argument
	at java.net.PlainSocketImpl.socketAccept(Native Method)
	at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:404)
	at java.net.ServerSocket.implAccept(ServerSocket.java:545)
	at java.net.ServerSocket.accept(ServerSocket.java:513)
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:70)
Exception in thread "Thread-1294" Exception in thread "Thread-1291" java.io.UncheckedIOException: java.net.SocketException: Invalid argument
	at AnotherSelectFdsLimit$WorkerThread.run(AnotherSelectFdsLimit.java:74)
Caused by: java.net.SocketException: Invalid argument
.........

_______________________________________________________
Run with fixed version of Java (JDK9/b04):
22bk02:JDK-8035897 aurora$ /net/sqenfs-1/export1/jdk/re/9/promoted/all/b04/binaries/macosx-amd64/bin/java AnotherSelectFdsLimit 1025
Using 1025 fds.
sc22bk02:JDK-8035897 aurora$ /net/sqenfs-1/export1/jdk/re/9/promoted/all/b04/binaries/macosx-amd64/bin/java AnotherSelectFdsLimit 1600

                                     
2014-03-28
SQE OK to take the fix into 7u60
                                     
2014-03-14
Removing CPU14_02-critical-watch. Management have indicated that a patch will be provided from the sustaining group, if needed. This change will make its way into the next available regular update in 7uXX and 8uXX.
                                     
2014-03-13
URL:   http://hg.openjdk.java.net/jdk9/jdk9/jdk/rev/a1df86f672a3
User:  lana
Date:  2014-03-11 02:12:41 +0000

                                     
2014-03-11
Adjusting Priority, to P1, as per ILW ( Impact:H, Likelihood:M, Workaround:H )
                                     
2014-03-11
URL:   http://hg.openjdk.java.net/jdk9/dev/jdk/rev/a1df86f672a3
User:  chegar
Date:  2014-03-02 19:26:11 +0000

                                     
2014-03-02
bsd_close.c has the same problem. It uses getrlimit() to figure out what the actual kernel limit is before allocating its fd array.
                                     
2014-02-27



Hardware and Software, Engineered to Work Together