JDK-8303616 : DatagramSocket performance regression with concurrent senders
  • Type: Bug
  • Component: core-libs
  • Sub-Component: java.net
  • Affected Version: 17,19,21
  • Priority: P4
  • Status: Open
  • Resolution: Unresolved
  • Submitted: 2023-03-04
  • Updated: 2024-03-06
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
Other
tbdUnresolved
Related Reports
Duplicate :  
JDK-8327363 - New DatagramSocket implementation does not allow parallel sending
Relates :  
JDK-8235674 - JEP 373: Reimplement the Legacy DatagramSocket API
Relates :  
JDK-8253119 - Remove the legacy PlainSocketImpl and PlainDatagramSocketImpl implementation
Description
We noticed a major degradation in throughput in our app after upgrading from JDK 11 to JDK 17/19/21-ea.

The application uses a shared instance of DatagramSocket in multiple threads. The root cause of the performance drop is JDK-8235674, which re-implemented DatagramSocket using DatagramChannel. The problem with the new implementation is that it acquires a ReentrantLock on every send/receive call [1], thus limiting I/O scalability. The old implementation - AbstractPlainDatagramSocketImpl - did not have a per-socket lock on send/receive path.

I attached the simple test case that demonstrates the issue: DatagramTest.java

Here are the sample performance results using 4 threads, measured in packets per second (the higher the better).

$ java DatagramTest 4
278309 pps
336652 pps
342375 pps
335843 pps
327992 pps

$ java -Djdk.net.usePlainDatagramSocketImpl DatagramTest 4
1616246 pps
1692003 pps
1696878 pps
1693931 pps
1710490 pps

The old implementation (AbstractPlainDatagramSocketImpl) shows 5x higher throughput than the new implementation (DatagramChannel).

The corresponding CPU profiles:
Old (faster): https://cr.openjdk.org/~apangin/8303616/old.html
New (slower): https://cr.openjdk.org/~apangin/8303616/new.html
(note the large volume of ReentrantLock.lock/unlock on the graph)

The workaround for JDK 17 is the option -Djdk.net.usePlainDatagramSocketImpl that enables the old implementation. Unfortunately, this no longer works since JDK-8253119.

send() and recv() calls are thread safe in POSIX and Windows, they should not be guarded by a Java level lock.

[1] https://github.com/openjdk/jdk/blob/5b2e2e4695768a6bd8090fb9a6c342fcddcbb3fd/src/java.base/share/classes/sun/nio/ch/DatagramChannelImpl.java#L827
https://github.com/openjdk/jdk/blob/5b2e2e4695768a6bd8090fb9a6c342fcddcbb3fd/src/java.base/share/classes/sun/nio/ch/DatagramChannelImpl.java#L891
Comments
I think the more typical usage would be a DatagramSocket per thread, maybe with SO_REUSEPORT when the source address/port needs to be the same. But, contention on the write lock is possible if there are concurrent senders with the same DatagramSocket/DatagramChannel. DatagramChannel requires a write lock to correctly implement its spec, support async close, interrupt, and work with virtual threads. We prototyped changes to DatagramChannel some time ago to allow concurrent senders. The spec changes aren't too bad but the implementation changes are somewhat disruptive and require send/write never block to avoid tracking writer threads for signalling. It wasn't clear at the time that it was worth doing as concurrent sender/writers should not be typical.
04-03-2023