JDK-8197565 : HTTP Client API
  • Type: CSR
  • Component: core-libs
  • Sub-Component: java.net
  • Priority: P2
  • Status: Closed
  • Resolution: Approved
  • Fix Versions: 11
  • Submitted: 2018-02-12
  • Updated: 2018-07-17
  • Resolved: 2018-03-14
Related Reports
CSR :  
JDK-8197564 - HTTP Client implementation
Description
Summary
-------

High level HTTP Client and WebSocket API.

Problem
-------

This CSR proposes to standardize the HTTP Client API that was introduced
as an incubating API in JDK 9 ( [JEP 110][110] ) and updated in JDK 10.
The incubated API has received a number of rounds of feedback that have
resulted in significant improvements, but at a high level it remains
largely the same. 

Solution
--------

The HTTP Client API provides non-blocking request and response semantics
through `CompletableFutures`, which can be chained to trigger dependent
actions. Back-pressure and flow-control of request and response bodies
is provided for via the Platform's [reactive-streams] [rx] support in
the `java.util.concurrent.Flow` API.

An [introduction to the API][intro] and [example usage][example] are
useful resources to quickly familiarize reviewers / readers with the
API. The code snippets in both of these resources demonstrate usage to
perform common tasks.

[rx]: http://www.reactive-streams.org
[intro]: http://openjdk.java.net/groups/net/httpclient/intro.html
[example]: http://openjdk.java.net/groups/net/httpclient/recipes.html


Specification
-------------

The HTTP Client API will be in the new `java.net.http` module, which
exports a single package of the same name, `java.net.http`. 


High-level API types and their function:

  * `HttpClient` - An HttpClient can be used to send requests and
  retrieve  their responses.
  * `HttpRequest` - An immutable request containing the request URI,
  headers, and optional body.
  * `HttpRequest.BodyPublisher` - A BodyPublisher converts high-level
  Java objects into a flow of byte buffers suitable for sending as a
  request body.
  * `HttpRequest.BodyPublishers` - Predefinded implementations of
  `BodyPublisher` that implement various useful publishers, such as
  publishing the request body from a String, or from a file.
  * `HttpResponse` - An `HttpResponse` is made available when the
  response status code and headers have been received, and typically
  after the response body has also been completely received.
  * `HttpResponse.BodyHandler` - Must be supplied for each
  `HttpRequest` sent. The `BodyHandler` interface allows inspection
  of the response code and headers, before the actual response body
  is received, and is responsible for creating the response
  `BodySubscriber`.
  * `HttpResponse.BodyHandlers` - Predefined implementations of
  `BodyHandler` that implement various useful handlers, such as handling
  the response body as a String, or streaming the response body to a
  file.
  * `HttpResponse.BodySubscriber` -  The `BodySubscriber` consumes
  the actual response body bytes and converts them into a higher-level
  Java type.
  * `HttpResponse.BodySubscribers` - Predefined implementations of
  `BodySubscriber` that implement various useful subscribers, such as
  converting the response body bytes into a String, or streaming the
  bytes to a file.
  * `WebSocket` - A WebSocket client providing low-level access to
  sending and receiving WebSocket messages.
  
The remaining API types can be seen the attached / linked javadoc.  

On-line link to the javadoc:

 http://cr.openjdk.java.net/~chegar/httpclient/01/javadoc/api/java/net/http/package-summary.html


### Removal of the incubating API

The `jdk.incubator.httpclient` module and ALL public types in the 
`jdk.incubator.http` package will be remove. This constitutes a complete
removal of the incubating API added by [JEP 110][110].

[110]: http://openjdk.java.net/jeps/110
Comments
For the methods defined in WebSocket WebSocket.Builder.buildAsync, and HttpClient.sendAsync, it would be convention to list the exceptional conditions in @throws tags. Should WebSocket.Builder.header��� state that "Headers defined in WebSocket Protocol" throw an IllegalArgumentException? Likewise for illegal subprotocol arguments to WebSocket.Builder.subprotocols. Any changes can be done under a follow-up bug. Moving to Approved.
14-03-2018
Joe and other CSR reviewers. The description, comments, attachments, and links have been updated. I believe all comments to date have been addressed.
27-02-2018
Note to reviewers. Changes over what has incubated in JDK 10. ( since the package name change from `jdk.incubator.http` to `java.net.http`, a specdiff is not very useful ) 1) The API documentation has been reorganized to improve readability. `HttpClient` is now the central entry point for high-level API usage and example. 2) To improve data integrity, which operates cleaner with downstream Subscribers, all Flows containing ByteBuffers now specify that the ByteBuffers returned by the HTTP Client are read-only. 3) The push promise support has been re-worked to reduce its impact on the API and bring it more in line with regular request/responses. The `MultiSubscriber` and `MultiResultMap` have been removed. Push Promises are now handled through a functional interface, `PushPromiseHandler`, that is optionally given during a send operation. 4) A few of `BodyHandlers` and corresponding `BodySubscribers` have been added, to improve usability in common scenarios: * discard(Object replacement) combined discarding/ignoring response body and allowing a given replacement. Feedback has indicated that this could appear confusing. It has been removed and replaced with two separate handlers: 1) discarding(), and 2) replacing(Object replacement) * Added `ofLines()` that returns a `BodyHandler<Stream<String>>`, to support streaming of response body as a Stream of lines, line by line. * Added `fromLineSubscriber���`, that supports adaptation of response body to a Flow.Subscriber of String lines. Provides similar semantics to that of BufferedReader.readLine(). * Added `BodySubscriber.mapping` for general purpose mapping from one response body type to another. 5) The predefined implementation of `BodyPublisher`, `BodyHandler`, and `BodySubscriber`, created through static factory methods, have been moved out to separate non-instantiable utility factory classes, following the pluralized naming convention. This improves readability of these relatively small interfaces. 6) The names of the satic factory methods have also been updated along the following broad categories (see attached outboarding.pdf) : - fromXxx: Adapters from standard Subscriber, e.g. takes a `Flow.Subscriber` returns a `BodySubscriber`. - ofXxx: Factories that create a new pre-defined `Body[Publiher|Handler|Subscriber]`, that perform useful common tasks, like handling the response body as a String, or streaming i to a File. - other: Combinators ( takes a `BodySubscriber` returns a `BodySubscriber` ) and other useful operations.
27-02-2018
Please note that In signatures like: ``` fromSubscriber(S subscriber, Function<? super S,? extends T> finisher) ``` the finisher function is a function that _consumes_ a `S` and _produces_ a `T`. As such, the natural way to express these constraints is to use `? super S` and `? extends T` - as it is common in all libraries using j.u.Function - see `Stream::map`. Therefore I think that the signature looks correct as is (both from a mathematical perspective, and from a use case perspective, as Chris shows).
22-02-2018
When the CSR is resubmitted, I'll review the generics again from the following angle: if I'm not mistaken, the signature public static <S extends Subscriber<? super List<ByteBuffer>>,T> BodySubscriber<T> fromSubscriber(S subscriber, Function<? super S,? extends T> finisher) should be functionally equivalent to public static <S extends Subscriber<? super List<ByteBuffer>>,T> BodySubscriber<T> fromSubscriber(S subscriber, Function<?,? extends T> finisher) That is, replacing "? super S" with "?" in the Function type arguments. The super-ness covers any supertype not just an immediate supertype. However, there may be a nuance to this I'm missing.
22-02-2018
The following is a concrete example demonstrating why we believe that the signature should be `Function<? super S,? extends T>`: static class NumberSubscriber implements Flow.Subscriber<List<ByteBuffer>> { @Override public void onSubscribe(Flow.Subscription subscription) { } @Override public void onNext(List<ByteBuffer> item) { } @Override public void onError(Throwable throwable) { } @Override public void onComplete() { } public Number getNumber() { return null; } } static class IntegerSubscriber extends NumberSubscriber { @Override public void onSubscribe(Flow.Subscription subscription) { } @Override public void onNext(List<ByteBuffer> item) { } @Override public void onError(Throwable throwable) { } @Override public void onComplete() { } public Integer getInteger() { return null; } } static class LongSubscriber extends NumberSubscriber { @Override public void onSubscribe(Flow.Subscription subscription) { } @Override public void onNext(List<ByteBuffer> item) { } @Override public void onError(Throwable throwable) { } @Override public void onComplete() { } public Long getLong() { return null; } } static final Function<NumberSubscriber,Number> numMapper = sub -> sub.getNumber(); static final Function<IntegerSubscriber,Integer> intMapper = sub -> sub.getInteger(); static final Function<LongSubscriber,Long> longMapper = sub -> sub.getLong(); public void test() { HttpClient client = null; HttpRequest request = null; IntegerSubscriber sub1 = new IntegerSubscriber(); // with Function<?,? extends T> the following two statements DO NOT compile // with Function<? super S,? extends T> these compile as expected client.sendAsync(request, BodyHandler.fromSubscriber(sub1, IntegerSubscriber::getInteger)); client.sendAsync(request, BodyHandler.fromSubscriber(sub1, NumberSubscriber::getNumber)); // with Function<?,? extends T> this compiles, when it SHOULD NOT // with Function<? super S,? extends T> does not compile as expected client.sendAsync(request, BodyHandler.fromSubscriber(sub1, longMapper)); } `Function<? super S,? extends T>` allows something like like this: static final Function<Object,Number> objectMapper = sub -> 1; client.sendAsync(request, BodyHandler.fromSubscriber(sub1, objectMapper)); , which makes no sense. But this is no different to any other use of `? super` that I am aware of. It's just what you get with Java generics. Additionally, `Function<?,? extends T>` allows something like like this: static final Function<Boolean,Number> booleanMapper = sub -> 1; client.sendAsync(request, BodyHandler.fromSubscriber(sub1, booleanMapper)); , which I'm not sure what would happen if we tried to implement it. I'm sure it would require casting and just fail at runtime when trying to coerce a Flow.Subscriber into the Function.
22-02-2018
Maurizio and I did a complete review of the type parameters in generic methods. - `BodyPublisher` is a _Producer_. As such it's generic methods accept arguments with covariant type parameters, `? extends`. - `BodyHandler`, and `BodySubscriber` are _Consumers_. As such, where appropriate, their generic methods accept arguments with contravariant type parameters, `? super`. Specifically, the `Flow.Subscriber` supertype is the actual _consumer_, therefore its type parameter is contravariant. The type parameter `S extends Flow.Subscriber` appears in a few places so that a finisher function of the same actual concrete type `S` can be used. This allows for method references to the methods of `S`. - All arguments accepting `Function<S,T>` have been updated to `Function<? super S,? extends T>` to accept the broadest possible set of functions that, make sense, and can be used. This is inline, and consistent, with other API signatures in the platform.
20-02-2018
The following types have been changed to be interfaces: - `HttpResponse`, - `HttpClient.Buidler`, and - `HttpRequest.Builder`.
20-02-2018
WebSocket.Listener.{onPing, onPong}: A clarification has been added to the spec regarding: 1. what happens if a WebSocket Protocol error occurs ( onError is invoked with a ProtocolException ), and 2. specifically that the WebSocket Protocol guarantees that a received ping/pong message is not more than 125 bytes. ( This guaranteed is significant as it allows received pings to be replied ( echoed with sendPong ) without the need to size check the received message. Fixed in the sandbox: http://hg.openjdk.java.net/jdk/sandbox/rev/eae2b2d7fe52
20-02-2018
1) What about removing the incubating classes? A note has been added to the CSR text to indicate the the incubating API will be removed. 2) Suspicious use of generics in HttpResponse.BodyHandler static ,T> HttpResponse.BodyHandler fromLineSubscriber���(S subscriber, Function finisher, String lineSeparator) The method-level example for `fromLineSubscriber` in the javadoc has been updated to help with understanding. The `fromSubscriber���` and `fromLineSubscriber` handlers provide interoperability with `Flow.Subscriber` of raw byte and text data, respectively. As such, their generic signatures accept the widest possible set of `Subscriber` and `Flow` types. Specifically, on `fromLineSubscriber���`: `static <S extends Flow.Subscriber<? super String>,T> HttpResponse.BodyHandler<T> fromLineSubscriber���(S subscriber, Function<S,T> finisher, String lineSeparator)` This method captures the actual `Subscriber` type, `S`, so that the `finisher` function accepting the same concrete type can be used, in order to be able to call methods defined by that concrete `Subscriber` subtype. `Flow.Subscriber<? super String>` is used to support Flows of `CharSequence` ( as well as `String`) There may be a minor issue with the `finisher` function types. It should be possible to use a function with a wider generic signature, `Function<? super S, ? extends T>`. We're looking into this. 3) The generics for HttpResponse.BodySubscriber merit a closer look. `BodySubscriber` typically has similarly named static factory methods for creating subscribers as that of the handlers in `BodyHandler`. They can almost be thought of as pairs ( but not quite ), albeit that they serve two distinct and separate purposes. In many cases it is expected that a developer will not need to to use `BodySubscriber` directly, hence it is deliberately not referenced from the higher-level `HttpClient` javadoc. Most of the usages of generics in `BodySubscriber` are straight forward, except for the fromXXX ones. The same explanation as for `BodyHandler` ( 2 ) above applies to these too. Or are you concerned about something else? 4) Not much in some of these types in and of themselves. `Bodyhandler` and `PushPromiseHandler` are functional interfaces, but re-using interfaces from `java.util.function` will not work for `PushPromiseHandler` ( no `TriFunction` ), and when we experimented with `BiFunction` for `BodyHandler` it significantly increased the complexity of the specification when attempting to define its behaviour. All types that are in the API are there for a specific reason, which should be clear from their javadoc. If this is not the case, then maybe the javadoc needs some adjustment to make it clear. What types are you thinking of? 5) Are there relevant RFC that should be referenced? References to the HTTP/1.1, HTTP/2, and WebSocket RFC's, from the package-level javadoc, have been added. 6) In WebSocket.Builder WebSocket.Builder header���(String name, String value) "Adds the given name-value pair to the list of additional HTTP headers sent during the opening handshake. Headers defined in WebSocket Protocol are illegal. If this method is not invoked, no additional HTTP headers will be sent." Should a link to that list be provided? A link as been added to https://tools.ietf.org/html/rfc6455#section-11.3. 7) Is the WebSocket protocol versioned? Yes, it is versioned. There is currently only one version specified and that is now linked to from the RFC links. 8) WebSocket.Listener onPing���, onPong: What is the expected behavior is the byte buffer is more than 125 bytes? Only the first 125 are used? The WebSocket Protocol RFC, and the `WebSocket.Listener` both guarantee that the ping/pong message payload will NOT contain more than 125 bytes. If the question is related to a malformed or malicious data, then a ping/pong with _> 125_ will result in a protocol error, so the listeners `onError` will be invoked. 9) Does the HttpClient constructor need to be protected rather than package private? How about it an interface? Can you discuss the design of making the almost-completely-abstract Http.* type as classes rather than interfaces? In general, most types are required to be subtype-able in order for better interaction with mocking, and similar, frameworks for offline testing. We have examples to demonstrate this in the repository. The `HttpRequest` and `HttpHeaders` types are abstract classes so that they can define their `equals` and `hashCode`. `HttpRequest` is used as a key in a `Map` as part of the API signature for push promise. `HttpHeaders` makes up part of the state of an `HttpRequst` so therefore must specify its `equals` and `hashCode`. `HttpClient` is the main entry point to the API. In general it is not expected, beyond testing, that this type will be subclassed. In theory it could be an interface, but we are not aware of any use-cases that would be better served by this. The `HttpRepsonse` type is an abstract class. It does not have an associated builder, since it is typically created and returned by the `HttpClient`. Again, it is only subtype-able for testing. I think that it probably makes more sense for this type to be interface. If you agree we will make the change. The `Builder` types are almost trivial. Again, we are not aware of any use-cases that would be better served by making these interfaces. 10) Please add some javadoc to the constructors of HttpTimeoutException and WebSocketHandshakeException. Done. 11) A webrev using a separately provided files.list argument to map files across the conceptual rename might help highlight what has changed. http://cr.openjdk.java.net/~chegar/httpclient/00/webrev/ ( just search for `src/java.net.http/share/classes/java/net/http/` to get the public API types )
19-02-2018
I'm moving this request to Provisional, but think a bit more iteration may be needed before this request is finalized. The type HttpResponse.BodySubscriber has one instance method defined in the class, a few inherited instance methods, and then about a dozen static methods.This may be exactly right, but it seems unbalanced at a first look. I recommend WebSocket.Listener.{onPing, onPong} explicitly state what happens if the message violates the length requirements. Thank you for the webrev against the incubating API; seeing that was helpful. I'd prefer HttpResponse to be an interface.
16-02-2018
Some comments on the API after a first pass: What about removing the incubating classes? I assume that will be done under other bug ids. Suspicious use of generics in HttpResponse.BodyHandler static <S extends Flow.Subscriber<? super String>,T> HttpResponse.BodyHandler<T> fromLineSubscriber���(S subscriber, Function<S,T> finisher, String lineSeparator) The generics for HttpResponse.BodySubscriber merit a closer look. Not much in some of these types in and of themselves. Are there relevant RFC that should be referenced? In WebSocket.Builder WebSocket.Builder header���(String name, String value) "Adds the given name-value pair to the list of additional HTTP headers sent during the opening handshake. Headers defined in WebSocket Protocol are illegal. If this method is not invoked, no additional HTTP headers will be sent." Should a link to that list be provided? Is the WebSocket protocol versioned? WebSocket.Listener onPing���, onPong: What is the expected behavior is the byte buffer is more than 125 bytes? Only the first 125 are used? Does the HttpClient constructor need to be protected rather than package private? How about it an interface? Can you discuss the design of making the almost-completely-abstract Http.* type as classes rather than interfaces? Please add some javadoc to the constructors of HttpTimeoutException and WebSocketHandshakeException. A webrev using a separately provided files.list argument to map files across the conceptual rename might help highlight what has changed.
13-02-2018