JDK-8218628 : Add detailed message to NullPointerException describing what is null.
  • Type: Enhancement
  • Component: hotspot
  • Sub-Component: runtime
  • Affected Version: 13
  • Priority: P4
  • Status: Resolved
  • Resolution: Fixed
  • Submitted: 2019-02-07
  • Updated: 2020-06-29
  • Resolved: 2019-10-17
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 14
14 b20Fixed
Related Reports
CSR :  
Duplicate :  
Duplicate :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Relates :  
Sub Tasks
JDK-8234223 :  
JDK-8234224 :  
Description
This Enhancement implements the algorithm to compute the null-detail message specified in JEP 358, "Helpful NullPointerExceptions" (https://openjdk.java.net/jeps/358).

The messages printed are described in the JEP. A list of examples is attached to the JEP and can be found here:

- Messages if classfiles contain debug info: http://cr.openjdk.java.net/~goetz/wr19/8218628-exMsg-NPE/21/output_with_debug_info.txt

- Messages if classfiles contain no debug info: http://cr.openjdk.java.net/~goetz/wr19/8218628-exMsg-NPE/21/output_no_debug_info.txt

This RFE gives some details on the implementation on top of what is described in the JEP.

### Modifications to class NullPointerException

The JEP states that the null-detail message is computed delayed and on-demand only.

This is implemented by adding a getMessage() method to NullPointerException. If the field Throwable.detailMessage is empty, it calls native method `getExtendedNPEMessage()` that is implemented in the virtual machine.  This method returns a String Object containing the null-detail message or null if it failed to compute the message.

### Basic algorithm to compute the message

This algorithm is passed the bytecodes of a method and the bytecode index where the exception occurred.

First, the information to step backwards over the bytecodes is computed by a dataflow analysis. This is implemented in the constructor of class `ExceptionMessageBuilder`.  

For each bytecode, it builds a stack of `StackSlotAnalysisData` called `SimulatedOperandStack` describing each operand stack slot. The `StackSlotAnalysisData` is the analysis information for a stack slot and contains the bytecode index of the producer and the Java type of the value on the stack.  It maintains an array indexed by bytecode indexes containing these stacks.

The analysis walks forward over the bytecodes simulating the effects of the bytecode.  Working on a certain bytecode, the simulation first duplicates the stack assigned to this bytecode. Then it pops as many `StackSlotAnalysisData` entries from the copied `SimulatedOperandStack` as the bytecode would pop stack slots.  It then pushes, according to the semantics of the bytecode, new `StackSlotAnalysisData` entries on the stack and assigns the copy to the next bytecode.  Let's look at a possible bytecode `8: getfield #13` at bytecode index 8. Let's assume there are already 5 slots on the stack for bytecode 8. The analysis then copies the stack of depth 5. The getfield bytecode pops an object reference, thus the entry at the top of the `SimulatedOperandStack` must be a reference. This entry is removed. If the constant pool entry #13 describes a reference field, a new entry <reference, 8> is pushed. The next bytecode is at index 11 and gets the new stack assigned.

If bytecode 11 is another getfield and causes a NullPointerException, the operand slot 5 contained null.  The `StackSlotAnalysisData` of the `SimulatedOperandStack` is <reference, 8> and tells that bytecode 8 pushed the null. This way we can step backwards over the bytecodes using the analysis information and find out where the null comes from.

The stack for the first bytecode is initialized to be empty. At bytecodes altering the control flow the stack is just copied to the target. In case of several control flow targets it is copied to all. At a merge point of control flow, the stacks of the predecessors must be merged.  All stack slots that are not identical are set to be undefined. Thus, walking backwards past control flow merges is not always possible. 

The access paths computed by this algorithm usually stem from expressions in Java code. Expressions usually do not contain control flow. (An exemption is the ? operator.)  Therefore the algorithm rarely tries to walk back past control flow merges. 

Assembling the first part of the message is implemented in `print_NPE_failed_action()`. It takes the bytecode at the index passed to the algorithm. A switch over the bytecodes listed in Table 1 of the JEP prints this part of the message.

Assembling the second part of the message is implemented in `print_NPE_cause()`. `print_NPE_cause()` is called with a bytecode index `bci` and an operand stack slot `slot`. It first steps back to the bytecode that pushed this operand stack slot. The index of this bytecode is taken from the corresponding `StackSlotAnalysisData` at `bci` and `slot`.

When `print_NPE_cause()` is called from the main algorithm, it steps back to the 
bytecode that pushed the null value.  The slot needed for this stepping is computed by `get_NPE_null_slot()` which is, similar to `print_NPE_cause()`, a switch over the bytecodes listed in Table 1 of the JEP.

The message part of `print_NPE_cause()` is generated by a switch over the bytecodes listed in Table 2 of the JEP. It calls itself recursively to compute access paths covering several bytecodes. For array loads, it even does two recursive calls: to the bytecode that pushed the array reference and to the bytecode that pushed the array index.

`print_NPE_cause()` keeps a counter of the steps walked back to limit its complexity and to limit the size of the access path printed. In the current implementation the counter limit is set to 6, so that the algorithm walks back at most 5 steps. 

`print_NPE_cause()` does cover all the bytecodes listed in Table 2 of the JEP. Arithmetic bytecodes as `iadd` and casts as `l2i` can be encountered in array index expressions but are not implemented. As stated in the JEP, '...' is printed instead.

All message text is printed directly to a StringStream passed to the algorithm.

### Naming locals

Local variables are managed in a section on the call stack that is split into 'slots'.
If a local variable needs to be named in the message, we print 'local<slot nr>'.
Longs and Doubles span two slots.

This is not very helpful to the user, as this can not be mapped to the source code easily.
We improve in several means:

Instead of 'local0' we print 'this' in dynamic methods.

If the method has parameters, these are mapped trivially to the local 
slots, parameter 0 is held in local slot 0 etc. For parameters, we print 'parameter<param nr>'.  
We parse the signature of the method to find parameters that
span two slots and adapt the count properly.

The slots used for parameters can be reused for other values once a parameter is not used
any more by the method. Code generated by javac of OpenJDK does not 
do so, but other tools generating bytecode optimize it so that this happens.
If the slot is used for another value, the first thing to happen is a store to the local slot.
Analysing the bytecode it can not be decided whether stores to local slots are assignments
to a parameter in the source code, or assignemnts to other locals mapped to this slot.
The dataflow analysis described above implements a simple additional analysis determining
whether a local slot has been assigned.  If this is true, we do not print 'parameter<param nr>', but 'local<slot>'.  Further, this analysis only covers up to 64 slots, i.e., for slots > 64 we don't print 'parameter<param nr>' either.  This is a simple approximation to the source code.

If the class file contains debug information, we can print more useful information. The debug information tells us the name of the variable or parameter of the source code that is accessed.  We print this name.

### Computing the message delayed

The algorithm to compute the message needs the bytecodes and the bytecode index of where the exception occurred.  Obviously, both is available in internal data structures of the virtual machine when the exception is thrown.

As stated in the JEP, the message is computed delayed. Thus, the bytecodes and the bytecode index must be preserved until the message is computed. 

Instead of preserving this information for the exception, we rely on an internal data structure of Throwable, the backtrace.  A Throwable contains an array of StackTraceElements representing the stack when the exception occurred. This array is also computed delayed and on demand only. The information needed to compute this delayed is preserved in the backtrace data structure that contains references to the internal representation of methods and bytecode indexes.

The top frame representation of this backtrace data structure contains just what we need to compute the exception delayed.

The method `java_lang_Throwable::get_method_and_bci()` accesses this data structure and returns a pointer to the method containing the bytecodes we need and the bci.

As we compute the message delayed, the method can be unloaded or rewritten in the meantime.  If so, the bytecodes are not found and `get_method_and_bci()` fails. No message is returned. Similarly, flag OmitStackTraceInFastThrow is on per default. If the stack trace is omitted for an NPE, the backtrace data structure is missing and no message is returned.

The internal backtrace data structure cannot be serialized. Therefore, if a Throwable is serialized, the Java-level StackTraceElements[] is computed.  As specified in the JEP, similar does not happen for the null-detail message of NPE.  Thus, the message of the NPE will be empty after deserialization, and the algorithm to compute it will be called if the message is
accessed. `get_method_and_bci()` will fail as the backtrace data structure is not there and no message is returned.


### Detecting cases where NPE is thrown explicitly

The null-detail message is not printed if the exception was thrown by the user. 

If this is the case, the bytecode at the bytecode index passed to the algorithm is an invoke of the NullPointerException::<init> method. This is recognized in `get_NPE_null_slot()` when we first look at the bytecode and null is returned for the message.

Alternatively, the NullPointerException can be constructed via JNI. If so, the holder of the method is NativeConstructorAccessorImpl. This is checked after obtaining the method with the bytecodes.

### Handling hidden frames

Exceptions print the stack trace of when the exception occurred. 
There can be methods on the stack of code that has been generated by 
the runtime. To not confuse the reader of the stackTrace, who only expects 
frames of methods visible in the code, 
the VM runtime omits these frames from the stack trace (if -XX:-ShowHiddenFrames, which is default.) 

The frames are already dropped when the backtrace data structure is 
built. 

If a NullPointerException is raised in a method whose stack 
frame is hidden, a message based on the wrong method 
will be printed. This is because the backtrace lacks the real top frame. 

We can not compute the proper NullPointerException string and thus must
recognize this situation and omit the message.

For this, this change adds a java.lang.Boolean(true) to the backtrace 
in case the real top frame is a hidden one and is dropped. 

When the NullPointerException message text is generated, this Boolean is 
checked and the message is skipped if the proper frame is not 
available. 

### A flag to switch off the feature.

This change adds a manageable flag called -XX:ShowCodeDetailsInExceptionMessages to 
configure the content of exception messages as above new message.
The default value is "false", i.e., the message is off per default. 
This was requested by Oracle. 
Comments
URL: https://hg.openjdk.java.net/jdk/jdk/rev/e3618c902d17 User: goetz Date: 2019-10-17 09:11:58 +0000
17-10-2019

I closed JDK-8221077 and JDK-8227255. I will handle all three aspects of JDK-8220715 in this bug, as asked by David.
16-07-2019

Coleen, I updated the description with a pointer to the JEP and the example messages at the beginning. After that I describe the algorithm in detail. Unfortunately the markup does not work.
26-06-2019

[~goetz] can you update the RFE with the updated message examples?
20-06-2019

Obviously, this would help others: "8218650: LineNumberTable records for method invocations with arguments" https://bugs.openjdk.java.net/browse/JDK-8218650 http://mail.openjdk.java.net/pipermail/compiler-dev/2019-February/012978.html
12-03-2019

Find the webrev implementing this here: http://cr.openjdk.java.net/~goetz/wr19/8218628-exMsg-NPE/ RFR: http://mail.openjdk.java.net/pipermail/core-libs-dev/2019-February/058447.html http://mail.openjdk.java.net/pipermail/hotspot-runtime-dev/2019-February/032458.html The original change was implemented by Ralf Schmelter.
08-02-2019

There are a number of pre-existing RFEs for this kind of enhancement (that have all been closed in the past). They should be found and linked for good measure.
08-02-2019