Summary
-------

Provide an interactive tool to evaluate declarations, statements, and
expressions of the Java programming language, together with an API so
that other applications can leverage this functionality.


Goals
-----

The JShell API and tool will provide a way to interactively evaluate
declarations, statements, and expressions of the Java programming
language within the JShell state.  The JShell state includes an evolving
code and execution state.  To facilitate rapid investigation and coding,
statements and expressions need not occur within a method, and
variables and method need not occur within a class.

The `jshell` tool will be a command-line tool with features to ease
interaction including: a history with editing, tab-completion, automatic
addition of needed terminal semicolons, and configurable predefined
imports and definitions.


Non-Goals
---------

A new interactive language is not the goal: All accepted input must match
grammar productions in the Java Language Specification (JLS).  Further,
within an appropriate surrounding context, all accepted input must be
valid Java code (JShell will automatically provide that surrounding
context -- the "wrapping").  That is, if `X` is an input that JShell
accepts (as opposed to rejects with error) then there is an `A` and `B`
such that `AXB` is a valid program in the Java programming language.

Out of scope are graphical interfaces and debugger support. The JShell API
is intended to allow JShell functionality in IDEs and other tools, but the `jshell`
tool is not intended to be an IDE.

Motivation
----------

Immediate feedback is important when learning a programming language and
its APIs. The number one reason schools cite for moving away from Java as a
teaching language is that other languages have a "REPL" and have far lower
bars to an initial `"Hello, world!"` program.  A Read-Eval-Print Loop (REPL) 
is an interactive programming tool which loops, continually reading user input, 
evaluating the input, and printing the value of the input or a description of the
state change the input caused.  Scala, Ruby, JavaScript,
Haskell, Clojure, and Python all have REPLs and all allow small initial
programs.  JShell adds REPL functionality to the Java platform.

Exploration of coding options is also important for developers
prototyping code or investigating a new API.  Interactive evaluation is
vastly more efficient in this regard than edit/compile/execute and
`System.out.println`.

Without the ceremony of `class Foo { public static void main(String[]
args) { ... } }`, learning and exploration is streamlined.


Description
-----------

### Functionality

The JShell API will provide all of JShell's evaluation functionality.
The code fragments that are input to the API are referred to as
"snippets".  The `jshell` tool will also use the JShell completion API to
determine when input is incomplete (and the user must be prompted for
more), when it would be complete if a semicolon were added (in which case the
tool will append the semicolon) and also how to complete input when
completion is requested with a tab.  The tool will have a set of commands
for query, saving and restoring work, and configuration.  Commands will
be distinguished from snippets by a leading slash.

### Documentation

The JShell module API Specifications can found here:

 - http://download.java.net/java/jdk9/docs/api/jdk.jshell-summary.html

Which includes the primary JShell API (`jdk.jshell` package) Specification:

 - http://download.java.net/java/jdk9/docs/api/jdk/jshell/package-summary.html

The `jshell` tool reference:

 - https://docs.oracle.com/javase/9/tools/jshell.htm

is part of the Java Platform, Standard Edition Tools Reference:

 - https://docs.oracle.com/javase/9/tools/tools-and-command-reference.htm

### Terms

In this document, the term
���class��� is meant in the sense used in the Java Virtual Machine
Specification (JVMS), which includes Java Language Specification (JLS) classes, interfaces, enums,
and annotation types. The text makes it clear if a different meaning is intended.

### Snippets

A snippet must correspond to one of the following JLS syntax productions:

  - *Expression*
  - *Statement*
  - *ClassDeclaration*
  - *InterfaceDeclaration*
  - *MethodDeclaration*
  - *FieldDeclaration*
  - *ImportDeclaration*

In JShell, a "variable" is a storage location and has an associated type.  A variable is created explicitly with a *FieldDeclaration* snippet:

    int a = 42;

or implicitly by an expression (see below).  Variables have a small amount of field semantics/syntax (for example,
the `volatile` modifier is allowed). However, variables have no user-visible class enclosing them
and will be generally viewed and used like local variables.

All expressions are accepted as snippets. This includes expressions without side effects, such as constants, variable accesses, and lambda expressions:

    1
    a
    2+2
    Math.PI
    x -> x+1
    (String s) -> s.length()

as well as expressions with side effects, such as assignments and method invocations:

    a = 1
    System.out.println("Hello world");
    new BufferedReader(new InputStreamReader(System.in))

Some forms of expression snippet implicitly create a variable to store the expression's value so it can be referred to later by other snippets.
By default, an implicitly created variable has the name `$`*X* where *X* is the snippet identifier.
A variable is not implicitly created if the expression is void (the `println` example),
or if the value of the expression can already be referred to by a
simple name (as in the case of 'a' and 'a=1' above; all other examples have variables implicitly created for them).

All statements are accepted as snippets, except for 'break', 'continue', and 'return'. However, a snippet may contain 'break', 'continue', or 'return' statements where they meet the usual rules of the Java programming language for enclosing context. For example, the return statement in this snippet is valid because it is enclosed in a lambda expression:

    () -> { return 42; }

A declaration snippet (*ClassDeclaration*, *InterfaceDeclaration*,
*MethodDeclaration*, or *FieldDeclaration*) is a snippet that explicitly introduces a name
that can be referred to by other snippets.  A declaration snippet is subject to the following rules:

  - the access modifiers (`public`, `protected`, and `private`) are ignored (all declaration snippets are accessible to all other snippets)
  - the modifier `final` is ignored (future changes/inheritance are permitted)
  - the modifier `static` is ignored (there is no user-visible containing class)
  - the modifiers `default` and `synchronized` are not allowed
  - the modifier `abstract` is allowed only on classes.

All snippets, except for those of the form *ImportDeclaration*, may contain nested declarations. For example, a snippet that is a class instance creation expression may specify an anonymous class body with nested method declarations. The usual rules of the Java programming language apply to modifiers on nested declarations, rather than the rules above. For example, the class snippet below is accepted, and the private modifier on the nested method declaration is respected, so the snippet "`new C().secret()`" would not be accepted:

    class C {
      int answer() { return 2 * secret(); }
      private int secret() { return 21; }
    }

A snippet may not declare a package or a module. All JShell code is placed in a single package in an unnamed module. The name of the package is controlled by JShell.

Within the `jshell` tool, the terminal semicolon of a snippet can be omitted if that semicolon would be the last character of the input
(excluding whitespace and comments).

### State

The JShell state is held in an instance of `JShell`.  A snippet is
evaluated in a `JShell` with the `eval(...)` method, producing an
error, declaring code, or executing a statement or expression.  In the
case of a variable with an initializer, both declaration and execution
occur.  An instance of `JShell` contains previously-defined and
modified variables, methods, and classes, previously-defined import
declarations, the side-effects of previously-entered statements and
expressions (including variable initializers), and external code bases.

### Modification

Since the desired use is exploration, the declarations (variables,
methods, and classes) must be able to evolve over time while, at the same
time, preserving evaluated data.  One choice would be to make a changed
declaration a new additional entity in some or all cases, but that is
certain to be confusing and does not play well with exploring the
interaction between declarations.  In JShell, each unique declaration key
has exactly one declaration at any given time.  For variables and classes
the unique declaration key is the name, and,
the unique declaration key for methods is the name and the parameter
types (to allow for overloading).  As this is Java, variable, methods,
and classes each have their own name spaces.

### Forward reference

In the Java programming language, within the body of a class,
references to members which will appear later
can appear; this is a forward reference.  As code is entered
and evaluated sequentially in JShell, these references will be
temporarily unresolved. In some cases, for example mutual recursion,
forward reference is required.  This can also occur in exploratory
programming while entering code, for example, realizing that another (so
far unwritten) method should be called.  JShell supports forward
references in method bodies, return type, and parameter types, in 
variable type, and, within a class.
Since the semantics requires them to be immediately
executed, forward references in variable initializers is not supported.

### Snippet dependencies

The code state is kept up-to-date and consistent; that is, when a snippet
is evaluated, any changes to dependent snippets are immediately
propagated.

When a snippet is successfully declared, the declaration will be one of
three kinds: *Added*, *Modified*, or *Replaced*.  A snippet is *Added* if
it is the first declaration with that key.  A snippet is *Replaced* if
its key matches a previous snippet, but their signatures differ.  A
snippet is *Modified* if its key matches a previous snippet and their
signatures match; in this case, no dependent snippets are impacted.  In
both the *Modified* and *Replaced* cases the previous snippet is no
longer part of the code state.

When a snippet is *Added* it may be providing an unresolved reference.
When a snippet is *Replaced* it may update an existing snippet.  For
example, if a method's return type is declared to be of class `C` and
then class `C` is *Replaced* then the method's signature has changed and
the method must be *Replaced*.  Note: This can cause previously-valid
methods or classes to become invalid.

The desire is that user data persist whenever possible.  This is attained
except in the case of variable *Replace*.  When a variable is replaced,
either directly by the user or indirectly via a dependency update, the
variable is set to its default value (`null` since this can only occur
with reference variables).

When a declaration is invalid, either because of a forward-reference or
becoming invalid through an update, the declaration is "corralled".  A
corralled declaration can be used in other declarations and code,
however, if an attempt is made to execute it a runtime exception will
occur which will explain the unresolved references or other issues.

### Wrapping

In the Java programming language, variables, methods, statements,
and expressions must be nested within other constructs, ultimately a class.
When the implementation of JShell compiles a variable, method,
statement, and expression snippet as
Java code an artificial context is needed, as follows:

  - Variables, Methods, and Classes
    - As static members of a synthetic class
  - Expressions and Statements
    - As expressions and statements within a synthetic static method within a
      synthetic class

This wrapping also enables snippet update, so, note that a snippet
class is also wrapped in a synthetic class.

### Modular Environment Configuration

The `jshell` tool has the following options for controlling the modular
environment:

  - `--module-path`
  - `--add-modules`
  - `--add-exports`

The modular environment can also be configured by direct addition
to the compiler and runtime options.
Compiler flags may be added with the `-C` option.  Runtime flags
may be added with the `-R` option.

All `jshell` tool options are documented in the Tool Reference (see
above).

The modular environment can be configured at the API level with
the `compilerOptions` and `remoteVMOptions` methods on
`JShell.Builder`.

The set of modules read by JShell's unnamed module is the same as the default set of root modules for the unnamed module, as established by JEP 261 "Root modules":

  - http://openjdk.java.net/jeps/261

### Naming

  - Module
    - `jdk.jshell`
  - Tool launcher
    - `jshell`
  - API package
    - `jdk.jshell`
  - SPI package
    - `jdk.jshell.spi`
  - Execution engine "library" package
    - `jdk.jshell.execution`
  - Tool launching API package
    - `jdk.jshell.tool`
  - Tool implementation package
    - `jdk.internal.jshell.tool`
  - OpenJDK Project
    - Kulla


Alternatives
------------

A simpler alternative is just to provide a batch scripting wrapper
without interactive/update support.

Another alternative is to maintain the status quo: Use another language
or use a third-party REPL such as [BeanShell](http://beanshell.org),
though that particular REPL has been dormant for many years, is based on
JDK 1.3, and makes arbitrary changes to the language.

Many IDEs, for example the NetBeans debugger and BlueJ's CodePad, provide
mechanisms to interactively evaluate expressions.  Preserved context and
code remains class-based, and method granularity is not supported.  They
use specially crafted parsers/interpreters.


Testing
-------

The API facilitates detailed point testing.  A test framework makes
writing tests straight-forward.

Because the evaluation and query functionality of the tool is built on
the API, most testing is of the API.  Command testing and sanity testing
of the tool is, however, also needed.  The tool is built with hooks for a
testing harness, which is used for tool testing.

Tests are comprised of three parts:

 1. Tests for the API.  These tests cover both positive and negative
    cases.  Each public method must be covered by tests which include
    adding variables, methods, and class, redefining them, etc.

 2. Testing of the jshell tool.  These tests check that `jshell` commands and
    compilation, and execution of Java code, have correct behavior.

 3. Stress testing.  To ensure that JShell can compile all allowed Java
    snippets, correct Java code from the JDK itself will be used.  These
    tests parse the sources, feed code chunks to the API, and test
    the behavior of the API.


Dependences
-----------

The implementation will make every effort to leverage the accuracy and
engineering effort of the existing language support in the JDK.  The
JShell state is modeled as a JVM instance.  Code analysis and the
production of executable code (`jdk.jshell` API) will be performed by the Java Compiler
(`javac`) through the Compiler API.  Code replacement (`jdk.jshell.execution`) will use the Java
Debug Interface (JDI).

Parsing of raw snippets (i.e., snippets that have not been wrapped) will
be done using the Compiler API with a small subclassing of the parser to
allow raw snippets.  The resulting information will be used to wrap the
snippet into a valid compilation unit including a class declaration with
imports for previously evaluated code.  Further analysis and generation
of the class file will be done with unmodified instances of the Java
compiler.  Generated class files will be kept in memory and never written
to storage.  `jdk.jshell.spi` SPI exists to configure the execution engine.  The default
execution engine behaves as follows.  Class files will be sent over a
socket to the remote process.  A remote agent will handle loading and
execution.  Replacement will be done via the JDI
`VirtualMachine.redefineClasses()` facility.

Tab-completion analysis (`jdk.jshell` API) will also use the Compiler API.  Completion
detection will use the `javac` lexer, custom and table-driven code. 

The `jshell` tool (`jdk.internal.jshell.tool`) will use 'jline2' for console input, editing, and history.
`jline2` has been, privately, rolled into the JDK.