JDK-7096014 : Javac tokens should retain state
  • Type: Enhancement
  • Component: tools
  • Sub-Component: javac
  • Affected Version: 8
  • Priority: P3
  • Status: Closed
  • Resolution: Fixed
  • OS: generic
  • CPU: unknown
  • Submitted: 2011-09-28
  • Updated: 2013-09-10
  • Resolved: 2012-02-24
The Version table provides details related to the release that this issue/RFE will be addressed.

Unresolved : Release in which this issue/RFE will be addressed.
Resolved: Release in which this issue/RFE has been resolved.
Fixed : Release in which this issue/RFE has been fixed. The release containing this fix may be available for download as an Early Access Release or a General Availability Release.

To download the current JDK release, click here.
JDK 8
8 b12Fixed
Related Reports
Relates :  
JDK-6729835 - Deprecated flag before import or package is not processed properly
Relates :  
JDK-7104201 - Refactor DocCommentScanner
Description
The javac lexer (Scanner.java) has a very complex state, that is very hard to maintain, should we add fancier features like lookahead or save/resume state capabilities.
Comments
SUGGESTED FIX A webrev of this fix is available at the following URL: http://hg.openjdk.java.net/jdk8/tl/langtools/rev/d346ab55031b
24-10-2011
EVALUATION The main problem is that a lot of information (i.e. current token position, current token name value, etc.) is now accessed as part of the scanner state, rather than as part of the token state. In principle, a Lexer should only provide methods for accessing the current token, and for fetching the next one. All other methods should be moved from Lexer to Token. This is currently not possible, as Token is simply an enum constant - as such it is not possible to create multiple tokens of the same kind with different state (i.e. start/end position). An important part of this refactoring will be to allow a clean separation of the token kind (which can be an enum) vs. the token itself, which should be a class with its own state.
28-09-2011