JDK-8205517 : JFR tool
  • Type: CSR
  • Component: hotspot
  • Sub-Component: jfr
  • Priority: P2
  • Status: Closed
  • Resolution: Approved
  • Fix Versions: 12
  • Submitted: 2018-06-22
  • Updated: 2019-05-28
  • Resolved: 2018-11-29
Related Reports
CSR :  
JDK-8205516 - JFR tool
Description
Summary
-------

Provide a tool for interacting with Flight Recorder files (.jfr).

The main functionality will allow binary file contents to be filtered, summarized and output in a human readable format. There will also be support for merging and splitting recording files.

Problem
-------

Developers need a straightforward way to extract contents from a JFR recording file that doesn't involve writing a Java program, or downloading a separate tool.

Solution
--------

The tool is built on top of the API available in jdk.jfr and the implementation located in the jdk.jfr module. The tool is well suited for demonstrating functionality in presentations, demos and documentation. It will work on all platforms, and in scenarios where a graphical client can���t be used.

An alternative is JMC, but it requires an always up to date JMC parser. Also, JMC doesn't support splitting and merging of recording files.

The functionality provided by the tool has been available in the JDK since the inception of JFR, but it has been undocumented and located in internal packages. In JDK 9, the functionality was cleaned up and proper testing added. The proposal is to move the functionality into a command line tool called jfr.

The commands supported by the tool are:

**Print**

Print the contents of a recording file to standard out. To reduce the amount of data, it is possible to filter out events or categories of events. The filter operates on the symbolic name of an event, which is set using the @Name annotation, or the category name, set by the @Category annotation. If multiple filters are used, events from both filters will be included. If no filter is used, all the events will be printed. If a combination of a category filter and event filter is used, the selected events will be the union of the two filters. For example, to show all GC events and the CPULoad event the following command could be used:

    $> jfr print --categories GC --events CPULoad recording.jfr

Event values are formatted according to the content types that are being used. For example, a field with the jdk.jfr.Percentage annotation that has the value 0.52 is formatted as 52%. Stack traces are by default truncated to 5 frames, but the number can be increased/decreased using a command-line option. It is also possible to output the data in XML and JSON. Such output will not be formatted according to the content type. This is to make the data machine-readable for scripts parsing the output.

**Metadata**

Print metadata about events, such as event names, categories and field layout.

**Assemble**

Assemble chunk files into a recording file, useful for recovering recording data from a crashed JVM. The files are concatenated in chronological order and files that are not finished (.part) are excluded.

**Disassemble**

Disassemble a recording file into chunk files, useful for debugging the parser and to repair a broken file by removing the faulty chunk. It can also be used to reduce the size of a file that is too large to transfer. The resulting chunk files are named  myfile_1.jfr, myfile_2.jfr etc. and padded with zeros if needed to preserve chronological order, for example myfile_001.jfr if the recording consists of more than 100 chunks.

**Summary**

Print statistics for a recording, for example the number of events and how much disk space they use. Useful for troubleshooting and understanding the impact of event settings. 

**Version**

Display the version of the tool, starting from 1.0. The version will be bumped when a new command-line flag is added/removed/changed, or the output format is changed.

**Help**

Display available commands, or detailed information about a particular command.




Specification
-------------
**Print command**

     jfr print [--xml|--json]
               [--categories <filters>]
               [--events <filters>]
               [--stack-depth <depth>]  
               <file>
    
     --xml                   Print recording in XML format
     --json                  Print recording in JSON format
     --categories <filters>  Select events matching a category name.
                             The filter is a comma-separated list of names,
                             simple and/or qualified, and/or quoted glob patterns
     --events <filters>      Select events matching an event name.
                             The filter is a comma-separated list of names,
                             simple and/or qualified, and/or quoted glob patterns
     --stack-depth <depth>   Number of frames in stack traces, by default 5
     <file>                  Location of the recording file (.jfr)

      
**Metadata command**

    jfr metadata <file>
    
    <file>                   Location of the recording file (.jfr)
      
**Summary command**

   
    jfr summary <file>
    
    <file>                   Location of the recording file (.jfr)

**Assemble command**


    jfr assemble <repository> 
                 <file>
    
    <repository>             Directory where the repository is located
    <file>                   Location of the recording file (.jfr)

**Disassemble command**


    jfr disassemble [--max-chunks <chunks>]
                    [--output <directory>]    
                    <file>
    
    --output <directory>  The location to write the disassembled file, 
                          by default the current directory                  
    --max-chunks <chunks> Maximum number of chunks per file, by default 5.
                          The chunk size varies, but is typically around 15 MB.
    --max-size <size>     Maximum number of bytes per file. 
    <file>                Location of the recording file (.jfr)

**Version command**

    jfr <--version|version>

**Help command**


    jfr <--help|help> 
        [command]
     
    command                The name of the command to get help for
Comments
Moving to Approved.
29-11-2018
Email exchange: On the assumption that the recording file format may evolve - how would that be handled by this tool going forward? The tool will handle compatibility the same way as the Flight Recorder API. An exception will be thrown if the parser detects a future version. In the tool, a message in the following format will be printed to std err. jfr print: could not read recording <filename>. File version x.0. Only Flight Recorder files of version 1.x and 2.x can be read by this JDK. So - does the tool handle today���s format - and what version is that? The current version 2.0. JDK 7/8: 0.9 JDK 9/10: 1.0 The tool (and API) can read recordings back to JDK 9. We plan to keep this support going forward unless the maintenance cost becomes too high. Today the JDK 9/10 support is about 20 LOC. If you were to update the format for JDK13 for example would the new tool support the new format and the old format? Would you expect an error message if there was a version mismatch in future? If we bump the major (an incompatible file format change), the JDK 12 version will print an error message when reading a JDK 13 recording. See above. The JDK 13 version of the tool will be able to read a JDK 12 recording, unless we deem it unfeasible to maintain (which is unlikely). If we bump the minor, the tool will ignore it and parse the recording. Is there a way to ask what version of the recording file this tool supports? Yes. the 'jfr summary' command will print the version of a recording file.
28-11-2018
Attached XML schema, added "$>" to the example command-line and made it clear that wildcard filters need to be quoted. Also, changed interface to "To add/remove command in JDK\bin"
21-11-2018
Moving to Provisional. Please include a reference to any schema for the XML before the request is finalized.
14-11-2018
Flight Recorder data is written in chunks and it's is not possible to divide a file further. A chunk is always valid. it contains all the information needed to be parsable and typically the events to make it useful for troubleshooting an application. I updated the CSR with a description of the naming convention of the resulting chunk files. I added an option to divide a file by size (--max-size) That said, main purpose of the tool is to divide by chunks because that is what you need to debug problematic files, I changed the command names to "assemble" and "disassemble". It makes it a bit more clear they are the opposite of each other. Merge (and split) sounds a bit too generic. It's not possible, split at an arbitrary position and when files are assembled it should happen in the correct order. I also added an option where you can specify the output directory for a disassembled file. I updated the print command with a description on how the filter works (--categories and --events). The help for the print command will display example usage of filters, so it will be easy to understand how they work. I also added a --version option which I set to "1.0". It will be updated when a new command-line flag is added, or the output format is changed. I removed the --fields option, but added a new command metadata, which will print event descriptions, but not their payload. I like to avoid the ToolProvider API as the proper way to access the data from Java is using the API. The tool exists so developers don't need to write their own pretty printer. To print the contents of a recording file in Java, a developer can simply do: try (Recorded event : RecordingFile.readAllEvents(Path.of("recording.jfr)) { System.out-println(event); } Users should not build their applications on top of these commands. I have talked to Mandy about what is needed to make sure the tool included in a modular image with jdk.jfr and I have also verified that this is the case.
13-11-2018
Please address the other questions and concerns from the previous comment.
06-11-2018
For the split option what is what are the semantics of max_chunks? Break up file into max_chunks pieces? What are the names of the resulting files, <file>.1, <file>.2? Is there utility in breaking the file into chunks of a given size rather than a given number of chunks? Was the name "merge" considered instead of "reconstruct"? I assume the splitting and merging are done semantically (i.e. at internal format boundaries) rather than just straight splits and merges of the file contents. In other words, are the individual files of a split themselves valid jfr files? What are categories? Where are the valid filter options for events, fields, and categories documented? Many command line tools also support a --version option. Consider implementing the java.util.spi.ToolProvider API. I recommend checking with Alan or Mandy about having appropriate module information so that the jfr command line tool is included or excluded based on the underlying implementation module being included or excluded.
02-11-2018
Good to know that it will work with modular images without updating jlink. Yes, the tool only makes sense to have if the jdk.jfr module is present. I think we want to avoid using jcmd as its purpose is to be a dumb client that can operate on diffrent (live) JVM versions (not files). The tool consists of four separate "commands" (print, reconstruct, summary, and split) and available options depend on which command is being used. For instance, options for the print command can't be combined with reconstruct, summary or split. It would be easier to understand how to use the tool if this could be seen in the syntax: $> jfr print --xml --events jdk.CPULoad recording.jfr $> jfr summary recording.jfr $> jfr split --max-chunks 5 recording.jfr Would that be OK?
09-10-2018
I think the likely users of a command to analyze jfr files would get by even if an explicit path had to be set. As I understand it, if the module associated with a executable is included or excluded with jlink, the the command is excluded or included as appropriate based on this modules are included in an image. Presumably the jdk.jfr.internal.cmd code is not so fundamental that is need always be included. Adding the ability to analyze jfr files to jcmd would be a reasonable approach too.
16-07-2018
Thanks Joe. I think we want the tools in the JDK, since they use internal JDK classes and it is hard ro maintain a copy elsewhere. There are a couple of more benefits with ordinary Java programs that I like to mention. 1) It will work regardless of directory, if javapath has been setup by the installer, for example: $> java jdk.jfr.cmd.Print <file> instead of: $>"c:\Program Files\Java\jdk-11\bin\jfr" print <file> A user could set the PATH to JDK_HOME\bin, but this will work out of the box. 2) It will also work if you have created a modular image, with let's say --add-modules java.base,jdk.jfr, and deployed it on a production server. We could fix this by updating the jlink tool to copy the JFR launcher if jdk.jfr is present, if that is OK? If we take the path with $JDK/bin/jfr, we should add a help text explaning this is a tool for post processing .jfr files and that they should use -XX:StartFlightRecording, or jcmd <pid> JFR.start to actually create a recording.
09-07-2018
Moving the request to Provisional. There have been efforts to reduce the number of commands in $JDK/bin and the command have been reduced. To me, there seems to be sufficient justification to creating a command line tool to process jfr recordings, or at least finding a home for this functionality somewhere other than the "java" command itself.
02-07-2018
I don't think the best approach is sufficiently clear at this point to Finalize and Approve the request. Please Withdraw and Request and then Propose it; I will then advance the request to the Provisional state while the discussion of how to provide the functionality in question continues.
28-06-2018
These are ordinary Java programs, not command-line tools located in $JDK/bin. The implementation is located in the jdk.jfr module. The java launcher will invoke the main method for these utility classes, for example jdk.jfr.cmd.Print#main(String[] args). Adding a --help seems like a good idea. I updated the CSR. ======================================================= Was adding a $JDK/bin/jfr tool for this purpose considered? ======================================================= It was our initial thought, and the implementation in the JDK today is built as launcher (jdk.jfr.internal.cmd.Execute) that can be invoked with commands, such as "print", "split" and "summary" . We didn't pursue building a launcher in JDK 9 because we thought that there was an effort to reduce the number of command line utilities in $JDK/bin? Regardless, there is also usability aspect to consider. The main functionality of JFR is to start and dump recordings. If there was a jfr tool in $JDK/bin, most users would assume that would be THE tool to use when working with JFR (instead of jcmd, or command line flags). The proposed tools are for post processing recording files and they may not carry their weight of a launcher in $JDK\bin folder?
27-06-2018
Was adding a $JDK/bin/jfr tool for this purpose considered?
27-06-2018
I am not clear on what exactly is being proposed. These are commands, but not command line tools that would live in $JDK/bin, if I am interpreting the request correctly. If command line tools are not being proposed, is there a proposal for a jar file that would let the command line $java jdk.jfr.cmd.Print recording.jfr execute successfully or is access to these packages supposed to be built into the java command itself? If command line tools are proposed, they would follow the conventions on accepting "--help", etc. Pending the request.
27-06-2018