Bug ID: JDK-8196422 Test Plan for JEP 331: Low-Overhead Heap Profiling

Type: Sub-task
Component: hotspot
Sub-Component: gc

Priority: P4
Status: Closed
Resolution: Duplicate

Submitted: 2018-01-30
Updated: 2018-05-03
Resolved: 2018-05-03

JVMTI Based Heap Sampling Testing
--------------------------------------------------

The following test plan covers various elements. 

First, the tests should cover:
  A) All new JVMTI functions tested
  B) All common use-cases tested
  C) Accuracy testing of the sampling
  D) Stability proven via being used in some simple application
  E) Proven to work with all openjdk supported GCs via the tests

Then, the test plan in terms of pass rate and stability are:
  E) Test pass rate 100%
  F) Test stability >99%

We will run the tests X amount of times in a row to ensure reaching A+B criteria.

Additionally, tests should have the following coverage criteria:
  G) Block coverage for new/changed java code 80%
  H) Line coverage for new/changed C/C++ code 80%

We will show this via a test coverage pass.

There are a few key tests that need to be implemented:
   I) Show that the data is sane via the stacktraces collected and amount of data shown to be collected.
        - This shows the accuracy of the sampling

Finally, we will show no overhead was induced by the feature implementation when not used via this criteria:
   J) No overhead incurred due to the feature turned off, measured via performance tests

Update on the criterias with current tests in the webrevs (as of http://cr.openjdk.java.net/~jcbeyler/8171119/heap_event5/): A) SetHeapSamplingRate and the SampledObjectAlloc event tested - This is done via the various tests but especially HeapMonitoringStatObjectCorrectnessTest checks various sample rates, so is perfect for checking SetHeapSamplingRate - The HeapMonitorStatRateTest also tests various sampling rates - HeapMonitoringStatObjectCorrectnessTest, HeapMonitoringStatArrayCorrectnessTest, the HeapMonitorGC* and HeapMonitorInterpreter* tests handle various use-cases of different GCs and runtime modes. B) All common use-cases tested - As said in (A), HeapMonitoringStatObjectCorrectnessTest and HeapMonitoringStatArrayCorrectnessTest cover the most common use-case with default GC and runtime options - As a corner case, HeapMonitorThreadOnOffTest and HeapMonitorThreadTest handle the thread information passed in the event and thread safety of the overall system - The native library attached to the tests, in libHeapMonitor.c has a simplified/naive C-implementation tracking events that is thread-safe - The HeapMonitorTest tests if the system is enabled then disabled - If there are multiple agents, the system still works as intended - If the user requests events for only a given set of threads, only events for those threads are obtained C) Accuracy testing of the sampling - This is handled by the HeapMonitorStatRateTest, HeapMonitoringStatObjectCorrectnessTest, and the HeapMonitoringStatArrayCorrectnessTest tests; they all test various event sending and the rate of sampling. D) Stability proven via being used in some simple application - This has been proven by benchmarking the Dacapo benchmark suite and not seeing a single functional regression with the system on/off; and a XX% regression with the system turned on with events being fired but ignored (not being processed by the native agent). E) Proven to work with all openjdk supported GCs via the tests - This is handled by the HeapMonitorStaGC*Test (which can be deprecated when CMS is gone). E) Test pass rate 100% - Currently deflaking it F) Test stability >99% - Currently deflaking it G) Code coverage for new/changed Java code: - There is no new Java code so this is not required H) Code coverage for new/changed C++ code: - Code was changed in these sub-groups: - Architecture dependent code: due to a change in TLAB field, each architecture needs to get tested with TLAB enabled (if not default), this will sufficiently test that code change - gc/shared/collectedheap code will be tested by any tests using TLAB since it will use that code with the code skipping over the new code - threadLocalBuffer* is really tested by any of the tests that are getting events and checking statistical sampling rates or stacks since they will stress that code - interpreter code is augmented with a sampled object collector, this is tested by the HeapMonitorInterpreter*Test tests - runtime is augmented with a sampled object collector, this is tested by all tests in default settings - opto is modified to handle the new tlab naming, this is tested by any test using C2, which is all of the tests in default settings We are running the tests 1000 amount of times in a row to ensure reaching E+F criteria. I) Show that the data is sane via the stacktraces collected and amount of data shown to be collected. - This shows the accuracy of the sampling: HeapMonitorStatRateTest, HeapMonitoringStatObjectCorrectnessTest, and the HeapMonitoringStatArrayCorrectnessTest tests J) Has been measured using the Dacapo benchmarks: - with the system turned off there is no performance regression - with the system turned on at 512kb sampling rate and no callback action (ie the JVM calls SampledObjectAlloc in the native agent but the call is empty), there is a 1% overhead - with the system turned on at 512kb sampling rate with the callback system used in the tests, there is a 3% overhead

31-03-2018

Update on the criterias with current tests in the webrevs (as of http://cr.openjdk.java.net/~jcbeyler/8171119/webrev.07/): A) All new JVMTI functions tested - This is done via the various tests but especially HeapMonitoringStatObjectCorrectnessTest checks the general use-case; HeapMonitoringFrequentTest; HeapMonitoringRecentTest, and HeapMonitorStatRateTest test all new JVMTI functions. B) All common use-cases tested - As said in (A), HeapMonitoringStatObjectCorrectnessTest and HeapMonitoringStatArrayCorrectnessTest cover the most common use-case - As a corner case, HeapMonitorThreadOnOffTest handles the thread safety of turning the system on/off and multiple threads hitting the allocation storage system C) Accuracy testing of the sampling - This is handled by the HeapMonitorStatRateTest, HeapMonitoringStatObjectCorrectnessTest, and the HeapMonitoringStatArrayCorrectnessTest tests D) Stability proven via being used in some simple application E) Proven to work with all openjdk supported GCs via the tests - This is handled by the HeapMonitorStatObjectCorrectnessParallelTest, HeapMonitorStatObjectCorrectnessSerialTest, HeapMonitorStatObjectCorrectnessTest (handling G1), HeapMonitorStatObjectCorrectnessCMSTest (which can be deprecated when CMS is gone). E) Test pass rate 100% - Currently deflaking it F) Test stability >99% - Currently deflaking it We are running the tests 1000 amount of times in a row to ensure reaching A+B criteria. I) Show that multiple threads can allocate while a thread turns the feature on/off. - Done via HeapMonitorThreadOnOffTest J) Show that the data is sane via the stacktraces collected and amount of data shown to be collected. - This shows the accuracy of the sampling: HeapMonitorStatRateTest, HeapMonitoringStatObjectCorrectnessTest, and the HeapMonitoringStatArrayCorrectnessTest tests

08-02-2018