checkmetrics

README

checkmetrics

Overview

The checkmetrics tool is used to check the metrics results files in JSON format. Results files are checked against configs stored in a TOML file that contains baseline expectations for the results.

checkmetrics checks for a matching results file for each entry in the TOML file with an appropriate json file extension. Failure to find a matching file is classified as a failure for that individual TOML entry.

checkmetrics continues to process all entries in the TOML file and prints its final results in a summary table to stdout.

checkmetrics exits with a failure code if any of the TOML entries did not complete successfully.

JSON file format

JSON results files only need to be valid JSON, and contain some form of numeric results that can be extracted into a string or list of numeric results using the jq JSON query tool.

baseline TOML layout

The baseline TOML file is composed of one [[metric]] section per result that is processed. Each section contains a number of parameters, some optional:

name	type	description
name	string	Filename containing results (minus .json ext.)
type	string	json (optional, json is the default)
description	string	Description of test (optional)
checkvar	string	jq query string to extract results from JSON
checktype	string	Property to check ("mean", "max" etc.)
minval	float	Minimum value the checked property should be
maxval	float	Maximum value the checked property should be
midval	float	Middle value used for percentage range check
minpercent	float	Minimum percentage from midval check boundary
maxpercent	float	Maximum percentage from midval check boundary

Supported file types

At this time only JSON formatted results files are supported.

Supported checktypes

The following checktypes are supported. All are tested to fall within the bounds set by the minval and maxval. That is:

minval <= Result <= maxval

check	description
mean	the mean of all the results extracted by the jq query
min	the minimum (smallest) result
max	the maximum (largest) result
sd	the standard deviation of the results
cov	the coefficient of variation (relative standard deviation)

Options

checkmetrics takes a number of options. Some are mandatory.

TOML base file path (mandatory)

--basefile value    path to baseline TOML metrics file

Debug mode

--debug             enable debug output in the log

Log file path

--log value         set the log file path

Metrics results directory path (mandatory)

--metricsdir value  directory containing results files

Percentage presentation mode

--percentage        present results as percentage differences

Note: Due to the way the internal comparisons are performed, if minpercent and maxpercent are not equal, then the percentage range and midpoint value shown in the results table represents the calculated midpoint between the min and max, and not the supplied midval. This makes the results appear skewed, but the calculations are correct.

Help

--help, -h          show help

Version

--version, -v       print the version

Output

The checkmetrics tool outputs a summary table after processing all metrics sections, and returns a non-zero return code if any of the metrics checks fail.

Example output:

Report Summary:
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| P/F |         NAME         |    FLR    |   MEAN    |   CEIL    |  GAP  |    MIN    |    MAX    | RNG  | COV  | ITS |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
| F   | boot-times           |      0.50 |      1.36 |      0.70 | 40.0% |      1.34 |      1.38 | 2.7% | 1.3% |   2 |
| F   | memory-footprint     | 100000.00 | 284570.56 | 110000.00 | 10.0% | 284570.56 | 284570.56 | 0.0% | 0.0% |   1 |
| P   | memory-footprint-ksm | 100000.00 | 101770.22 | 110000.00 | 10.0% | 101770.22 | 101770.22 | 0.0% | 0.0% |   1 |
+-----+----------------------+-----------+-----------+-----------+-------+-----------+-----------+------+------+-----+
Fails: 2, Passes 1

Example percentage mode output:

Report Summary:
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| P/F |         NAME         |  FLR  |  MEAN  |  CEIL  |  GAP  |  MIN   |  MAX   | RNG  | COV  | ITS |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
| *F* | boot-times           | 83.3% | 226.8% | 116.7% | 33.3% | 223.8% | 229.8% | 2.7% | 1.3% |   2 |
| *F* | memory-footprint     | 95.2% | 271.0% | 104.8% | 9.5%  | 271.0% | 271.0% | 0.0% | 0.0% |   1 |
| P   | memory-footprint-ksm | 92.7% | 99.3%  | 107.3% | 14.6% | 99.3%  | 99.3%  | 0.0% | 0.0% |   1 |
+-----+----------------------+-------+--------+--------+-------+--------+--------+------+------+-----+
Fails: 2, Passes 1

Output Columns

name	description
P/F	Pass/Fail
NAME	Name of the test/check
FLR	Floor - the minval to check against
MEAN	The mean of the results
CEIL	Ceiling - the maxval to check against
GAP	The range (gap) between the minval and maxval, as a % of minval
MIN	The minimum result in the data set
MAX	The maximum result in the data set
RNG	The % range (spread) between the min and max result, WRT min
COV	The coefficient of variation of the results
ITS	The number of results (iterations)

Example invocation

For example, to invoke the checkmetrics tool, enter the following:

BASEFILE=`pwd`/../../metrics/baseline/baseline.toml
METRICSDIR=`pwd`/../../metrics/results

$ ./checkmetrics --basefile ${BASEFILE} --metricsdir ${METRICSDIR}

See also

• CI worker reference files