oasis-test-runner

command
v0.2202.5 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 23, 2023 License: Apache-2.0 Imports: 7 Imported by: 0

README

Oasis Test Runner

The Oasis Test Runner initializes and executes different types of tests (e.g. end-to-end, remote signer) also referred to as scenarios.

To list all available scenarios and their corresponding parameters, use:

oasis-test-runner list

If no flags are provided, all scenarios are executed.

To run a specific scenario, e.g e2e/runtime/runtime-dynamic, pass the --scenario flag (or its short version, -s) followed by scenario's name:

oasis-test-runner --scenario e2e/runtime/runtime-dynamic

Benchmarking

To benchmark scenarios, set the --metrics.address flag to the address of the Prometheus push gateway along with the --metrics.interval flag. Additionally, you can set scenario-specific parameters and the number of runs of each scenario with the --num_runs flag.

Benchmark analysis with oasis-test-runner cmp command

The cmp sub-command connects to the Prometheus server instance containing benchmark results generated by Oasis Test Runner and the corresponding Oasis Node workers. It compares the benchmark results of the last (source) and pre-last (target) scenario batch (also called instance). You need to pass the address of the Prometheus query server using --metrics.address flag, e.g.:

oasis-test-runner cmp \
  --metrics.address http://prometheus.myorg.com:9090

By default, the oasis-test-runner cmp command will fetch the results of all scenarios and metrics supported by the Oasis Test Runner. If you want to compare specific metric(s) and scenario(s), set the --metrics and --scenario flags appropriately, e.g.:

oasis-test-runner cmp \
  --metrics time \
  --scenario e2e/runtime/multiple-runtimes

The cmp sub-command takes thresholds for each metric into account. Currently, avg_ratio and max_ratio are supported which correspond to the average and maximum ratio of metric values of all scenario runs in the benchmark batch.

For each ratio, you can set the max_threshold and min_threshold. The former specifies which ratio should not be exceeded and the latter which ratio must be reached. If a value is not within these thresholds, the oasis-test-runner cmp command will exit with an error code.

This is useful for integration into CI pipelines. Thresholds can be set using the following flag specification:

--{min|max}_threshold.<metric name>.{avg|max}_ratio

For example:

oasis-test-runner cmp \
  --metrics time \
  --scenario e2e/runtime/multiple-runtimes \
  --max_threshold.time.avg_ratio 1.1

will require that the average duration of the scenario in the last benchmark batch should be at most 10% slower than from the pre-last benchmark batch.

If you are developing or improving a feature in a separate git branch, you will want to perform benchmarks and compare the results of your branch to the ones from the master branch. The Oasis Test Runner automatically sends information of the git branch it was compiled on to Prometheus, so all you need to do is perform benchmarks on both, the master and a feature branch, using the same Prometheus instance.

Afterwards, pass the name of source and target branch names to the cmp sub-command and it will compare the last benchmark batch from each git branch (respectively):

oasis-test-runner cmp \
  --metrics.source.git_branch=feature_branch \
  --metrics.target.git_branch=master

For detailed help, run:

oasis-test-runner cmp --help

Documentation

Overview

Oasis network integration test harness.

Directories

Path Synopsis
cmd
Package cmd implements the commands for the test-runner executable.
Package cmd implements the commands for the test-runner executable.
cmp
common
Package common contains common constants and variables.
Package common contains common constants and variables.
Package env defines a scenario environment.
Package env defines a scenario environment.
Package log provides utilities for watching log files.
Package log provides utilities for watching log files.
Package oasis provides the Oasis network/node/client related test helpers.
Package oasis provides the Oasis network/node/client related test helpers.
cli
Package cli contains helpers for various oasis-node subcommands.
Package cli contains helpers for various oasis-node subcommands.
Package rust contains a Go interface to the Rust build system.
Package rust contains a Go interface to the Rust build system.
Package scenario implements the test scenario abstract interface.
Package scenario implements the test scenario abstract interface.
e2e
Package e2e implements the Oasis e2e test scenarios.
Package e2e implements the Oasis e2e test scenarios.
pluginsigner
Package pluginsigner implements the Oasis plugin-signer test scenario.
Package pluginsigner implements the Oasis plugin-signer test scenario.
pluginsigner/example_signer_plugin
Package main implements an example oasis-node signer plugin.
Package main implements an example oasis-node signer plugin.
remotesigner
Package remotesigner implements the Oasis remote-signer test scenarios.
Package remotesigner implements the Oasis remote-signer test scenarios.
signer
Package signer implements the common signer test cases.
Package signer implements the common signer test cases.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL