iter8ctl

README

Iter8ctl

Iter8 command line utility for service operators to understand and diagnose their iter8 experiments.

Iter8ctl can be used with iter8-kfserving experiments.

Installation

GO111MODULE=on GOBIN=/usr/local/bin go get github.com/iter8-tools/iter8ctl@v0.1.2

The above command installs iter8ctl under the /usr/local/bin directory. To install under a different directory, change the value of GOBIN above.

Removal

rm <path-to-install-directory>/iter8ctl

Usage

Example 1

Describe an iter8 Experiment resource object present in your Kubernetes cluster.

kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml > experiment.yaml
iter8ctl describe -f experiment.yaml

Example 2

Supply experiment YAML using console input.

kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml > experiment.yaml
cat experiment.yaml | iter8ctl describe -f -

Example 3

Periodically fetch an iter8 Experiment resource object present in your Kubernetes cluster and describe it. You can change the frequency by adjusting the sleep interval below.

while clear; do
    kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml | iter8ctl describe -f -
    sleep 10.0
done

Sample Output

The following is the output of executing iter8ctl describe -f testdata/experiment10.yaml; the testdata folder is part of the iter8ctl GitHub repo and contains sample experiments used in tests.

$ ./iter8ctl describe -f testdata/experiment10.yaml
****** Overview ******
Experiment name: experiment-1
Experiment namespace: knative-test
Target: knative-test/sample-application
Testing pattern: Canary
Deployment pattern: Progressive

****** Progress Summary ******
Experiment stage: Completed
Number of completed iterations: 8

****** Winner Assessment ******
App versions in this experiment: [sample-application-v1 sample-application-v2]
Winning version: sample-application-v2
Version recommended for promotion: sample-application-v2

****** Objective Assessment ******
+--------------------------+-----------------------+-----------------------+
|        OBJECTIVE         | SAMPLE-APPLICATION-V1 | SAMPLE-APPLICATION-V2 |
+--------------------------+-----------------------+-----------------------+
| mean-latency <= 2000.000 | true                  | true                  |
+--------------------------+-----------------------+-----------------------+
| error-rate <= 0.010      | true                  | true                  |
+--------------------------+-----------------------+-----------------------+

****** Metrics Assessment ******
+-----------------------------+-----------------------+-----------------------+
|           METRIC            | SAMPLE-APPLICATION-V1 | SAMPLE-APPLICATION-V2 |
+-----------------------------+-----------------------+-----------------------+
| request-count               |              1022.565 |               514.445 |
+-----------------------------+-----------------------+-----------------------+
| mean-latency (milliseconds) |                 5.881 |                 4.702 |
+-----------------------------+-----------------------+-----------------------+
| error-rate                  |                 0.000 |                 0.000 |
+-----------------------------+-----------------------+-----------------------+

Contributing

Documentation and code PRs are welcome. When contributing to this repository, please first discuss the change you wish to make using Issues, Discussion, or Slack.

Documentation

Overview

Package iter8ctl provides iter8's command line utility for service operators to understand and diagnose their iter8 experiments.

Installation

The following command installs `iter8ctl` under the `/usr/local/bin` directory. To install under a different directory, change the value of `GOBIN` below.

GOBIN=/usr/local/bin go install github.com/iter8-tools/iter8ctl

Usage Example 1

Describe an iter8 Experiment resource object present in your Kubernetes cluster.

kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml > experiment.yaml
iter8ctl describe -f experiment.yaml

Usage Example 2

Supply experiment YAML using console input.

kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml > experiment.yaml
iter8ctl describe -f experiment.yaml

Usage Example 3

Periodically fetch an iter8 Experiment resource object present in your Kubernetes cluster and describe it. You can change the frequency by adjusting the sleep interval below.

kubectl get experiment sklearn-iris-experiment-1 -n kfserving-test -o yaml > experiment.yaml
iter8ctl describe -f experiment.yaml

Sample output

The following is the output of executing `iter8ctl describe -f testdata/experiment8.yaml`; the `testdata` folder is part of the `iter8ctl` GitHub repo and contains sample experiments used in tests.

******
Experiment name: sklearn-iris-experiment-1
Experiment namespace: kfserving-test
Experiment target: kfserving-test/sklearn-iris

******
Number of completed iterations: 10

******
Winning version: canary

******
Objectives
+----------------------+---------+--------+
|      OBJECTIVE       | DEFAULT | CANARY |
+----------------------+---------+--------+
| mean-latency <= 1000 | true    | true   |
+----------------------+---------+--------+
| error-rate <= 0.010  | true    | true   |
+----------------------+---------+--------+
******
Metrics
+--------------------------------+---------------+---------------+
|             METRIC             |    DEFAULT    |    CANARY     |
+--------------------------------+---------------+---------------+
| 95th-percentile-tail-latency   | 330.681818182 | 310.319302313 |
| (milliseconds)                 |               |               |
+--------------------------------+---------------+---------------+
| mean-latency (milliseconds)    | 228.419047620 | 229.001070304 |
+--------------------------------+---------------+---------------+
| error-rate                     |             0 |             0 |
+--------------------------------+---------------+---------------+
| request-count                  | 117.444444445 |  57.714400001 |
+--------------------------------+---------------+---------------+

Removal

Remove `iter8ctl` as follows.

rm <GOBIN-value-used-during-install>/iter8ctl