cob

README

Abstract

cob compares benchmarks between the latest commit (HEAD) and the previous commit (HEAD{@1}). The program will fail if the change in score is worse than the threshold. This tools is suitable for CI/CD to detect a regression of a performance automatically.

cob runs go test -bench before and after commit internally, so it depends on go command.

CAUTION: Note that git reset is executed when you run cob. You should commit all changes before running cob.

Table of Contents

• Abstract
• Continuous Integration (CI)
  • GitHub Actions
  • Travis CI
  • CircleCI
• Example
  • Run only those benchmarks matching a regular expression
  • Show only benchmarks with worse score
  • Specify a threshold
  • Specify a base commit compared with HEAD
  • Compare only memory allocation
• Usage
• Q&A
  • A result of benchmarks is unstable

Continuous Integration (CI)

See cob-example for details.

GitHub Actions

name: Bench
on: [push, pull_request]
jobs:
  test:
    name: Bench
    runs-on: ubuntu-latest
    steps:

    - name: Set up Go 1.13
      uses: actions/setup-go@v1
      with:
        go-version: 1.13
      id: go

    - name: Check out code into the Go module directory
      uses: actions/checkout@v1

    - name: Install cob
      run: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

    - name: Run Benchmark
      run: cob

Travis CI

dist: bionic
language: go
go:
  - 1.13.x

before_script:
  - curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin

script:
  - cob

CircleCI

version: 2
jobs:
  bench:
    docker:
      - image: circleci/golang:1.13
    steps:
      - checkout
      - run:
          name: Install cob
          command: curl -sfL https://raw.githubusercontent.com/knqyf263/cob/master/install.sh | sudo sh -s -- -b /usr/local/bin
      - run:
          name: Run cob
          command: cob
workflows:
  version: 2
  build-workflow:
    jobs:
      - bench

Example

Override a command to measure benchmarks

To measure benchmarks by make bench, you can use -bench-cmd and -bench-args options.

$ cob -bench-cmd make -bench-args bench

Run only those benchmarks matching a regular expression

$ cob -bench-args "test -bench Append -benchmem ./..."

Result

2020/01/12 17:32:30 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:32:32 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      117 B/op     |
+                             +----------+---------------+-------------------+
|                             | HEAD@{1} |  115.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 55.65%  |      408.70%      |
+-----------------------------+---------+-------------------+

Show only benchmarks with worse score

$ cob -only-degression

Result

2020/01/12 17:48:35 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD{@1}
2020/01/12 17:48:38 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 | 52.34%  |      347.83%      |
+-----------------------------+---------+-------------------+

2020/01/12 17:48:39 This commit makes benchmarks worse

Specify a threshold

The following option means the program fails if a benchmark score gets worse than 50%.

$ cob -threshold 0.5 ./...

Specify a base commit compared with HEAD

By default, cob uses HEAD~1. If you compare benchmarks with different commit, you can use --base option.

$ cob --base origin/master ./...

Compare only memory allocation

You can use -compare option.

$ cob -compare B/op

Result

2020/01/15 14:46:31 Run Benchmark: 4363944cbed3da7a8245cbcdc8d8240b8976eb24 HEAD~1
2020/01/15 14:46:33 Run Benchmark: 599a5523729d4d99a331b9d3f71dde9e1e6daef0 HEAD

Result
======

+-----------------------------+----------+---------------+-------------------+
|            Name             |  Commit  |    NsPerOp    | AllocedBytesPerOp |
+-----------------------------+----------+---------------+-------------------+
| BenchmarkAppend_Allocate-16 |   HEAD   |  179.00 ns/op |      121 B/op     |
+                             +----------+---------------+-------------------+
|                             | HEAD@{1} |  104.00 ns/op |      23 B/op      |
+-----------------------------+----------+---------------+-------------------+
|      BenchmarkCall-16       |   HEAD   |   0.50 ns/op  |       0 B/op      |
+                             +----------+---------------+                   +
|                             | HEAD@{1} |   0.49 ns/op  |                   |
+-----------------------------+----------+---------------+-------------------+

Comparison
==========

+-----------------------------+---------+-------------------+
|            Name             | NsPerOp | AllocedBytesPerOp |
+-----------------------------+---------+-------------------+
| BenchmarkAppend_Allocate-16 |    -    |      426.09%      |
+-----------------------------+---------+-------------------+
|      BenchmarkCall-16       |    -    |       0.00%       |
+-----------------------------+---------+-------------------+

2020/01/15 14:46:35 This commit makes benchmarks worse

Skip running cob

If your commit message contains [skip cob], cob is skipped.

$ git add README.md
$ git commit -m "[skip cob] update README.md"
$ cob
2020/04/19 12:46:57 [skip cob] is detected, so the benchmark is skipped

Usage

NAME:
   cob - Continuous Benchmark for Go project

USAGE:
   cob [global options] command [command options] [arguments...]

COMMANDS:
   help, h  Shows a list of commands or help for one command

GLOBAL OPTIONS:
   --only-degression   Show only benchmarks with worse score (default: false)
   --threshold value   The program fails if the benchmark gets worse than the threshold (default: 0.2)
   --base value        Specify a base commit compared with HEAD (default: "HEAD~1")
   --compare value     Which score to compare (default: "ns/op,B/op")
   --bench-cmd value   Specify a command to measure benchmarks (default: "go")
   --bench-args value  Specify arguments passed to -cmd (default: "test -run '^$' -bench . -benchmem ./...")
   --help, -h          show help (default: false)

Q&A

Benchmarks with the same name

Specify a package name.

$ cob -bench-args "test -bench . -benchmem ./foo" 
$ cob -bench-args "test -bench . -benchmem ./bar" 

A result of benchmarks is unstable

You can specify -benchtime.

$ cob -bench-args "test -bench . -benchmem -benchtime 10s ./..." 

License

This repository is available under the MIT

Author

Teppei Fukuda (knqyf263)