k6bench

README

k6bench

Run k6 scenarios as Go benchmarks.

A k6 script becomes a testing.B benchmark: k6 metrics are reported through b.ReportMetric in benchstat-compatible units, k6 checks and thresholds become test failures, and parameter grids materialize as sub-benchmarks whose names carry the full configuration. go test -count=10 plus benchstat replaces duration guesswork with real replications and confidence intervals.

k6bench has no dependency on k6's code — it drives a k6 binary (the K6 option, $K6BENCH_K6, or k6 on PATH) and parses the run's own json output stream. Scripts that use xk6 extensions just need a binary built with them (xk6 build); the extensions' custom metrics are reported like any others.

Usage

package bench

import (
	"testing"

	"github.com/tmc/k6bench"
)

func BenchmarkAPI(b *testing.B) {
	k6bench.Run(b, k6bench.Script{Path: "script.js"})
}

go test -bench . -benchtime 200x -count 10 | tee new.txt
benchstat old.txt new.txt

Each measurement runs k6 once with b.N mapped to k6 iterations, so ns/op is wall time per k6 iteration:

Unit app_latency_p50_ms better=lower
Unit checks_rate better=higher
BenchmarkHello-16    200    134529 ns/op    14.00 app_bytes/op    0.09 app_latency_p50_ms    1.000 checks_rate    15.2 child_startup_ms

The Unit better= metadata lines teach benchstat which direction is an improvement, so throughput gains and latency gains both read correctly with no flags.

Measurement honesty

The design bias throughout: an instrument that produces nothing must be distinguishable from a subject that had nothing to give.

• The measured window is anchored to the first and last metric sample, so k6 startup, VU init, and setup() are excluded by construction; the excluded overhead stays visible as child_startup_ms. Warmup(n) discards cold-start iterations.
• The script's resolved options are read with k6 inspect before the run: a threshold-gated metric that produced zero samples fails the benchmark instead of passing vacuously, and a script that declares its own options.scenarios fails fast unless run AsDeclared — mapping b.N to --iterations would silently replace the declared execution, and a harness must not lie about what it ran.
• TagMetrics values are declared, never discovered: an observed undeclared value fails, a declared value with zero samples is fatal.
• A failed benchmark emits no result line, so a broken run can never contaminate a benchstat corpus.
• Every run records provenance as test attributes: script SHA-256, the k6 binary's version, and the exact execution shape.

Scenarios and grids

Grid runs a script at every point of a parameter cross product, one sub-benchmark per point, each axis in the name (BenchmarkX/streams=4/msg_size=1024) so configurations can never pool under one label. Open-model scripts (constant-arrival-rate and friends) run exactly as written with AsDeclared() and -benchtime 1x; dropped_iterations is always reported, an explicit zero meaning the system kept up with the declared schedule.

Artifacts and outputs

Every run leaves its evidence in b.ArtifactDir() (keep it with go test -artifacts): the full metric-sample stream as gzipped k6 JSON lines — the very stream k6bench measures from — and the k6 log. Output("name=arg") adds any further k6 output the binary supports, including xk6 output extensions it was built with; OutputEnv passes a declared variable through the otherwise-frozen k6 environment. See examples/outputs.

Examples

• examples/basic — minimal wiring against a local server
• examples/grid — parameter grids and per-tag breakdowns
• examples/asdeclared — an open-model scenario run as declared
• examples/outputs — the artifact evidence trail and extra outputs

Limitations

• A k6 binary must be present at bench time; k6bench does not build one. xk6 build remains the way to get extension-bearing binaries.
• Bytes() MB/s is computed from the parent's timer and so includes k6 startup; prefer script-defined throughput metrics, whose units use the sample-anchored window.
• The API is v0 and may still move.

License

MIT

Documentation

Overview

Package k6bench runs k6 scenarios as Go benchmarks.

A k6 script becomes a testing.B benchmark: k6 metrics are reported through b.ReportMetric in benchstat-compatible units, k6 checks and thresholds become test failures, and parameter grids materialize as sub-benchmarks whose names carry the full configuration.

func BenchmarkAPI(b *testing.B) {
	k6bench.Run(b, k6bench.Script{Path: "script.js"})
}

Each measurement runs a k6 binary — the K6 option, the K6BENCH_K6 environment variable, or k6 on PATH — with b.N mapped to k6 iterations, so ns/op is wall time per k6 iteration and benchstat can compare runs directly. k6bench imports nothing of k6: the measurement is parsed from the run's json output stream, which is also kept in the benchmark's artifact directory as the run's complete evidence trail. Scripts that use xk6 extensions need a binary built with them (xk6 build); outputs the binary was built with are reachable via Output.

Index

• func Grid(b *testing.B, s Script, axes Axes, opts ...Option)
• func Run(b *testing.B, s Script, opts ...Option)
• type Axes
• type Axis
• type Mode
• type Option
  • func AsDeclared() Option
  • func Bytes(metric string) Option
  • func CheckMode(m Mode) Option
  • func HigherIsBetter(metrics ...string) Option
  • func K6(path string) Option
  • func K6Args(args ...string) Option
  • func LowerIsBetter(metrics ...string) Option
  • func Metrics(names ...string) Option
  • func OmitTime() Option
  • func Output(spec string) Option
  • func OutputEnv(key, value string) Option
  • func Percentiles(p ...float64) Option
  • func Setup(f func() error) Option
  • func SetupEnv(f func() (map[string]string, error)) Option
  • func TagMetrics(metric, tagKey string, values ...string) Option
  • func Teardown(f func() error) Option
  • func ThresholdMode(m Mode) Option
  • func VUs(n int) Option
  • func Warmup(n int) Option
• type Script

Functions

func Grid

func Grid(b *testing.B, s Script, axes Axes, opts ...Option)

Grid runs the script at every point of the axes' cross product, as sub-benchmarks named from the axes in declaration order, e.g. BenchmarkX/streams=4/msg_size=1024. Each point's env values are set in __ENV on top of Script.Env.

func Run

func Run(b *testing.B, s Script, opts ...Option)

Run executes the script as the body of b. b.N maps to k6 iterations (shared-iterations across the configured VUs), so ns/op is wall time per k6 iteration. Under AsDeclared the script's own scenarios run once instead.

Types

type Axes

type Axes []Axis

Axes is an ordered set of grid dimensions. Sub-benchmark names list every axis in declaration order, so a configuration can never pool with another under one name.

type Axis

type Axis struct {
	Env    string
	Values []string
}

Axis is one grid dimension: an __ENV variable and its values.

type Mode

type Mode int

Mode selects how check or threshold failures are treated.

const (
	Fail   Mode = iota // report the failure via b.Error (default)
	Report             // report metrics only; failures do not fail the benchmark
	Ignore             // checks only: do not even report
)

type Option

type Option func(*config)

Option configures Run and Grid.

func AsDeclared

func AsDeclared() Option

AsDeclared runs the script's own scenarios exactly as written instead of mapping b.N to iterations. The scenario executes once per benchmark; run with -benchtime 1x. Metrics are reported in absolute units, and dropped_iterations is always reported.

func Bytes

func Bytes(metric string) Option

Bytes designates the counter whose sum is passed to b.SetBytes, making benchstat print MB/s. Never auto-detected.

func CheckMode

func CheckMode(m Mode) Option

CheckMode sets how failed k6 checks are treated (default Fail).

func HigherIsBetter

func HigherIsBetter(metrics ...string) Option

HigherIsBetter declares that larger values of the named k6 metrics are better (throughput-like). Every reported unit derived from them gets a "Unit <name> better=higher" metadata line, so benchstat colors their deltas correctly. Undeclared metrics get no polarity (except time-valued units, which are better=lower automatically).

func K6

func K6(path string) Option

K6 sets the k6 binary to run. The default is the K6BENCH_K6 environment variable, then k6 on PATH. Scripts that use xk6 extensions need a binary built with them.

func K6Args

func K6Args(args ...string) Option

K6Args appends raw arguments to the child's k6 run invocation. Escape hatch; anything load-bearing deserves a real Option.

func LowerIsBetter

func LowerIsBetter(metrics ...string) Option

LowerIsBetter declares that smaller values of the named k6 metrics are better (latency- or error-like). See HigherIsBetter.

func Metrics

func Metrics(names ...string) Option

Metrics restricts reporting to the named metrics. The default reports every custom (script- or extension-defined) metric.

func OmitTime

func OmitTime() Option

OmitTime blanks the ns/op column (testing omits a zero ns/op from the result line). Use when wall time per iteration is not the quantity under study and would invite misreading — the script's own metrics remain the reported observables. AsDeclared implies it.

func Output

func Output(spec string) Option

Output adds a k6 output to the run, in k6 --out syntax: "csv=out.csv", or "name=arg" for an output extension the k6 binary was built with. The json output k6bench measures from always runs; outputs are additive.

func OutputEnv

func OutputEnv(key, value string) Option

OutputEnv passes one environment variable through to the k6 process, for outputs configured by environment. The k6 env is otherwise minimal by design: the script sees only Script.Env, and nothing the benchmark did not declare leaks in.

func Percentiles

func Percentiles(p ...float64) Option

Percentiles sets which trend percentiles are reported (default 50, 99).

func Setup

func Setup(f func() error) Option

Setup registers a function run before the k6 child starts, outside the timer. Use it to start servers the script targets.

func SetupEnv

func SetupEnv(f func() (map[string]string, error)) Option

SetupEnv registers a function run before each k6 run, after Setup, whose returned variables are merged over Script.Env for that run (the caller's map is not modified). Use it for values that exist only once setup has run — a listen address, a connection ticket — so the script receives them through __ENV like any declared input. The merged environment is also what k6 inspect sees.

func TagMetrics

func TagMetrics(metric, tagKey string, values ...string) Option

TagMetrics additionally reports metric broken down by a declared tag key. Values are declared, never discovered: an observed undeclared value fails the benchmark, and a declared value with zero samples is fatal (the anti-vacuous rule applies per tag value).

func Teardown

func Teardown(f func() error) Option

Teardown registers a function run after the k6 child exits, outside the timer.

func ThresholdMode

func ThresholdMode(m Mode) Option

ThresholdMode sets how crossed k6 thresholds are treated (default Fail).

func VUs

func VUs(n int) Option

VUs sets the VU count for the injected shared-iterations scenario (default 1). Ignored under AsDeclared.

func Warmup

func Warmup(n int) Option

Warmup runs n extra iterations before the measured b.N. The collector discards everything they produce — metrics, checks, and the measurement window all start after them — so cold-start effects (goja warmup, first connections) stay out of the numbers. A check that fails only during warmup is therefore not reported; keep n small. Incompatible with AsDeclared.

type Script

type Script struct {
	Path   string            // script file, or
	Source []byte            // inline source (Path, if set, is used as the display name)
	Env    map[string]string // becomes __ENV; the child process env carries nothing else
}

Script names a k6 script and its inputs.