bench

Documentation

Overview

Package bench implements the budget-gated benchmark suite for graphi (story SW-010). It measures four metrics at the owning-layer boundaries, compares each against a pinned, version-stamped budget in bench-budget.yml, and emits a machine-readable report. The suite is hermetic: it uses only loopback/local resources (temp files + the pure-Go modernc SQLite backend), runs under CGO_ENABLED=0, and performs no network I/O (reusing the SW-008 posture).

This file holds a CONSTRAINED YAML reader for the bench-budget.yml schema. It is intentionally NOT a general YAML parser: it supports block mappings with 2-space indentation, scalar leaves (ints, quoted strings, bare strings), blank lines, and '#' comments. It exists so the suite reads its manifest without adding a dependency, keeping the default module hermetic for SW-013 packaging. Any unsupported construct returns an error so a malformed manifest fails loudly rather than silently.

Index

• Constants
• func Median(samples []time.Duration) time.Duration
• func P95(samples []time.Duration) time.Duration
• type Comparator
• type GateReport
  • func Gate(measured map[string]float64, man *Manifest) GateReport
  • func (r GateReport) FormatFailure() string
• type HarnessConfig
• type Manifest
  • func LoadManifest(path string) (*Manifest, error)
• type MetricBudget
• type MetricResult
• type Metrics
  • func Run(ctx context.Context, cfg HarnessConfig) (Metrics, error)
  • func (m Metrics) Map() map[string]float64
• type ProfileMetric
• type Severity

Constants

const (
	ExternalBinaryBuildContract = "external-binary/unverified"
	CustomBuildContract         = "custom-build/unverified"
)

Functions

func Median

func Median(samples []time.Duration) time.Duration

Median returns the median duration of samples. Empty input returns zero.

func P95

func P95(samples []time.Duration) time.Duration

P95 returns the 95th-percentile duration (nearest-rank) of samples. Empty input returns zero.

Types

type Comparator

type Comparator string

Comparator is the comparison a metric must satisfy against its budget. All benchmark metrics use CmpLE: the measured value must not exceed the budget.

const CmpLE Comparator = "<="

type GateReport

type GateReport struct {
	Pass    bool           `json:"pass"`
	Results []MetricResult `json:"results"`
	Failed  []string       `json:"failed,omitempty"`
}

GateReport is the full budget-gate outcome.

func Gate

func Gate(measured map[string]float64, man *Manifest) GateReport

Gate compares measured metrics against the manifest budgets. A metric fails when it exceeds its budget (CmpLE); the report names every failing metric and its delta versus the pinned baseline. Metrics present in the measurement but absent from the manifest are ignored; metrics in the manifest but absent from the measurement cause a gate failure (unmeasured budgeted metric).

func (GateReport) FormatFailure

func (r GateReport) FormatFailure() string

FormatFailure renders a human-readable, metric-naming failure message for a failing gate (AC: over-budget regression names the metric and its delta).

type HarnessConfig

type HarnessConfig struct {
	FixtureDir   string   // default: <module root>/bench/fixture
	Samples      int      // default 15 (plus warmup)
	Warmup       int      // default 2 (discarded cold samples)
	BinaryTarget string   // default ./cmd/graphi/
	BinaryPath   string   // if set, skip the build; provenance is read from this external Go binary
	CGOEnabled   string   // default "0"
	BuildTags    []string // build tags for the measured binary; default release.DefaultGrammarSubsetTags
}

HarnessConfig parameterizes a benchmark run.

type Manifest

type Manifest struct {
	Version         int
	BaselineVersion string
	FixtureDigest   string
	Metrics         map[string]MetricBudget
}

Manifest is the parsed, validated bench-budget.yml.

func LoadManifest

func LoadManifest(path string) (*Manifest, error)

LoadManifest reads and schema-validates the bench-budget manifest at path.

type MetricBudget

type MetricBudget struct {
	Baseline int64      // pinned reference value (delta = measured - baseline)
	Budget   int64      // fail threshold (measured must not exceed for CmpLE)
	Unit     string     // "ms" or "bytes"
	Op       Comparator // comparator, defaults to CmpLE
	Severity Severity   // severity, defaults to SeverityFail
}

MetricBudget is a single pinned metric definition in bench-budget.yml.

type MetricResult

type MetricResult struct {
	Name     string     `json:"name"`
	Measured float64    `json:"measured"`
	Baseline float64    `json:"baseline"`
	Budget   float64    `json:"budget"`
	Delta    float64    `json:"delta"` // measured - baseline
	Unit     string     `json:"unit"`
	Op       Comparator `json:"op"`
	Severity Severity   `json:"severity"`
	Pass     bool       `json:"pass"`
}

MetricResult is the per-metric gate outcome.

type Metrics

type Metrics struct {
	ColdStartP95MS   float64 `json:"cold_start_p95_ms"` // daemon/engine cold-start to first served query, P95
	FullIndexMS      float64 `json:"full_index_ms"`     // engine/ingest IngestAll over the frozen fixture, median
	FreshnessLagMS   float64 `json:"freshness_lag_ms"`  // hot-index IngestChanged + query round-trip
	BinarySizeBytes  int64   `json:"binary_size_bytes"` // size of the static default binary
	BuildContract    string  `json:"build_contract"`
	BuildGoVersion   string  `json:"build_go_version"`
	BuildGOOS        string  `json:"build_goos"`
	BuildGOARCH      string  `json:"build_goarch"`
	BuildGOAMD64     string  `json:"build_goamd64,omitempty"`
	BuildCGOEnabled  string  `json:"build_cgo_enabled"`
	BuildVCSRevision string  `json:"build_vcs_revision,omitempty"`
	BuildVCSModified string  `json:"build_vcs_modified,omitempty"`
	FixtureDigest    string  `json:"fixture_digest"`
	BaselineVersion  string  `json:"baseline_version"`
	Samples          int     `json:"samples"`
	// ProfileMetrics holds per-profile index/db/edge/query metrics keyed by
	// profile name (fast/balanced/deep). Added for EP-022 profile-aware budgets.
	ProfileMetrics map[string]ProfileMetric `json:"profile_metrics,omitempty"`
}

Metrics are the four measured benchmark values plus provenance. Durations are expressed as float milliseconds; binary size as bytes.

func Run

func Run(ctx context.Context, cfg HarnessConfig) (Metrics, error)

Run executes the full benchmark harness and returns the measured Metrics. It is hermetic: every store/meta dir is a temp dir removed at the end; the binary is built under CGO_ENABLED=0; no network I/O is performed.

func (Metrics) Map

func (m Metrics) Map() map[string]float64

Map returns the metric-name → value mapping consumed by Gate.

type ProfileMetric

type ProfileMetric struct {
	IndexMS        float64 `json:"index_ms"`
	DBSizeBytes    int64   `json:"db_size_bytes"`
	EdgeCount      int64   `json:"edge_count"`
	QueryLatencyMS float64 `json:"query_latency_ms"`
}

ProfileMetric is the measured data for a single index profile.

type Severity

type Severity string

Severity is the gate behavior for a metric: fail causes the gate to fail, warn only reports the delta but keeps the gate passing.

const (
	SeverityFail Severity = "fail"
	SeverityWarn Severity = "warn"
)