eval

Documentation

Overview

Package eval is the harness the rest of ari is tested with: golden files for every renderer, offline replay sets for the loop, token budgets for the demo tasks, and goleak for every package (D23).

It exists before the features it gates. The only tree dependency it is allowed is event/, the shared vocabulary; depending on a feature package would make it circular with the code it tests, and a test asserts the import set stays that small.

Index

• func Golden(t TB, name, got string)
• func Main(m *testing.M)
• func NoLeaks(t TB)
• func Replay(t TB, s Script, drive DriveFunc)
• func SaveScript(path string, s Script) error
• func Updating() bool
• func WithinBudget(t TB, u Usage, b Budget)
• type Budget
• type DriveFunc
• type Script
  • func LoadScript(path string) (Script, error)
• type TB
• type Usage
  • func (u *Usage) Add(input, output, cacheRead int64)

Functions

func Golden

func Golden(t TB, name, got string)

Golden compares got against testdata/<name>.golden in the calling package, creating or rewriting the file under -update. Names may carry slashes for grouping (theme/dark/read_result).

func Main

func Main(m *testing.M)

Main is the TestMain every package uses so a leaked goroutine fails the build (doc 01 section 6.6):

func TestMain(m *testing.M) { eval.Main(m) }

func NoLeaks

func NoLeaks(t TB)

NoLeaks asserts no unexpected goroutine is alive right now. Tests that start and stop a component mid-test use this to catch the leak at the call site rather than at package exit.

func Replay

func Replay(t TB, s Script, drive DriveFunc)

Replay drives the script and asserts the produced events match the recorded sequence exactly, after normalization: Seq and Time are zeroed on both sides because they are assigned at run time and would make every replay a diff. Everything else, order included, must be byte-identical, which is what "the loop is deterministic" means as a test (D6, D23).

func SaveScript

func SaveScript(path string, s Script) error

SaveScript writes a replay set, indented, for the recorder path.

func Updating

func Updating() bool

Updating reports whether this test run should rewrite goldens.

func WithinBudget

func WithinBudget(t TB, u Usage, b Budget)

WithinBudget fails the test when the usage crosses the budget, naming the dimension and the overrun so a regression reads like a perf report.

Types

type Budget

type Budget struct {
	Name   string
	Input  int64 // input tokens, cache reads included
	Output int64
	Turns  int // provider calls
}

Budget is a named token ceiling for a demo task. The budgets are asserted constants, not aspirations (plan/01 section 6); zero means that dimension is unbounded.

type DriveFunc

type DriveFunc func(s Script) ([]event.Event, error)

DriveFunc runs one scripted turn and returns the events it produced. Slice 2 points this at the real core with a scripted provider; the harness itself only asserts the sequence.

type Script

type Script struct {
	// Name identifies the fixture in failures.
	Name string `json:"name"`
	// Prompt is the user turn the script answers.
	Prompt string `json:"prompt"`
	// Responses are the scripted provider replies, in the order the
	// provider will be called within the turn. The shape is owned by the
	// provider layer; the harness treats each as opaque JSON.
	Responses []json.RawMessage `json:"responses"`
	// Want is the exact event sequence the turn must emit.
	Want []event.Event `json:"want"`
}

Script is one replay set: a recorded sequence of provider responses and the events driving them through the core must produce. The format is checked-in JSON a human can read and a recorder can regenerate, so a real session is captured once and re-run offline forever (D23).

func LoadScript

func LoadScript(path string) (Script, error)

LoadScript reads a replay set from disk.

type TB

type TB interface {
	Helper()
	Errorf(format string, args ...any)
	Fatalf(format string, args ...any)
}

TB is the slice of testing.TB the helpers need. It exists so the harness self-tests can hand in a recorder and prove a drifted golden or a blown budget really fails, not just hopefully fails.

type Usage

type Usage struct {
	Input  int64
	Output int64
	Turns  int
}

Usage is what a task actually spent, summed from ledger.turn events.

func (*Usage) Add

func (u *Usage) Add(input, output, cacheRead int64)

Add accumulates one ledger turn into the usage.