grade

Documentation

Overview

Package grade scores how far an agent got toward an objective, not just whether it finished. An objective is expressed as a Ladder of ordered Milestones, each a deterministic ground-truth Check; grading awards partial credit for the rungs the agent actually reached. This turns a binary pass/fail into a dense signal: a run that finds and reproduces a fault but does not exploit it scores meaningfully above one that does nothing, which is what makes graded runs useful to learn from and to compare.

Checks evaluate against Evidence, the recorded effects of a run (its spine events) plus the answers it gave to posed questions. Grading itself is pure: the same evidence grades identically on every machine and under replay, so a recorded run can be re-graded for free when the grader improves.

Index

• Constants
• func Record(ctx context.Context, log spine.Log, stream string, g Grade) error
• type Check
  • func AnswerEquals(key, want string) Check
  • func AnswerMatches(key string, match func(answer string) bool) Check
  • func EventRecorded(eventType string, match func(payload map[string]any) bool) Check
• type Evidence
• type Grade
• type Ladder
  • func (l Ladder) Grade(ctx context.Context, ev Evidence) (Grade, error)
• type MapEvidence
  • func (m MapEvidence) Answer(key string) (string, bool)
  • func (m MapEvidence) Events() []spine.Event
• type Milestone
• type Mode
• type Rung

Constants

const (
	// EvMilestone is the event type recorded for one graded rung.
	EvMilestone = "grade.milestone"
	// EvSummary is the event type recorded for a ladder's overall partial-credit
	// score.
	EvSummary = "grade.summary"
)

Functions

func Record

func Record(ctx context.Context, log spine.Log, stream string, g Grade) error

Record appends a grade to the log under stream: one EvMilestone event per rung in ladder order, then one EvSummary event carrying the score. A grade becomes durable, auditable, and replayable on the foundation like any other effect, so a run can be re-graded by folding the stream and an improved grader can be compared against the one that produced these events. The events are written on the runtime's authority, so their actor is system.

Types

type Check

type Check func(ctx context.Context, ev Evidence) (passed bool, detail string, err error)

Check is a deterministic predicate over evidence: did this milestone's ground-truth condition hold? It returns the outcome and a short human-readable detail. A Check must not consult a clock or randomness, so a grade is reproducible.

func AnswerEquals

func AnswerEquals(key, want string) Check

AnswerEquals passes when the run's answer for key equals want after trimming surrounding whitespace and folding case. It is the posed-question primitive: ask a question whose ground-truth answer is known (where the fault is, the final token) and verify the response against it.

func AnswerMatches

func AnswerMatches(key string, match func(answer string) bool) Check

AnswerMatches passes when the run's answer for key satisfies match. Use it when an answer is not a single canonical string: several acceptable forms, or a pattern.

func EventRecorded

func EventRecorded(eventType string, match func(payload map[string]any) bool) Check

EventRecorded passes when the evidence holds a recorded event of eventType for which match returns true; a nil match accepts any event of that type. It is the ground-truth check: a milestone counts only when the effect is on the spine, not when the agent claims it happened.

type Evidence

type Evidence interface {
	// Events returns the recorded events for the run under grading.
	Events() []spine.Event
	// Answer returns the run's answer to the question posed under key, and whether
	// one was given.
	Answer(key string) (string, bool)
}

Evidence is what a Check reasons over: the ground-truth record of a run. Events are the effects already durable on the spine (files touched, exit codes, results), and Answer looks up the run's response to a posed question by key. Grading never reads the agent's narration, only this recorded truth.

type Grade

type Grade struct {
	Ladder   string
	Mode     Mode
	Rungs    []Rung
	Attained float64
	Total    float64
	Score    float64
	Reached  int
}

Grade is a ladder's partial-credit result: the per-rung outcomes, the attained and total weight, the normalized Score in [0,1], and Reached, the number of passed rungs (in Cumulative mode this is the highest rung reached).

type Ladder

type Ladder struct {
	Name       string
	Mode       Mode
	Milestones []Milestone
}

Ladder is an ordered set of milestones graded under a Mode.

func (Ladder) Grade

func (l Ladder) Grade(ctx context.Context, ev Evidence) (Grade, error)

Grade evaluates the ladder against ev and returns the partial-credit result. In Cumulative mode it stops crediting at the first failed rung and leaves the rest unreached; in Independent mode it scores every rung. An empty ladder scores 0 over a total of 0. A Check error aborts grading and is returned.

type MapEvidence

type MapEvidence struct {
	Recorded []spine.Event
	Answers  map[string]string
}

MapEvidence is a ready Evidence built from a slice of events and a map of posed answers, for tests and for adapting an in-memory run.

func (MapEvidence) Answer

func (m MapEvidence) Answer(key string) (string, bool)

Answer returns the answer posed under key, if any.

func (MapEvidence) Events

func (m MapEvidence) Events() []spine.Event

Events returns the recorded events.

type Milestone

type Milestone struct {
	Name   string
	Weight float64
	Check  Check
}

Milestone is one rung of a ladder: a named condition worth Weight toward the total, satisfied when its Check passes. Weight defaults to 1 when zero; a negative weight is treated as 0.

type Mode

type Mode int

Mode selects how a ladder accumulates credit.

const (
	// Cumulative awards credit for the leading run of passed rungs and stops at the
	// first failure: the classic progress ladder (find, then reproduce, then exploit)
	// where a later rung is meaningless without the earlier ones. Rungs after the
	// first failure are not evaluated.
	Cumulative Mode = iota
	// Independent evaluates and scores every rung on its own, awarding credit for any
	// that pass regardless of order.
	Independent
)

type Rung

type Rung struct {
	Name    string
	Weight  float64
	Passed  bool
	Reached bool
	Detail  string
}

Rung is one milestone's graded outcome. Reached reports whether the rung was evaluated (always true in Independent mode; false for rungs past the first failure in Cumulative mode).