grade

Documentation

Overview

Package grade evaluates case assertions against an agent's workspace and final output: deterministic checks (files, regexes, commands) plus an LLM judge for subjective assertions. The judge always runs through the claude CLI regardless of the model under test, so grading stays comparable across providers.

Index

• Constants
• func Assertion(ctx context.Context, a evalspec.Assertion, opts Options) (passed *bool, evidence string)
• func Describe(a evalspec.Assertion) string
• type Options
• type Runner

Constants

const DefaultJudgeModel = "claude-sonnet-4-6"

DefaultJudgeModel pins LLM-judge grading to one model so verdicts stay comparable across runs and providers under test.

Functions

func Assertion

func Assertion(ctx context.Context, a evalspec.Assertion, opts Options) (passed *bool, evidence string)

Assertion grades one assertion. passed is tri-state: nil means skipped (e.g. a required binary is not installed).

func Describe

func Describe(a evalspec.Assertion) string

Describe renders an assertion as the human-readable statement that results files carry as the expectation text (grading.json's expectations[].text). The templates are stable: committed results diff only when grading does.

Types

type Options

type Options struct {
	Runner         Runner
	Workspace      string        // the eval's throwaway workspace
	Output         string        // the agent's final response text
	ExpectedOutput string        // the eval author's success description, judge context only
	Timeout        time.Duration // shared by command assertions and the judge
	JudgeModel     string        // "" = DefaultJudgeModel
	// ToolCalls are the agent's observed tool invocations; nil when the harness
	// cannot report them (a tool_call assertion is then skipped), a non-nil
	// empty slice when it reported zero (the assertion fails).
	ToolCalls []model.ToolCall
}

Options configures grading for one eval.

type Runner

type Runner interface {
	Run(ctx context.Context, spec model.CommandSpec, timeout time.Duration,
		scan *runner.Scan) (runner.Result, error)
}

Runner runs grading subprocesses (shell commands and the judge CLI). scan is always nil here (collect mode); the *runner.Scan shape matches the agent runner so one Exec implementation serves both.