eval

Documentation

Overview

Package eval is a lightweight, Docker-free task harness for measuring whether a prompt, tool, or model change made octo better or worse at real edits.

Each task is a local fixture repo with a hidden verify step: it clones nothing and builds no Docker images. One task = one octo agentic run + one verify command, so a suite runs in seconds.

Index

• type Options
• type Result
  • func RunTask(ctx context.Context, t Task, opt Options) Result
• type Task
  • func LoadTasks(tasksDir string) ([]Task, error)

Types

type Options

type Options struct {
	OctoBin     string        // path to the octo binary
	WorkDir     string        // scratch root; per-task copies live under it
	Model       string        // empty = octo default
	Provider    string        // empty = octo default
	MaxTurns    int           // octo --max-turns
	MaxTokens   int           // octo --max-tokens per response (0 = provider default)
	AllowNet    bool          // allow octo network access (default false)
	Timeout     time.Duration // per-task default; a task's own timeout overrides
	VerifyAfter time.Duration // cap on the verify command itself
}

Options drive a suite run.

type Result

type Result struct {
	Task     string
	Resolved bool          // verify exited 0
	Duration time.Duration // octo + verify wall-clock
	OctoLog  string        // path to octo's transcript
	Verify   string        // combined verify output (tail kept by caller)
	Err      error         // harness error (copy/octo-launch/verify-launch failed)
}

Result is the outcome of one task.

func RunTask

func RunTask(ctx context.Context, t Task, opt Options) Result

RunTask copies the fixture, drives octo, injects the hidden files, and runs verify.sh. A non-zero verify exit means unresolved, not a harness error; Err is set only when the harness itself could not complete a step.

type Task

type Task struct {
	Name   string `yaml:"name"`
	Prompt string `yaml:"prompt"`
	// Timeout caps a single octo run for this task. Empty means the suite
	// default. Parsed from a Go duration string ("5m", "90s").
	TimeoutRaw string `yaml:"timeout"`

	Timeout time.Duration `yaml:"-"`
	Dir     string        `yaml:"-"` // absolute path to the task directory
}

Task is one eval case. Its directory layout is:

<dir>/
  task.yaml      // this struct
  repo/          // fixture source octo edits (copied to a scratch dir)
  hidden/        // files injected before verify; octo never sees them
  verify.sh      // run after hidden injection; exit 0 == resolved

The hidden/ + verify.sh split is what stops octo gaming the score: the judging files don't exist in the working copy while octo edits, so it can't read or rewrite them — it only ever sees repo/.

func LoadTasks

func LoadTasks(tasksDir string) ([]Task, error)

LoadTasks scans tasksDir for immediate subdirectories that contain a task.yaml, returning them sorted by name. A subdirectory without a task.yaml is skipped silently (lets you keep a README or shared helper alongside).