eval

Documentation

Overview

Package eval is the offline harness for measuring in-band tool-routing (S7a) quality — the Layer 6 evaluation called out in docs/high-tool-count-scaling.md. Given a fixed environment of MCP capabilities + tools and a user message, it runs the agent once and checks which capability groups the model chose to load (via load_tools) against an expected set. The aggregate top-1 / exact / skip rates are the empirical basis for the S7a→S7b graduation decision.

The harness is provider-agnostic and performs no network I/O itself: pass a deterministic *llm.MockProvider for CI self-tests, or a real provider for an opt-in accuracy run. See routing_eval_test.go, which gates the real-provider run behind an environment variable so CI stays hermetic.

Index

• type Report
  • func RunSuite(ctx context.Context, provider llm.Provider, scenarios []Scenario) Report
  • func (r Report) ExactAccuracy() float64
  • func (r Report) SkipRate() float64
  • func (r Report) String() string
  • func (r Report) Top1Accuracy() float64
• type Result
  • func RunScenario(ctx context.Context, provider llm.Provider, sc Scenario) Result
• type Scenario
  • func DefaultScenarios() []Scenario
• type ToolSpec

Types

type Report

type Report struct {
	Results []Result
}

Report aggregates scenario results into headline routing-quality metrics.

func RunSuite

func RunSuite(ctx context.Context, provider llm.Provider, scenarios []Scenario) Report

RunSuite runs every scenario against the same provider and returns a Report.

func (Report) ExactAccuracy

func (r Report) ExactAccuracy() float64

ExactAccuracy is the fraction of scenarios where the loaded set exactly matched the expected set (no over- or under-loading).

func (Report) SkipRate

func (r Report) SkipRate() float64

SkipRate is the fraction of scenarios that *expected a load* but where the model loaded nothing — the key risk signal for in-band routing.

func (Report) String

func (r Report) String() string

String renders a human-readable per-scenario table plus the headline metrics.

func (Report) Top1Accuracy

func (r Report) Top1Accuracy() float64

Top1Accuracy is the fraction of scenarios whose expected groups were all loaded (recall == 1.0), including correctly-skipped cases.

type Result

type Result struct {
	Name      string
	Expected  []string
	Loaded    []string
	Hit       bool // every expected group was loaded (recall == 1.0)
	Exact     bool // loaded set == expected set
	Skipped   bool // model loaded nothing despite a non-empty menu
	LoadCalls int
}

Result is the scored outcome of one scenario.

func RunScenario

func RunScenario(ctx context.Context, provider llm.Provider, sc Scenario) Result

RunScenario builds the synthetic environment, runs the agent once with the supplied provider, and scores which groups were loaded. It never calls the network itself; any network access comes from the provider.

type Scenario

type Scenario struct {
	Name     string
	Message  string
	Mode     string // "read-only" (default) or "read-write"
	Caps     []capability.Capability
	Tools    []ToolSpec
	Expected []string
	// ExpandThreshold enables S8 intra-group expansion for this scenario: any
	// group larger than this is offered tool-by-tool. 0 keeps every group
	// whole (group-level S7a). Loading an individual tool still surfaces its
	// owning group in GroupsLoaded, so Expected is scored the same way.
	ExpandThreshold int
}

Scenario is one routing eval case: a fixed capability/tool environment, a user message, and the capability groups we expect the model to load. An empty Expected means the model should load nothing (e.g. a greeting).

func DefaultScenarios

func DefaultScenarios() []Scenario

DefaultScenarios is the seed routing-eval suite. Each scenario shares the netappCaps/netappTools environment so the menu the model sees is identical across cases — only the user message (and expected routing) changes. Extend this set as real misroutes are observed in production telemetry.

type ToolSpec

type ToolSpec struct {
	Name        string
	Capability  string // capability/group ID this tool belongs to
	Description string
	ReadOnly    bool
}

ToolSpec describes one MCP tool in a scenario's synthetic environment. The owning capability ID doubles as the server name (the synthetic environment keeps a 1:1 server↔capability mapping, matching production).