evals

Documentation

Overview

conflict_grader.go is a minimal type/interface placeholder for the conflict-detection eval category (superseded-chain resolution across memory items). Real detection logic is explicitly out of scope for Phase 6.2 — 2026-07-20 Lead re-scope: "conflict:只做最小 interface stub 佔位,真邏輯歸 P3 記憶治理".

TODO(P3): 真實 conflict 偵測邏輯歸 P3 記憶治理 — implement Grade() to walk each item's SupersededBy chain and assert it matches ExpectedConflictIDs, the same way ContextRelevanceGrader.Grade() (context_relevance_grader.go) exercises contextpack's real Assemble() pipeline.

context_relevance_grader.go wires internal/contextpack's PUBLIC API (NewAssembler + Assemble) into the evals harness against the fixtures in testdata/context_relevance_cases.json, proving three properties of the real (non-mocked) contextpack scoring/exclusion/budget pipeline:

1. Score ordering — the fixture's expected_top_ids score strictly above every other candidate via contextpack's real score() function (repo/ task match, recency, keyword match — see contextpack/scorer.go), not via the fixture's own advisory "score" field (which this grader never reads into any store).
2. Stale/superseded exclusion — a behavior rule whose content marks it "deprecated" is filtered out by retrieveBehaviorRules before scoring ever runs, the same production contract wayneblacktea's other callers rely on.
3. Budget trimming — re-running Assemble with BudgetChars sized to fit only the winning item's Summary reports the rest via Pack.Omitted rather than silently dropping them.

contextpack.NewAssembler wires 11 narrow, consumer-owned read ports (see internal/contextpack/ports.go) rather than the 11 full domain StoreIfaces — only gtd/decision/knowledge/behaviorRule carry fixture-driven state below; the remaining 7 stubs implement just the one method their read port requires (returning zero values), since no fixture item routes through those sources.

Package evals is a deterministic, network-free evaluation harness for wayneblacktea's memory/learning behavior. Graders assert behavior against fixed fixtures under testdata/ and MUST NOT call an LLM provider or make any network request — see Phase 6 of docs/internal/wayneblacktea-2.0-development-prompt.md.

provider_eval.go wires the existing deterministic graders (context relevance, conflict placeholder, outcome learning, discipline, budget) into a single suite runnable from the standalone eval CLI (cmd/wbt-eval, Phase 6.5), independent of `go test` / `task check`.

Fixtures are embedded via go:embed rather than read with LoadFixtures (fixture_loader.go): LoadFixtures takes a *testing.T and relies on `go test` setting the process CWD to the package directory, neither of which holds for a standalone CLI invoked from an arbitrary working directory.

The client parameter threaded through ProviderEvalSuite.Run is accepted for future LLM-graded cases — TODO(Phase 6+): LLM-graded cases. Phase 6.5 wires zero provider-backed cases; every case below is the same deterministic logic `go test ./internal/evals/...` already exercises via evals_test.go/context_test.go/discipline_test.go/budget_test.go/ outcome_learning_test.go, never client.CompleteJSON.

Index

• Constants
• Variables
• func FailedOutcomeSurfaced(outcomes []outcome.Outcome, query string) bool
• func LoadFixtures[T any](t *testing.T, name string) []T
• func Run(t *testing.T, graders []Grader)
• type BudgetEnforcer
• type BudgetItem
• type CloseoutChecker
• type CloseoutSnapshot
• type ConflictCase
• type ConflictGrader
  • func (g *ConflictGrader) Grade(_ context.Context) EvalResult
• type ConflictItem
• type ContextRelevanceCase
• type ContextRelevanceGrader
  • func (g *ContextRelevanceGrader) Grade(ctx context.Context) EvalResult
• type ContextRelevanceItem
• type DisciplineChecker
• type EvalCase
• type EvalResult
• type Grader
• type OutcomeLearner
• type OutcomeLearningCase
• type OutcomeLearningGrader
  • func (g OutcomeLearningGrader) Grade(_ context.Context) EvalResult
• type ProviderEvalSuite
  • func (s ProviderEvalSuite) Run(ctx context.Context, _ llm.JSONClient) []EvalResult

Constants

const (
	CategoryContextRelevance = "context_relevance"
	CategoryConflict         = "conflict"
	CategoryOutcomeLearning  = "outcome_learning"
	CategoryDiscipline       = "discipline"
	CategoryBudget           = "budget"
	CategoryAll              = "all"
)

Category name constants — the --category flag values cmd/wbt-eval accepts, plus the CategoryAll aggregate.

Variables

var ProviderEvalCategories = []string{
	CategoryContextRelevance,
	CategoryConflict,
	CategoryOutcomeLearning,
	CategoryDiscipline,
	CategoryBudget,
}

ProviderEvalCategories lists every individual (non-"all") category ProviderEvalSuite recognizes, in the fixed order CategoryAll iterates them. Exported so cmd/wbt-eval can validate --category exhaustively client-side (backend-security-design.md §5.2) before ever calling Run, rather than relying on Run's internal fallback as the validation gate.

Functions

func FailedOutcomeSurfaced

func FailedOutcomeSurfaced(outcomes []outcome.Outcome, query string) bool

FailedOutcomeSurfaced reports whether a failure recorded in outcomes would surface for a caller searching with query — i.e. at least one outcome with Result == "failure" whose Notes contains query as a substring. This models the retrospection path (find_failed_patterns / ListFailedOutcomes, internal/outcome/iface.go) that must never silently swallow a failure: if the failure's own notes mention the thing being searched for, the caller MUST be able to find it.

func LoadFixtures

func LoadFixtures[T any](t *testing.T, name string) []T

LoadFixtures reads and JSON-decodes a fixture file from internal/evals/testdata into a slice of T. name MUST be a bare filename with no path separators or ".." segments — this guards against a fixture name turning into a path-traversal read of arbitrary files on disk (backend-security-design.md §2.2).

func Run

func Run(t *testing.T, graders []Grader)

Run executes every grader and reports failures through t. It is the standard entrypoint for wiring graders into `go test` / `task check`.

Types

type BudgetEnforcer

type BudgetEnforcer interface {
	Apply(items []BudgetItem, budgetBytes int) (included []BudgetItem, omittedCount int)
}

BudgetEnforcer decides which items fit within budgetBytes and reports how many were omitted. Implementations MUST report every omission — silently dropping items without surfacing omittedCount would let a caller believe it received the complete result set (dispatch threat surface: unbounded retrieval / silent omission).

type BudgetItem

type BudgetItem struct {
	ID        string
	SizeBytes int
	Priority  int
}

BudgetItem is one retrievable context item competing for inclusion within a fixed byte budget. Priority is the ranking signal (higher = more important); SizeBytes is its cost against the budget.

type CloseoutChecker

type CloseoutChecker interface {
	IsComplete(CloseoutSnapshot) bool
}

CloseoutChecker reports whether a closeout snapshot represents a "nothing left open" state.

type CloseoutSnapshot

type CloseoutSnapshot struct {
	OpenTaskCount    int
	PendingProposals int
	HandoffSet       bool
}

CloseoutSnapshot is a point-in-time view of session-end state, mirroring the fields internal/mcp/tools_closeout.go's closeoutReport aggregates (OpenTaskCount, PendingProposals, HandoffSet) without importing the mcp package — evals stays dependency-free of the server it evaluates.

type ConflictCase

type ConflictCase struct {
	ID                  string         `json:"id"`
	Items               []ConflictItem `json:"items"`
	ExpectedConflictIDs []string       `json:"expected_conflict_ids"`
}

ConflictCase mirrors testdata/conflict_cases.json's shape so LoadFixtures[ConflictCase] can decode it today, ahead of P3's real detector.

type ConflictGrader

type ConflictGrader struct {
	Case ConflictCase
}

ConflictGrader wraps a ConflictCase so it satisfies Grader for wiring purposes only. Grade below intentionally performs no conflict detection.

func (*ConflictGrader) Grade

func (g *ConflictGrader) Grade(_ context.Context) EvalResult

Grade is a placeholder: it does not walk SupersededBy chains or compare against ExpectedConflictIDs — that is P3's responsibility (see TODO above). It always reports Passed: true so wiring this grader into the harness today never fails task check; the Reason makes the placeholder nature explicit so a passing result here is never mistaken for a verified conflict-detection pass.

type ConflictItem

type ConflictItem struct {
	ID           string  `json:"id"`
	Content      string  `json:"content"`
	SupersededBy *string `json:"superseded_by"`
}

ConflictItem is one candidate memory item in a conflict fixture case.

type ContextRelevanceCase

type ContextRelevanceCase struct {
	ID             string                 `json:"id"`
	Description    string                 `json:"description"`
	Items          []ContextRelevanceItem `json:"items"`
	ExpectedTopIDs []string               `json:"expected_top_ids"`
}

ContextRelevanceCase is one fixture scenario loaded via LoadFixtures[ContextRelevanceCase](t, "context_relevance_cases.json").

type ContextRelevanceGrader

type ContextRelevanceGrader struct {
	Case ContextRelevanceCase
}

ContextRelevanceGrader evaluates one ContextRelevanceCase through contextpack's real Assemble() pipeline.

func (*ContextRelevanceGrader) Grade

func (g *ContextRelevanceGrader) Grade(ctx context.Context) EvalResult

Grade builds an Assembler from the case's fixture items, runs it through contextpack's real Assemble() twice — once with a generous budget to check score ordering and exclusion, once with a budget sized to exactly fit the winning item to check budget-trim reporting — and returns the combined verdict.

type ContextRelevanceItem

type ContextRelevanceItem struct {
	ID      string  `json:"id"`
	Source  string  `json:"source"`
	Content string  `json:"content"`
	Score   float64 `json:"score"`
}

ContextRelevanceItem is one candidate item in a context-relevance fixture case. Score documents the fixture author's intended relative ranking for human readers only — the grader never wires it into any store field; ranking is instead proven through contextpack's own, real score() computation (see buildContextRelevanceFixture).

type DisciplineChecker

type DisciplineChecker interface {
	RequiresVerification(toolName string) bool
}

DisciplineChecker reports whether an MCP tool name requires a preceding verification (log_decision / confirm_plan) before its call is considered non-drifting. Graders depend on this interface rather than a concrete type so a test can swap in a fake without touching the live allowlists.

type EvalCase

type EvalCase struct {
	ID          string
	Description string
	Category    string
}

EvalCase identifies a single evaluation scenario. Category groups cases for reporting (e.g. "context_relevance", "conflict", "outcome_learning").

type EvalResult

type EvalResult struct {
	CaseID string
	Passed bool
	Reason string
}

EvalResult is the outcome of grading one EvalCase.

type Grader

type Grader interface {
	Grade(ctx context.Context) EvalResult
}

Grader evaluates a fixture-backed scenario deterministically. Grade takes a context.Context — not *testing.T — so the same grader can run both inside `go test` and from the standalone eval CLI (cmd/wbt-eval, Phase 6.5) where no testing.T is available. Grade MUST NOT make network calls or invoke an LLM provider.

type OutcomeLearner

type OutcomeLearner interface {
	// ShouldProposeSkill reports whether outcomes contain enough repeated
	// success evidence to warrant proposing a skill candidate.
	ShouldProposeSkill(outcomes []outcome.Outcome) bool
}

OutcomeLearner decides whether a set of recorded outcomes is strong enough evidence to propose promoting a procedure to a skill candidate, using an invented deterministic stub built for grading purposes. This interface — and stubOutcomeLearner below — is NOT wired to and does NOT validate internal/scheduler.runKnowledgeToSkillCandidate, the real job that creates pending_proposal rows. The real job selects knowledge_items where recall_count > 3 (cognitive_jobs.go:457); it has no notion of outcome.Outcome{Result:"success"} counts at all. This interface exists so the eval harness can exercise a *decision-logic shape* deterministically without pulling in the scheduler, a DB pool, or an LLM provider (package doc, evals.go).

type OutcomeLearningCase

type OutcomeLearningCase struct {
	CaseID          string
	SkillName       string
	Outcomes        []outcome.Outcome
	ExpectCandidate bool
}

OutcomeLearningCase is a single grading scenario for the outcome-learning eval category: given outcomes, does the learner propose a skill candidate, and does that match ExpectCandidate?

type OutcomeLearningGrader

type OutcomeLearningGrader struct {
	Learner OutcomeLearner
	Cases   []OutcomeLearningCase
}

OutcomeLearningGrader adapts an OutcomeLearner plus a set of cases to the shared Grader contract (evals.go) so this category runs alongside every other one through Run. A nil Learner defaults to stubOutcomeLearner.

func (OutcomeLearningGrader) Grade

func (g OutcomeLearningGrader) Grade(_ context.Context) EvalResult

Grade implements the evals.Grader contract (Grade(ctx context.Context) EvalResult). It makes no network call, no LLM call, and touches no store — it only exercises Learner against in-memory Cases.

type ProviderEvalSuite

type ProviderEvalSuite struct {
	Category string
}

ProviderEvalSuite runs the deterministic eval graders through the same Grader contract `go test` already exercises (evals.go), from outside a `go test` process — see package doc above. Category restricts which category's cases run; "" or CategoryAll runs every wired category.

func (ProviderEvalSuite) Run

func (s ProviderEvalSuite) Run(ctx context.Context, _ llm.JSONClient) []EvalResult

Run executes every grader selected by s.Category and returns one EvalResult per fixture case. client is unused today (see package doc's TODO(Phase 6+)) — it is threaded through now so cmd/wbt-eval's --provider/--model flags have a real parameter to eventually pass an llm.JSONClient into once LLM-graded cases exist, without a signature change at that point.

A fixture-loading error (embed.FS is compiled in, so this should only happen if a category's testdata file is malformed) surfaces as a single failing EvalResult rather than a panic or silent empty result, so a broken fixture is visible in the CLI's JSON output and exit code.