eval

Documentation

Overview

artifacts.go — write the harness's per-scenario result JSON, per-scenario TRANSCRIPTS, and the classifier summary to disk (§4.4 result JSON, §4.7 transcript audit). These are the artifacts every iteration tool consumes; transcript capture is non-negotiable.

Layout under outDir:

outDir/
  report.json                       — the full HarnessReport (median, band, classifier)
  run-<i>/                           — one dir per corpus pass
    <scenario_id>.json               — graded ScenarioResult (§4.4)
    <scenario_id>.transcript.json    — full transcript: every model call (§4.7)
  classifier.json                    — the ClassifierSummary (the loop's to-do list)

backend.go — the inference-Client backends the harness drives the core with, and the per-scenario GOLDEN SCRIPT generator for the creds-free MERGE-GATE.

THE {base_url, key} PIVOT (§4.4, direct.go doc): the harness is parameterized over a single agent.Client seam. Two modes share it:

• MERGE-GATE (mode "canned"): a cannedbackend is scripted, PER SCENARIO, to return that scenario's EXPECTED verdict (golden responses). With golden responses the deterministic grader's chain_action axis PASSES for every scenario — 0 pp of model noise. This gates HARNESS + GRADER regressions: if someone breaks the loader, the runner, the verdict parser, the gate wiring, or the grader, the merge-gate goes RED even though no model ran. It is creds-free (no network beyond localhost) so CI runs it on every push.

  THE MERGE-GATE IS EXPLICITLY NOT THE ACCURACY NUMBER. Golden responses say "if the model returned the right answer, does the pipeline grade it right?" — a pipeline-integrity check. The real accuracy number comes ONLY from the real-model run, where the model decides the verdict and the eval measures how often it is correct. Reporting the merge-gate's 100% as accuracy would be a lie; the doc string and the README field both say so.

• REAL-MODEL (mode "real"): a core/inference.DirectClient pointed at MALLCOP_INFERENCE_URL with MALLCOP_API_KEY. THIS path is WIRED but DELIBERATELY NOT RUN here (no creds in this environment). RealClientFromEnv builds it; the harness CLI / test refuses to run it without both env vars set.

This file imports core/inference ONLY for the real path — it is the package boundary where the network client is allowed (core/eval is a harness, not the shipped product runtime; the import-lint guards core/, and a harness that wires the DirectClient is exactly the intended seam, mirrored on cmd/ + test/).

classifier.go — the ~40-line failure classifier (portable-agent-architecture.md §4.8). It bins each scenario into one of: PASS / I*-infra / A1/A2-algorithm / R*-rubric. The cluster name names the fix the next iteration needs — the classifier output is the to-do list for the loop.

Bins (lifted from §4.8's classifier vocabulary):

PASS                            — chain_action passed.
I3_no_inference                 — 0 model calls AND not a force-escalate route
                                  (the model was never reached → infra/wiring bug).
I2_chain_drop_no_terminal       — no terminal action recorded (chain dropped).
A1_invest_should_resolve_but_escalated — expected resolved, got escalated
                                  (over-escalation cluster, §4.3).
A2_should_escalate_but_resolved — expected escalated, got resolved
                                  (under-escalation cluster — the dangerous tail).
R_rubric_axis_fail              — chain_action passed but a non-gating axis
                                  (mentions / no_mentions) failed. Reported for
                                  provenance; does NOT count as a harness failure.

corpus.go — the SHA-PINNED, PROVENANCE-SAFE scenario corpus loader.

This is the self-extension FOUNDATION (portable-agent-architecture.md §4.1, §4.9): the eval is an INTERLOCK. Before any scenario runs, the loader asserts two integrity gates that a self-extension loop (an agent that appends new scenarios) MUST clear:

1. COUNT — the corpus has exactly the expected number of scenarios.
2. SHA — the corpus content hashes to the exact pinned digest.

A mismatch on EITHER is a HARD FAIL (Load returns an error; nothing runs). The pin lives in a committed file (corpus.pin) the loader verifies on every run. An agent that adds or edits a scenario MUST also update the pin in the same change — so a tampered or drifted corpus cannot silently change the accuracy number. This is the eval-as-interlock property the closed loop depends on.

THE LEADING-UNDERSCORE FOOTGUN (§4.10): the walker SKIPS every path component that begins with "_" — both FILES (`_schema.yaml`) and DIRECTORIES (`_test/`). exams/scenarios/ ships `_schema.yaml` (the schema, not a scenario) and a `_test/` directory (harness probes, not graded scenarios). Including either would inflate the count, corrupt the hash, and feed non-scenarios to the grader. The skip is on ANY component, at ANY depth — `a/_b/c.yaml` is skipped.

THE CANONICAL MANIFEST (the hashed artifact): the corpus SHA is NOT a hash of a tarball or a directory listing (both are order/zip/timestamp-sensitive). It is the SHA-256 of a CANONICAL MANIFEST: for each included scenario, one line

<forward-slash relpath><two spaces><lowercase hex sha256 of file bytes>\n

lines sorted lexicographically by relpath, concatenated. This is reproducible across OSes (forward slashes), independent of walk order (sorted), and independent of file mtime/permissions (content hash only). The Go loader is the SOURCE OF TRUTH for the format; corpus.pin holds the digest the loader emits.

detectfidelity.go — the LOAD-BEARING detect-fidelity accounting for -mode e2e.

THE HEADLINE RESULT e2e EXPOSES: core/detect produces DIFFERENT / FEWER / ZERO findings than the YAML finding blocks the eval modes inject. Examples the detect map proved:

• AC-01: no `new-external-access` detector exists and the event type (repo.add_collaborator) is in no detector's event-type set → ZERO findings.
• UT-03: `unusual-timing` is not reproduced (the scenario event lacks the hour-profile the detector needs); `unusual-login` fires for the actor instead → the EXPECTED `unusual-timing` finding is not reproduced.
• VA-03: volume-anomaly's frequency key does not match → ZERO findings.

So detect-fidelity is NOT incidental — it is the headline. This file MEASURES it explicitly and the harness reports it; it is NEVER silently passed or folded into the chain_action pass-rate as if the agent had decided correctly.

Per scenario, after pipeline.Run, classify into one of three outcomes by comparing detect's stored findings against the scenario's YAML finding block:

REPRODUCED  — a stored finding matches the expected (Actor, detector-family).
              The resolution for THAT finding is graded against expected
              chain_action. This is the only case that contributes a real
              agent-accuracy data point comparable to -mode real.
MISMATCH    — detect emitted finding(s) but NONE match the expected family+actor.
              We grade the closest-by-actor resolution as a "substitute" for
              provenance, but the type drift is flagged.
DETECT-MISS — detect emitted ZERO findings. No resolution to grade. A miss on an
              expected-ESCALATE scenario is a real production false-negative (the
              attack was never surfaced) — counted as an END-TO-END FAIL. A miss
              on an expected-RESOLVE scenario means nothing was flagged, which is
              the correct production outcome (nothing to escalate) — reported as
              "no-finding-correct", but still tracked separately and NEVER
              conflated with an agent resolve.

Package eval is the PORTABLE eval harness — the replatformed academy (portable-agent-architecture.md §4), lifted off the deleted legion/campfire transport onto the in-process core.

It is the most portable artifact in the project (§4.9): the corpus is the value, the substrate is replaceable. This package keeps the corpus, the deterministic grader, the median-of-N discipline, the failure classifier, and transcript capture — and drops campfire, legion, cf, rd, and the .toml.tmpl rendering entirely.

Pipeline:

Load(repoRoot)         — SHA-pinned, provenance-safe corpus loader. Skips
                         leading-underscore paths (files AND dirs). Asserts the
                         pinned count + SHA-256; a mismatch HARD-FAILS (the
                         eval-as-interlock gate the self-extension loop needs).
RunScenario(...)       — runs ONE scenario through core/agent.ResolveFindingWith
                         IN-PROCESS via a controllable Client ({base_url,key}
                         pivot), capturing a full per-scenario transcript (§4.7).
Grade(run)             — DETERMINISTIC structural grader (no LLM in pass/fail).
                         chain_action is the only gating axis; the rest are
                         reported provenance. Emits per-scenario result JSON.
Run(cfg)               — the full driver: load → N passes → median pass-rate
                         with the 8pp band → classifier summary.

Two backends share the Client seam: a creds-free cannedbackend MERGE-GATE (golden responses → deterministic green; gates harness+grader regressions; NOT the accuracy number) and a real DirectClient parity run (WIRED via RealClientFromEnv, NOT run without creds).

examdetect.go — the OFFLINE detect-layer exam (`mallcop exam-detect`).

This is the K1 keystone of the self-extension loop: a deterministic, LLM-free grader that runs the REAL core/detect.Detect over every LABELED corpus scenario and grades the emitted findings against the scenario's expected_detection ground truth (must_fire / must_not_fire detector family tokens). A labeled-and-unfixed detection gap (e.g. VA-03's volume-anomaly false negative) shows up as a RED row — the exam is the interlock the loop closes against when it grows a detector.

Grading contract:

• The corpus is loaded via Load(repoRoot) — the corpus.pin integrity interlock stays in the path (a drifted corpus runs NOTHING).
• Events/baseline are projected through the SAME unexported projections the e2e runner uses (scenarioEvents / baselineFromScenario) — detectors read payloadMeta discriminators from the projected payload, so a naive re-projection would silently blind them.
• Findings are graded on family PRESENCE over the whole emitted set (findingFamilyToken), NOT counts or actors — multi-firing detectors (e.g. unusual-timing) legitimately emit several findings per family.
• UNLABELED scenarios (nil ExpectedDetection) are skipped-but-counted: grading covers only explicit labels; the corpus-wide backfill is a deferred human decision.

grader.go — the DETERMINISTIC structural grader (portable-agent-architecture.md §4.4, §4.9). Lifted from cmd/mallcop-academy/grading.go: the LOGIC is the same (compare terminal chain_action to expected; substring mentions; tool/iteration counts), the CAMPFIRE TRANSPORT is dropped — this grades a ScenarioRun captured in-process, not a disposition chain reconstructed from campfire messages.

THE GRADER HAS NO LLM IN THE PASS/FAIL PATH. Pass/fail is the chain_action axis alone — string equality of the terminal action against expected.chain_action (plus the "escalate-or-stronger" token). §4.1's rubric-strictness study found every miss was on chain_action; mentions/tools/iterations contributed zero false failures. They are GRADED and REPORTED as structural axes (provenance for the classifier) but DO NOT gate the harness verdict. "Don't tune the rubric, tune the system."

harness.go — the end-to-end eval driver: load (SHA+count gate) → run the corpus N times → grade deterministically → report MEDIAN chain_action pass-rate with the §4.6 8pp noise band → classify failures (§4.8). MEDIAN-OF-N is mandatory: "single-run results lie" (§4.6) — a single run can swing ±10 pp on identical code, so the harness NEVER gates on one run.

reporoot.go — self-resolving repo root for the eval harness (portable-agent-architecture.md §3.5, §4 determinism via SetRepoRootForTest).

The harness reads the scenario corpus from exams/scenarios/ under the repo root. To find that directory it walks UP from the binary's own location to a project marker — NOT from CWD (the runner relocates CWD) and NOT from an env var that "should be set". A test-only override (SetRepoRootForTest) pins the root DETERMINISTICALLY so `go test -count=N -race` always resolves the same corpus regardless of where the toolchain places the test binary — the same flake-closing seam core/agent uses.

runner.go — the IN-PROCESS scenario runner (portable-agent-architecture.md §4).

Each scenario runs through the PORTABLE core (core/agent.ResolveFindingWith) IN THE SAME PROCESS — no subprocess, no campfire, no legion. The runner:

1. Builds a core finding.Finding from the scenario's finding: block, with finding.Type = scenario.Detector so the data-driven PRE-LLM floor's hard-constraint routes (priv-escalation, injection-probe, log-format-drift, ...) fire BEFORE any model call — exactly as in production (§4.3: not every finding deserves agent inference).
2. Projects the scenario's events + baseline into a deterministic ToolEvidence the cascade boxes and the structural gate scores (the eval's per-scenario baseline is what makes runs reproducible, §4.1).
3. Drives the cascade through a CONTROLLABLE inference Client — a cannedbackend for the creds-free merge-gate, a real DirectClient for the parity run (the {base_url,key} pivot). The runner is parameterized over the Client; it never dials the network itself.
4. CAPTURES A PER-SCENARIO TRANSCRIPT — every model request (system + boxed user prompt + advertised tools) and every model reply — via a recording Client wrapper. Transcript capture is NON-NEGOTIABLE (§4.7): it is how a silent-empty tool return or a bypassed channel becomes diagnosable.

The runner imports core/agent (the portable core) and pkg/finding ONLY. It does NOT import core/inference — the caller injects whatever Client it wants. That keeps the {cannedbackend ⇄ real} pivot in the caller's hands and keeps this file substrate-free.

runner_e2e.go — the END-TO-END scenario runner. Where RunScenario (runner.go) injects a SYNTHETIC finding built from the scenario's YAML finding: block and calls agent.ResolveFindingWith directly, RunScenarioE2E drives the SAME pipeline.Run the production `mallcop scan` command calls (cmd/mallcop/scan.go): connect → detect.Detect (the REAL detector fleet) → store findings → resolveAll → store resolutions. The finding the cascade resolves is WHATEVER core/detect emits over the scenario's raw events — NOT the YAML block. Tools come from the PRODUCTION core/toolrun.Runner over the same isolated store.

This is the only path that exercises connect+detect+toolrun+store together as a real scan does. Everything downstream of the verdict (Grade, median-of-N, the classifier, the corpus interlock) is SHARED with -mode real, so the e2e number is directly comparable to the -mode real number — but ONLY over the scenarios detect actually reproduces. The detect-fidelity accounting (detectfidelity.go) measures and surfaces that reproduction gap explicitly; it is the headline result e2e exposes and is NEVER silently passed or folded into the pass-rate.

PER-SCENARIO ISOLATION: each scenario gets a FRESH temp git store (os.MkdirTemp + gitInit + store.Open), torn down before the next scenario. toolrun.Runner re-reads the live store per finding, so a shared store would leak scenario A's events/findings into scenario B's tool reads. Workers:1 keeps the per-scenario resolve ordering deterministic.

core/eval is the harness layer (not the shipped runtime), so importing core/pipeline + core/toolrun + core/connect + core/store here is the intended seam — the import-lint bans only vendor SDKs / orchestration / dropped transport (see imports_test.go), none of which this adds.

scenario_tools.go — the PER-SCENARIO live ToolRunner that backs the ModeReal parity run with REAL telemetry (portable-agent-architecture.md §3.8, §4.1).

THE GAP THIS CLOSES: RunScenario drives the portable cascade (agent.ResolveFindingWith) which reaches tools ONLY through the injected agent.ToolRunner seam (CascadeOptions.Tools). Before this file the runner passed whatever single ToolRunner the caller set — nil for the merge-gate — so on ModeReal the live agent investigated with NO scenario evidence: search-events returned nothing scenario-specific, check-baseline returned nothing, and the parity number measured a model staring at an empty toolbox. That number is meaningless.

THE FIX: for EACH scenario, scenarioToolRunner seeds a per-scenario evidence source from the exam.Scenario (its events, baseline, finding) and dispatches tool calls to the REAL core/tools over THAT data — the SAME tool surface the agent gets in production:

• search-events → tools.SearchEventsWrapped over a git-backed store seeded with the scenario's events, FOLDING the operator-decisions matched_rules (§3.8) keyed on the finding family + the finding's flat metadata.
• check-baseline → tools.CheckBaseline over a baseline.Baseline reconstructed from the scenario's known_entities + frequency_tables (so the agent can answer "is this routine for this actor").
• search-findings → tools.SearchFindings over the same store (seeded with the scenario's finding) so the deeper sweep has the finding stream.

Each scenario gets its OWN runner over its OWN store + baseline: scenario A's agent sees ONLY scenario A's telemetry. That per-scenario isolation is what makes the eval reproducible (§4.1) — the baseline the gate scores is the scenario's, not a shared/static fixture.

core/eval is the harness layer (not the shipped runtime), so importing the real core/tools + core/store here is the intended seam — the import-lint bans only vendor SDKs / orchestration / dropped transport, none of which this adds.

Index

• func RealClientFromEnv() (agent.Client, error)
• func RepoRoot() (string, error)
• func SearchEventsEnvelopeFromSnapshot(events []tools.EventView, rules []tools.OperatorRule) tools.SearchEventsEnvelope
• func SetRepoRootForTest(dir string)
• func WriteArtifacts(outDir string, report HarnessReport) (int, error)
• type AxisResult
• type ChainStep
• type ClassifierSummary
  • func Classify(runs []RunResult) ClassifierSummary
• type Corpus
  • func Load(repoRoot string) (Corpus, error)
  • func (c Corpus) Manifest() string
• type CorpusPin
• type DetectFidelity
• type DetectFidelityRow
• type DetectOutcome
• type E2EOutcome
  • func RunScenarioE2E(ctx context.Context, client agent.Client, ls LoadedScenario, ...) (E2EOutcome, error)
• type ExamDetectReport
  • func RunExamDetect(repoRoot string) (ExamDetectReport, error)
  • func RunExamDetectExtra(repoRoot, extraScenariosDir string) (ExamDetectReport, error)
• type ExamDetectRow
• type ExamDetectTotals
• type FailBin
• type HarnessReport
  • func Run(ctx context.Context, cfg RunConfig) (HarnessReport, error)
• type LoadedScenario
  • func LoadExtraScenarios(dir string) ([]LoadedScenario, error)
• type Mode
• type RunConfig
• type RunResult
• type ScenarioResult
  • func Grade(run ScenarioRun) ScenarioResult
• type ScenarioRun
  • func RunScenario(ctx context.Context, client agent.Client, ls LoadedScenario, ...) ScenarioRun
• type StructuralAxes
• type TranscriptEntry

Functions

func RealClientFromEnv

func RealClientFromEnv() (agent.Client, error)

RealClientFromEnv builds the real-model DirectClient from the environment. It is the {base_url,key} pivot's REAL leg: BaseURL=MALLCOP_INFERENCE_URL, Key=MALLCOP_API_KEY, Model=MALLCOP_MODEL (optional). It returns an error when either required var is unset — the harness REFUSES to run real mode without creds, which is why no real call happens in this environment.

func RepoRoot

func RepoRoot() (string, error)

RepoRoot returns the project root that holds exams/scenarios.

Resolution order (§3.5):

1. SetRepoRootForTest override (test-only seam, deterministic).
2. The walk up from os.Executable() — the PRIMARY production path.
3. MALLCOP_REPO_ROOT env override — last resort, checked AFTER the walk so a stale env var cannot shadow a correct walk result.

func SearchEventsEnvelopeFromSnapshot

func SearchEventsEnvelopeFromSnapshot(events []tools.EventView, rules []tools.OperatorRule) tools.SearchEventsEnvelope

SearchEventsEnvelopeFromSnapshot reconstructs the minimal SearchEventsEnvelope writeSearchEvents renders — its Events and MatchedRules — from the frozen snapshot slices. It exists so RunTools renders the SAME compact transcript it always did while reading ONLY immutable snapshot data (no store re-read). Only the two fields writeSearchEvents consumes are populated; the rest of the envelope (FilterApplied, Notes) is not rendered by writeSearchEvents and is left at its zero value deliberately. The input slices are treated as read-only.

func SetRepoRootForTest

func SetRepoRootForTest(dir string)

SetRepoRootForTest pins the repo root the harness resolves, taking precedence over the os.Executable() walk. It panics if dir is non-empty and does not contain exams/scenarios, so a typo fails loudly instead of silently resolving an empty corpus. Tests defer SetRepoRootForTest("") to clear it.

Production never calls this; the walk + MALLCOP_REPO_ROOT fallback cover real deployments. Pinning the override (checked FIRST in RepoRoot) removes the non-determinism where the resolved root depends on binary placement — the reason the determinism harness (-count=10 + -race) needs an explicit seam.

func WriteArtifacts

func WriteArtifacts(outDir string, report HarnessReport) (int, error)

WriteArtifacts persists the report, per-scenario results, per-scenario transcripts, and the classifier summary under outDir. Transcripts are read from each RunResult (the single source). Returns the number of transcript files written (one per scenario per run) for caller assertions.

Types

type AxisResult

type AxisResult string

AxisResult is one structural axis outcome. Mirrors grading.go's vocabulary.

const (
	AxisPass AxisResult = "pass"
	AxisFail AxisResult = "fail"
	AxisNA   AxisResult = "n/a"
)

type ChainStep

type ChainStep struct {
	Seq   int    `json:"seq"`
	Model string `json:"model"`
	Reply string `json:"reply"`
	Err   string `json:"err,omitempty"`
}

ChainStep is one step of the captured chain (§4.3/§4.4: full chain = every sub-agent dispatch). Here, one model call = one step (model + reply snippet).

type ClassifierSummary

type ClassifierSummary struct {
	Counts map[FailBin]int `json:"counts"`
	// PerScenario maps scenario id → its bin, for transcript pull (§4.8 step 3).
	PerScenario map[string]FailBin `json:"per_scenario"`
}

ClassifierSummary aggregates bins across the worst (most-failing) run so the loop sees the dominant cluster. Counts is bin → number of scenarios.

func Classify

func Classify(runs []RunResult) ClassifierSummary

Classify bins the run with the LOWEST pass-rate (the worst case names the cluster to fix). Median-of-N reports the median rate; the classifier reports the worst run's failure shape so the loop targets the real weakness.

type Corpus

type Corpus struct {
	// Scenarios are sorted lexicographically by RelPath — deterministic order so
	// the runner, the grader, and the manifest all agree.
	Scenarios []LoadedScenario
	// SHA is the corpus digest: sha256 of the canonical manifest. It equals the
	// pinned digest (Load fails otherwise).
	SHA string
	// Count is len(Scenarios) — equals the pinned count (Load fails otherwise).
	Count int
}

Corpus is the loaded, integrity-verified scenario set.

func Load

func Load(repoRoot string) (Corpus, error)

Load scans the corpus under repoRoot, then HARD-VERIFIES it against the committed pin. This is the integrity interlock: a count OR sha mismatch returns an error and NOTHING runs. Both halves of the gate are reported in the error so a self-extension agent knows exactly what drifted.

repoRoot is the directory holding exams/scenarios (the repo root). Callers resolve it via RepoRoot (walk up from the binary) or pin it in tests via SetRepoRootForTest — the same self-locating discipline §3.5 prescribes.

func (Corpus) Manifest

func (c Corpus) Manifest() string

Manifest returns the canonical manifest (the exact bytes that get hashed): one "<relpath><two spaces><filesha>\n" line per scenario, sorted by relpath. Exposed so the pin-regeneration helper and tests can recompute the digest from the same source of truth the loader uses.

type CorpusPin

type CorpusPin struct {
	Count int
	SHA   string
}

CorpusPin is the committed integrity pin: the expected count + digest. A mismatch between the loaded corpus and this pin HARD-FAILS the run.

type DetectFidelity

type DetectFidelity struct {
	Total      int `json:"total"`
	Reproduced int `json:"reproduced"`
	Mismatch   int `json:"mismatch"`
	DetectMiss int `json:"detect_miss"`
	// ReproductionRate = Reproduced / Total. Reported PROMINENTLY: a low rate means
	// the validated -mode real number does NOT yet transfer to live scan because the
	// detector fleet cannot produce most scenario findings.
	ReproductionRate float64 `json:"reproduction_rate"`
	// EndToEndPassRate is passes / Total over ALL scenarios where DETECT-MISS-on-
	// escalate and MISMATCH-on-escalate count as fails — the honest "live scan from
	// raw events" number. Far below the agent-reasoning rate when reproduction is low.
	EndToEndPassRate float64 `json:"end_to_end_pass_rate"`
	// Rows is the per-scenario fidelity detail.
	Rows []DetectFidelityRow `json:"rows"`
}

DetectFidelity is the report block aggregating per-scenario fidelity (§e2e).

type DetectFidelityRow

type DetectFidelityRow struct {
	ScenarioID string `json:"scenario_id"`
	// ExpectedDetector is the scenario's YAML finding detector family (the family
	// the eval modes would inject).
	ExpectedDetector string `json:"expected_detector"`
	// ExpectedActor is the scenario's finding actor.
	ExpectedActor string `json:"expected_actor"`
	// ExpectedAction is the scenario's expected chain_action ("escalated" /
	// "resolved" / "escalate-or-stronger").
	ExpectedAction string `json:"expected_action"`
	// EmittedDetectors lists the (detector-family/actor) of every finding detect
	// emitted for this scenario — the visible drift.
	EmittedDetectors []string `json:"emitted_detectors"`
	// Outcome is the fidelity bucket.
	Outcome DetectOutcome `json:"outcome"`
	// MatchedFindingID is the id of the stored finding that matched (REPRODUCED) or
	// the closest-by-actor substitute (MISMATCH). Empty on DETECT-MISS.
	MatchedFindingID string `json:"matched_finding_id,omitempty"`
	// GradedOnSubstitute is true when the graded resolution belongs to a MISMATCH
	// substitute finding (provenance: the pass/fail did NOT come from a reproduced
	// finding). Always false for REPRODUCED.
	GradedOnSubstitute bool `json:"graded_on_substitute,omitempty"`
	// EndToEndPass is the TRUE "does live scan get the right answer from raw events"
	// verdict over ALL scenarios: DETECT-MISS / MISMATCH on an expected-escalate is a
	// FAIL; a DETECT-MISS on an expected-resolve is a PASS (correct: nothing flagged);
	// a REPRODUCED scenario inherits the chain_action grade. This is NOT the same as
	// the ScenarioResult.Pass (the agent-reasoning number over REPRODUCED only).
	EndToEndPass bool `json:"end_to_end_pass"`
	// NoFindingCorrect is true for the DETECT-MISS-on-expected-resolve case: nothing
	// was flagged and nothing should have been. Reported separately, never conflated
	// with an agent resolve.
	NoFindingCorrect bool `json:"no_finding_correct,omitempty"`
}

DetectFidelityRow is the per-scenario fidelity record surfaced in the report.

type DetectOutcome

type DetectOutcome string

DetectOutcome is the per-scenario detect-fidelity bucket.

const (
	// OutcomeReproduced — detect emitted a finding matching the expected
	// (actor, family). The matched resolution is the graded data point.
	OutcomeReproduced DetectOutcome = "REPRODUCED"
	// OutcomeMismatch — detect emitted finding(s) but none match the expected
	// family+actor (type/actor drift).
	OutcomeMismatch DetectOutcome = "MISMATCH"
	// OutcomeDetectMiss — detect emitted ZERO findings (no resolution to grade).
	OutcomeDetectMiss DetectOutcome = "DETECT-MISS"
)

type E2EOutcome

type E2EOutcome struct {
	Result   ScenarioResult
	Fidelity DetectFidelityRow
	// Transcript is the captured model exchanges across all findings in this
	// scenario's run (§4.7). Carried so the harness writes artifacts from one source.
	Transcript []TranscriptEntry
}

E2EOutcome is the per-scenario result of an end-to-end run: the graded ScenarioResult (shared shape with -mode real, so the median/classifier spine works unchanged) plus the detect-fidelity record that says whether detect even reproduced the scenario's expected finding.

func RunScenarioE2E

func RunScenarioE2E(ctx context.Context, client agent.Client, ls LoadedScenario, opts agent.CascadeOptions, repoRoot string) (E2EOutcome, error)

RunScenarioE2E runs ONE scenario through the REAL pipeline.Run over a fresh, isolated temp git store and returns the graded outcome + detect-fidelity row.

Steps (mirrors cmd/mallcop/scan.go's pipeline.Config, minus the file I/O):

1. Fresh temp git store per scenario (seedScenarioStore's gitInit+store.Open pattern, but WITHOUT pre-seeding the finding — detect must produce it).
2. baselineFromScenario reconstructs the typed baseline the prod detectors + toolrun read.
3. memConnector returns the scenario's eventRecord-projected events.
4. recordingClient wraps the supplied Client so the transcript is captured.
5. pipeline.Run with Workers:1, ConsensusRuns threaded from opts, and the PROD toolrun.Runner over the same store + baseline.
6. Read resolutions back from the store IN-PROCESS (store.Load), map the action string (resolve/escalate → resolved/escalated), classify detect fidelity, build a ScenarioRun, and Grade it.
7. Tear down the temp store.

opts carries the per-tier model ids + ConsensusRuns (defaulted ON by harness.Run, identical to -mode real). repoRoot pins the §3.8 operator-decisions corpus for toolrun (pass the eval RepoRoot; "" lets the binary-walk resolve it).

type ExamDetectReport

type ExamDetectReport struct {
	Rows   []ExamDetectRow  `json:"rows"`
	Totals ExamDetectTotals `json:"totals"`
}

ExamDetectReport is the full run result: one row per labeled scenario (in corpus order — sorted by relpath, deterministic) plus totals.

func RunExamDetect

func RunExamDetect(repoRoot string) (ExamDetectReport, error)

RunExamDetect loads the pinned corpus under repoRoot and grades the REAL core/detect.Detect output of every labeled scenario against its expected_detection ground truth. Offline, deterministic, LLM-free — no inference client is constructed anywhere on this path.

This is RunExamDetectExtra(repoRoot, "") — byte-identical to the prior behavior (no extra scenarios dir touched, no wire-shape change: Extra is always its zero value and thus omitted from JSON).

func RunExamDetectExtra

func RunExamDetectExtra(repoRoot, extraScenariosDir string) (ExamDetectReport, error)

RunExamDetectExtra is RunExamDetect, additionally UNIONING the labeled scenarios found under extraScenariosDir (mallcoppro-f95) into the grading pass — a customer detector's OWN co-located efficacy scenarios (detectors/<name>/scenarios/*.yaml), loaded via LoadExtraScenarios (NO pin check, NEVER touching corpus.pin or the reference corpus's own digest). extraScenariosDir == "" reproduces RunExamDetect's exact prior behavior.

Every row this grades — reference or extra — runs through the IDENTICAL core/detect.Detect call over the SAME scenario/baseline projections; the only difference is provenance, carried on the row as Extra so callers (e.g. core/selfgate's customer-tree stage) can tell a detector's OWN proof scenarios apart from the reference corpus's.

type ExamDetectRow

type ExamDetectRow struct {
	// ScenarioID is the scenario's YAML id.
	ScenarioID string `json:"scenario_id"`
	// MustFire lists the detector family tokens that must appear among the
	// emitted findings' families (normalized lowercase).
	MustFire []string `json:"must_fire"`
	// MustNotFire lists the detector family tokens that must be absent.
	MustNotFire []string `json:"must_not_fire"`
	// Emitted lists the family token of every finding detect emitted, in
	// emission order (duplicates preserved — the visible multi-fire).
	Emitted []string `json:"emitted"`
	// Pass is true when every must_fire family is present and every
	// must_not_fire family is absent.
	Pass bool `json:"pass"`
	// Extra is true when this row came from an --extra-scenarios-dir UNION
	// (mallcoppro-f95) rather than the pinned reference corpus — a customer
	// detector's OWN co-located efficacy scenarios (detectors/<name>/scenarios/
	// *.yaml), graded through the identical real .wasm/detecthost path but
	// UNPINNED: they never touch corpus.pin and never count toward the
	// reference corpus's own integrity digest. Omitted (false) for every
	// ordinary reference-corpus row, so the wire shape is unchanged for every
	// existing caller that never passes an extra dir.
	Extra bool `json:"extra,omitempty"`
}

ExamDetectRow is the per-labeled-scenario grading record.

type ExamDetectTotals

type ExamDetectTotals struct {
	// Labeled is the number of scenarios carrying an expected_detection block.
	Labeled int `json:"labeled"`
	// Unlabeled is the number of scenarios WITHOUT the block — skipped from
	// grading but counted so coverage drift is visible.
	Unlabeled int `json:"unlabeled"`
	// Passed / Failed partition the labeled set.
	Passed int `json:"passed"`
	Failed int `json:"failed"`
}

ExamDetectTotals aggregates the run.

type FailBin

type FailBin string

FailBin is a classifier bin name.

const (
	BinPass           FailBin = "PASS"
	BinNoInference    FailBin = "I3_no_inference"
	BinChainDrop      FailBin = "I2_chain_drop_no_terminal"
	BinShouldResolve  FailBin = "A1_invest_should_resolve_but_escalated"
	BinShouldEscalate FailBin = "A2_should_escalate_but_resolved"
	BinRubricAxisFail FailBin = "R_rubric_axis_fail"
)

type HarnessReport

type HarnessReport struct {
	Mode           Mode              `json:"mode"`
	CorpusCount    int               `json:"corpus_count"`
	CorpusSHA      string            `json:"corpus_sha256"`
	Runs           []RunResult       `json:"runs"`
	MedianPassRate float64           `json:"median_pass_rate"`
	NoiseBandPP    float64           `json:"noise_band_pp"`
	WithinBand     bool              `json:"runs_within_band"`
	Classifier     ClassifierSummary `json:"classifier"`
	// DetectFidelity is the e2e-only block measuring how many scenario findings the
	// REAL detector fleet reproduced (vs mismatched/missed). Nil/zero for ModeCanned
	// and ModeReal (which inject the finding and so never exercise detect). It is the
	// headline result e2e exposes and must never be silently dropped.
	DetectFidelity *DetectFidelity `json:"detect_fidelity,omitempty"`
	// ReproducedPassRate is the e2e-only chain_action pass-rate over REPRODUCED
	// scenarios ONLY — the agent-reasoning number, directly comparable to ModeReal.
	// Computed as the median across runs of (passes among reproduced / reproduced
	// count). Zero for non-e2e modes.
	ReproducedPassRate float64 `json:"reproduced_pass_rate,omitempty"`
	// Note states, in the report itself, what the number means — so a reader can
	// never mistake the merge-gate's 100% for the accuracy number (§4.4), nor the
	// e2e all-scenario rate for the agent-reasoning rate.
	Note string `json:"note"`
}

HarnessReport is the harness's full output: the corpus integrity facts, every run, the MEDIAN pass-rate + noise band, and the classifier summary.

func Run

func Run(ctx context.Context, cfg RunConfig) (HarnessReport, error)

Run executes the harness: load → N passes → grade → median → classify.

The corpus integrity gate (count + SHA) runs FIRST via Load. A tampered or drifted corpus HARD-FAILS here and NOTHING runs (the eval interlock).

type LoadedScenario

type LoadedScenario struct {
	// RelPath is the forward-slash path under exams/scenarios (e.g.
	// "auth/AF-01-fat-finger-benign.yaml"). Stable across OSes; used in the
	// manifest and as a provenance key.
	RelPath string
	// FileSHA is the lowercase-hex sha256 of the scenario file's bytes.
	FileSHA string
	// Scenario is the parsed, validated scenario (exam.Load).
	Scenario *exam.Scenario
}

LoadedScenario pairs a parsed scenario with its provenance: the corpus-relative path (stable id used in result filenames + the manifest) and the per-file content hash that fed the corpus digest.

func LoadExtraScenarios

func LoadExtraScenarios(dir string) ([]LoadedScenario, error)

LoadExtraScenarios scans dir (any directory — e.g. a customer detector's own co-located detectors/<name>/scenarios/ sidecar, mallcoppro-f95) for scenario YAML files using the SAME leading-underscore skip and exam.Load parser scanCorpus applies to the pinned corpus, but performs NO pin verification and contributes NOTHING to any corpus.pin digest or count — this is the UNIONED, UNPINNED efficacy set RunExamDetectExtra grades alongside the reference corpus. It is never part of the reference corpus and never mutates it.

dir == "" (no extra scenarios shipped) returns (nil, nil) — the caller's union is then just the reference corpus, unchanged.

type Mode

type Mode string

Mode selects the inference backend.

const (
	// ModeCanned is the creds-free merge-gate: cannedbackend golden responses.
	ModeCanned Mode = "canned"
	// ModeReal is the parity run against a live model via DirectClient. WIRED but
	// not run here (no creds). ModeReal injects a SYNTHETIC finding built from the
	// scenario's YAML finding: block and calls agent.ResolveFindingWith directly —
	// the detector is bypassed.
	ModeReal Mode = "real"
	// ModeE2E is the END-TO-END scan run: it drives the REAL pipeline.Run (the same
	// function `mallcop scan` calls) so the scenario's raw events flow through
	// connect → detect.Detect (the PROD detector fleet) → store → resolve → store.
	// The finding the cascade resolves is whatever the REAL detectors emit, NOT the
	// YAML block. It uses the same {base_url,key} pivot client as ModeReal (or a
	// cannedbackend for the $0 canned-verification test). Its headline output is the
	// detect-fidelity block: how many scenario findings detect even reproduces.
	ModeE2E Mode = "e2e"
)

type RunConfig

type RunConfig struct {
	// Mode selects the backend: ModeCanned (merge-gate, golden responses) or
	// ModeReal (parity, live model). ModeReal requires creds (RealClientFromEnv).
	Mode Mode
	// N is the number of full-corpus passes for the median (§4.6). Default 3 when
	// <= 0 — single-run gating is forbidden.
	N int
	// Opts carries per-tier model ids + the (optional) live ToolRunner. The
	// merge-gate runs with a nil ToolRunner (golden responses need no live tools).
	Opts agent.CascadeOptions
	// RealClient is the live client for ModeReal. Ignored in ModeCanned (a fresh
	// golden cannedbackend is started per scenario). Must be non-nil for ModeReal.
	RealClient agent.Client
}

RunConfig configures one harness invocation.

type RunResult

type RunResult struct {
	Index    int              `json:"index"`
	Results  []ScenarioResult `json:"results"`
	Passed   int              `json:"passed"`
	Total    int              `json:"total"`
	PassRate float64          `json:"pass_rate"`
	// Transcripts maps scenario id → the full captured transcript for this pass
	// (§4.7). Carried on the run so artifacts derive from a single source.
	Transcripts map[string][]TranscriptEntry `json:"-"`
	// Fidelity is the per-scenario detect-fidelity rows for this pass (ModeE2E
	// only; nil otherwise). detect is deterministic, so these are stable across
	// passes — the report aggregates run 0's rows.
	Fidelity []DetectFidelityRow `json:"-"`
	// ReproducedPassRate is the chain_action pass-rate over REPRODUCED scenarios
	// only for THIS pass (ModeE2E only) — the agent-reasoning number. Zero when no
	// scenario reproduced.
	ReproducedPassRate float64 `json:"reproduced_pass_rate,omitempty"`
}

RunResult is one full corpus pass: the graded per-scenario results + the chain_action pass-rate for the pass.

type ScenarioResult

type ScenarioResult struct {
	ScenarioID     string         `json:"scenario_id"`
	RelPath        string         `json:"rel_path"`
	FindingID      string         `json:"finding_id"`
	Category       string         `json:"category"`
	Difficulty     string         `json:"difficulty"`
	TerminalAction string         `json:"terminal_action"`
	ExpectedAction string         `json:"expected_chain_action"`
	TerminalReason string         `json:"terminal_reason"`
	ForceEscalated bool           `json:"force_escalated"`
	RouteID        string         `json:"route_id,omitempty"`
	ModelCalls     int            `json:"model_calls"`
	WallMillis     int64          `json:"wall_millis"`
	FullChain      []ChainStep    `json:"full_chain"`
	Structural     StructuralAxes `json:"structural"`
	// Pass is the harness verdict for this scenario: TRUE iff the chain_action
	// axis passed. The ONLY axis that gates (§4.4).
	Pass bool `json:"pass"`
}

ScenarioResult is the per-scenario, per-run result JSON (§4.4): scenario id, terminal action, the full chain, and the structural axes. This is the artifact every iteration tool (the classifier, the aggregator) consumes.

func Grade

func Grade(run ScenarioRun) ScenarioResult

Grade turns a ScenarioRun into a graded ScenarioResult. DETERMINISTIC: identical inputs → identical output, no clock, no network, no LLM.

type ScenarioRun

type ScenarioRun struct {
	Scenario       LoadedScenario
	TerminalAction string
	TerminalReason string
	ForceEscalated bool
	RouteID        string
	ModelCalls     int
	Transcript     []TranscriptEntry
	WallMillis     int64
	// SeedErr is non-empty when the per-scenario live ToolRunner could not be
	// seeded (a genuine store failure). On ModeReal this is a DEGRADED run — the
	// agent investigated without the scenario's live tools — and the harness can
	// surface it rather than silently scoring an empty-toolbox verdict.
	SeedErr string
}

ScenarioRun is the raw outcome of running ONE scenario through the core: the terminal Resolution, the captured transcript, and the call count. The grader (grader.go) turns this into a graded ScenarioResult. Kept separate so the runner stays free of grading policy.

func RunScenario

func RunScenario(ctx context.Context, client agent.Client, ls LoadedScenario, opts agent.CascadeOptions, liveTools bool) ScenarioRun

RunScenario runs one scenario through the portable core IN-PROCESS against the supplied Client, capturing the full transcript. The Client is the {cannedbackend ⇄ real} pivot — RunScenario never decides which; the caller does.

LIVE TOOLS ARE WIRED PER SCENARIO when liveTools is true. opts carries the per-tier model ids; its Tools field (the ToolRunner) is OVERRIDDEN here with a per-scenario runner backed by the REAL core/tools over THIS scenario's events + baseline + finding. This is the whole point of the parity run: on ModeReal the live agent must investigate against the scenario's OWN telemetry — search-events returns the scenario's events (folding §3.8 matched_rules), check-baseline returns the scenario's frequencies, search-findings returns the scenario's finding stream. Each scenario sees ONLY its own data (per-scenario isolation, §4.1). A caller that pre-set opts.Tools is intentionally ignored: a shared/static runner would leak telemetry across scenarios and make the number meaningless.

liveTools is false for the MERGE-GATE (ModeCanned): golden responses prove the grader pipeline, not the model's tool use, and the live ToolEmpty fail-safe would inject model-independent escalations that have nothing to do with what a golden response is testing. On ModeReal liveTools is true and a real per-scenario runner is ALWAYS wired — the run NEVER silently investigates with an empty toolbox (the very gap this closes). If liveTools is true but the runner cannot be seeded (a genuine store failure), opts.Tools is left unchanged and the failure is recorded in SeedErr so the harness surfaces a degraded run rather than scoring an empty-toolbox verdict as if it were real.

The per-scenario store lives in a temp dir cleaned up before RunScenario returns.

type StructuralAxes

type StructuralAxes struct {
	ChainAction  AxisResult `json:"chain_action"`
	TriageAction AxisResult `json:"triage_action"`
	Mentions     AxisResult `json:"mentions"`
	NoMentions   AxisResult `json:"no_mentions"`
	ToolsUsed    AxisResult `json:"tools_used"`
	Iterations   AxisResult `json:"iterations"`
}

StructuralAxes holds the per-axis grading for one scenario (§4.4). Only ChainAction gates pass/fail; the rest are reported provenance.

type TranscriptEntry

type TranscriptEntry struct {
	// Seq is the 0-based call index within this scenario (triage=0,
	// investigate=1, escalate-formatter / deep-panel = 2+).
	Seq int `json:"seq"`
	// Model is the model id the cascade requested for this call (the tier).
	Model string `json:"model"`
	// System is the tier's system prompt (the ported POST.md).
	System string `json:"system"`
	// UserPrompt is the boxed user message text (USER_DATA-wrapped finding +
	// tool transcript) the model saw. This is where a planted injection is
	// visible — the audit value of §4.7.
	UserPrompt string `json:"user_prompt"`
	// Tools are the names of tools advertised on this call (the model's actual
	// API surface, §3.8).
	Tools []string `json:"tools,omitempty"`
	// Reply is the model's first text block — the verdict-carrying reply the
	// cascade parsed (or the escalate formatter's free-text alert).
	Reply string `json:"reply"`
	// Err is non-empty when the model call returned an error (the cascade
	// fail-safes such a call to escalate).
	Err string `json:"err,omitempty"`
}

TranscriptEntry is one model exchange the harness observed: the full request the cascade sent (model, system prompt, boxed user prompt, advertised tools) and the model's reply. §4.7 demands every tool call, input, output, and model message be captured — this is the unit.