eval

package
v0.17.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 18, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

artifacts.go — write the harness's per-scenario result JSON, per-scenario TRANSCRIPTS, and the classifier summary to disk (§4.4 result JSON, §4.7 transcript audit). These are the artifacts every iteration tool consumes; transcript capture is non-negotiable.

Layout under outDir: 

outDir/
  report.json                       — the full HarnessReport (median, band, classifier)
  run-<i>/                           — one dir per corpus pass
    <scenario_id>.json               — graded ScenarioResult (§4.4)
    <scenario_id>.transcript.json    — full transcript: every model call (§4.7)
  classifier.json                    — the ClassifierSummary (the loop's to-do list)

backend.go — the inference-Client backends the harness drives the core with, and the per-scenario GOLDEN SCRIPT generator for the creds-free MERGE-GATE.

THE {base_url, key} PIVOT (§4.4, direct.go doc): the harness is parameterized over a single agent.Client seam. Two modes share it:

  • MERGE-GATE (mode "canned"): a cannedbackend is scripted, PER SCENARIO, to return that scenario's EXPECTED verdict (golden responses). With golden responses the deterministic grader's chain_action axis PASSES for every scenario — 0 pp of model noise. This gates HARNESS + GRADER regressions: if someone breaks the loader, the runner, the verdict parser, the gate wiring, or the grader, the merge-gate goes RED even though no model ran. It is creds-free (no network beyond localhost) so CI runs it on every push.

    THE MERGE-GATE IS EXPLICITLY NOT THE ACCURACY NUMBER. Golden responses say "if the model returned the right answer, does the pipeline grade it right?" — a pipeline-integrity check. The real accuracy number comes ONLY from the real-model run, where the model decides the verdict and the eval measures how often it is correct. Reporting the merge-gate's 100% as accuracy would be a lie; the doc string and the README field both say so.

  • REAL-MODEL (mode "real"): a core/inference.DirectClient pointed at MALLCOP_INFERENCE_URL with MALLCOP_API_KEY. THIS path is WIRED but DELIBERATELY NOT RUN here (no creds in this environment). RealClientFromEnv builds it; the harness CLI / test refuses to run it without both env vars set.

This file imports core/inference ONLY for the real path — it is the package boundary where the network client is allowed (core/eval is a harness, not the shipped product runtime; the import-lint guards core/, and a harness that wires the DirectClient is exactly the intended seam, mirrored on cmd/ + test/).

classifier.go — the ~40-line failure classifier (portable-agent-architecture.md §4.8). It bins each scenario into one of: PASS / I*-infra / A1/A2-algorithm / R*-rubric. The cluster name names the fix the next iteration needs — the classifier output is the to-do list for the loop.

Bins (lifted from §4.8's classifier vocabulary): 

PASS                            — chain_action passed.
I3_no_inference                 — 0 model calls AND not a force-escalate route
                                  (the model was never reached → infra/wiring bug).
I2_chain_drop_no_terminal       — no terminal action recorded (chain dropped).
A1_invest_should_resolve_but_escalated — expected resolved, got escalated
                                  (over-escalation cluster, §4.3).
A2_should_escalate_but_resolved — expected escalated, got resolved
                                  (under-escalation cluster — the dangerous tail).
R_rubric_axis_fail              — chain_action passed but a non-gating axis
                                  (mentions / no_mentions) failed. Reported for
                                  provenance; does NOT count as a harness failure.

corpus.go — the SHA-PINNED, PROVENANCE-SAFE scenario corpus loader.

This is the self-extension FOUNDATION (portable-agent-architecture.md §4.1, §4.9): the eval is an INTERLOCK. Before any scenario runs, the loader asserts two integrity gates that a self-extension loop (an agent that appends new scenarios) MUST clear:

  1. COUNT — the corpus has exactly the expected number of scenarios.
  2. SHA — the corpus content hashes to the exact pinned digest.

A mismatch on EITHER is a HARD FAIL (Load returns an error; nothing runs). The pin lives in a committed file (corpus.pin) the loader verifies on every run. An agent that adds or edits a scenario MUST also update the pin in the same change — so a tampered or drifted corpus cannot silently change the accuracy number. This is the eval-as-interlock property the closed loop depends on.

THE LEADING-UNDERSCORE FOOTGUN (§4.10): the walker SKIPS every path component that begins with "_" — both FILES (`_schema.yaml`) and DIRECTORIES (`_test/`). exams/scenarios/ ships `_schema.yaml` (the schema, not a scenario) and a `_test/` directory (harness probes, not graded scenarios). Including either would inflate the count, corrupt the hash, and feed non-scenarios to the grader. The skip is on ANY component, at ANY depth — `a/_b/c.yaml` is skipped. go:embed's own directory-embed default applies the IDENTICAL skip (files and dirs named "_*" or ".*" are excluded, recursively) — verified empirically and documented on corpusembed.go's ScenariosFS — so the embedded corpus (below) and the on-disk corpus always agree on which files are "in".

THE CANONICAL MANIFEST (the hashed artifact): the corpus SHA is NOT a hash of a tarball or a directory listing (both are order/zip/timestamp-sensitive). It is the SHA-256 of a CANONICAL MANIFEST: for each included scenario, one line 

<forward-slash relpath><two spaces><lowercase hex sha256 of file bytes>\n

lines sorted lexicographically by relpath, concatenated. This is reproducible across OSes (forward slashes), independent of walk order (sorted), and independent of file mtime/permissions (content hash only). The Go loader is the SOURCE OF TRUTH for the format; corpus.pin holds the digest the loader emits.

Disk-first / embed-last precedence

scanCorpus and readPin operate over io/fs.FS rather than raw os calls, so the SAME walking, hashing, and pin-verification logic runs against either source:

  • Load(repoRoot) — os.DirFS(repoRoot): the on-disk corpus. This is what dev and CI use, and it is what preserves edit-and-reload — a scenario edited on disk is picked up on the next Load, no rebuild required.
  • LoadEmbedded() — mallcop.ScenariosFS (corpusembed.go, repo root): the corpus baked into the binary at build time. This is the fallback for a SHIPPED binary running in a customer deploy repo that has no exams/scenarios directory on disk at all — RepoRoot() still resolves (the customer repo has its own go.mod/.git marker), but Load(repoRoot) then fails to find exams/scenarios under it. LoadEmbedded lets `mallcop eval` / exam-detect run from the shipped reference corpus in that case.

On-disk ALWAYS wins when present — callers that want the fallback behavior call Load first and fall back to LoadEmbedded only on a "corpus root not found" failure, never the other way around. Neither function silently prefers the embed; each is an explicit, separately-named entry point.

coverage.go — the per-detector-family COVERAGE MATRIX over an exam-detect run.

This is the corpus-expansion STEERING artifact (mallcoppro selfeval-c7): it rolls the flat per-scenario exam-detect rows up by detector family so a human (or the self-heal loop) can see, at a glance, which families are THIN — few labels, or labels the current detector can't satisfy — and therefore which family the corpus should grow next.

The matrix is a pure re-aggregation of an already-graded ExamDetectReport: it runs NO detector, reads NO corpus, and adds NO grading semantics. Every count is derived from the rows the grader already produced (examdetect.go), so the matrix is exactly as offline/deterministic/LLM-free as the report it summarizes. It is exposed as an additive `coverage` field on the exam-detect JSON (ExamDetectReport.Coverage) — the recall-first view (recall.go) answers "how many attacks did we catch"; this answers "how is that spread across families, and where is the denominator still a toy."

Definitions (mechanical, no benign/attack moral judgment — a must_fire label is the recall/attack side, a must_not_fire label is the precision/benign side, exactly as the grader treats them):

  • AttackLabels — # rows carrying this family in must_fire.
  • BenignLabels — # rows carrying this family in must_not_fire.
  • Missed — # must_fire rows where this family is ABSENT from the emitted set AND is not reserved-pending: a real recall gap (a registered detector that should fire but doesn't, or an unregistered family on a NON-reserved row).
  • FalseAlarms — # must_not_fire rows where this family IS present in the emitted set: a precision gap (the detector fired on a benign twin it should have stayed silent on).
  • Reserved — # must_fire rows where this family is reserved-pending (ExamDetectRow.ReservedPending): a TRACKED expected-miss for a family with no registered detector yet — counted apart from Missed so a not-yet-authored family doesn't read as a detector regression.

detectfidelity.go — the LOAD-BEARING detect-fidelity accounting for -mode e2e.

THE HEADLINE RESULT e2e EXPOSES: core/detect produces DIFFERENT / FEWER / ZERO findings than the YAML finding blocks the eval modes inject. Examples the detect map proved:

  • AC-01: no `new-external-access` detector exists and the event type (repo.add_collaborator) is in no detector's event-type set → ZERO findings.
  • UT-03: `unusual-timing` is not reproduced (the scenario event lacks the hour-profile the detector needs); `unusual-login` fires for the actor instead → the EXPECTED `unusual-timing` finding is not reproduced.
  • VA-03: volume-anomaly's frequency key does not match → ZERO findings.

So detect-fidelity is NOT incidental — it is the headline. This file MEASURES it explicitly and the harness reports it; it is NEVER silently passed or folded into the chain_action pass-rate as if the agent had decided correctly.

Per scenario, after pipeline.Run, classify into one of three outcomes by comparing detect's stored findings against the scenario's YAML finding block: 

REPRODUCED  — a stored finding matches the expected (Actor, detector-family).
              The resolution for THAT finding is graded against expected
              chain_action. This is the only case that contributes a real
              agent-accuracy data point comparable to -mode real.
MISMATCH    — detect emitted finding(s) but NONE match the expected family+actor.
              We grade the closest-by-actor resolution as a "substitute" for
              provenance, but the type drift is flagged.
DETECT-MISS — detect emitted ZERO findings. No resolution to grade. A miss on an
              expected-ESCALATE scenario is a real production false-negative (the
              attack was never surfaced) — counted as an END-TO-END FAIL. A miss
              on an expected-RESOLVE scenario means nothing was flagged, which is
              the correct production outcome (nothing to escalate) — reported as
              "no-finding-correct", but still tracked separately and NEVER
              conflated with an agent resolve.

Package eval is the PORTABLE eval harness — the replatformed academy (portable-agent-architecture.md §4), lifted off the deleted legion/campfire transport onto the in-process core.

It is the most portable artifact in the project (§4.9): the corpus is the value, the substrate is replaceable. This package keeps the corpus, the deterministic grader, the median-of-N discipline, the failure classifier, and transcript capture — and drops campfire, legion, cf, rd, and the .toml.tmpl rendering entirely.

Pipeline: 

Load(repoRoot)         — SHA-pinned, provenance-safe corpus loader. Skips
                         leading-underscore paths (files AND dirs). Asserts the
                         pinned count + SHA-256; a mismatch HARD-FAILS (the
                         eval-as-interlock gate the self-extension loop needs).
RunScenario(...)       — runs ONE scenario through core/agent.ResolveFindingWith
                         IN-PROCESS via a controllable Client ({base_url,key}
                         pivot), capturing a full per-scenario transcript (§4.7).
Grade(run)             — DETERMINISTIC structural grader (no LLM in pass/fail).
                         chain_action is the only gating axis; the rest are
                         reported provenance. Emits per-scenario result JSON.
Run(cfg)               — the full driver: load → N passes → median pass-rate
                         with the 8pp band → classifier summary.

Two backends share the Client seam: a creds-free cannedbackend MERGE-GATE (golden responses → deterministic green; gates harness+grader regressions; NOT the accuracy number) and a real DirectClient parity run (WIRED via RealClientFromEnv, NOT run without creds).

examdetect.go — the OFFLINE detect-layer exam (`mallcop exam-detect`).

This is the K1 keystone of the self-extension loop: a deterministic, LLM-free grader that runs the REAL core/detect.Detect over every LABELED corpus scenario and grades the emitted findings against the scenario's expected_detection ground truth (must_fire / must_not_fire detector family tokens). A labeled-and-unfixed detection gap (e.g. VA-03's volume-anomaly false negative) shows up as a RED row — the exam is the interlock the loop closes against when it grows a detector.

Grading contract:

  • The corpus is loaded via Load(repoRoot) — the corpus.pin integrity interlock stays in the path (a drifted corpus runs NOTHING).
  • Events/baseline are projected through the SAME unexported projections the e2e runner uses (scenarioEvents / baselineFromScenario) — detectors read payloadMeta discriminators from the projected payload, so a naive re-projection would silently blind them.
  • Findings are graded on family PRESENCE over the whole emitted set (findingFamilyToken), NOT counts or actors — some detectors legitimately emit multiple findings per family in one scan (e.g. unusual-timing: one finding per distinct (actor, hour) GROUP per scan, not one per event — mallcoppro-d73 collapsed the old per-event fan-out — so a scan touching several novel actor-hours still yields several findings).
  • UNLABELED scenarios (nil ExpectedDetection) are skipped-but-counted: grading covers only explicit labels; the corpus-wide backfill is a deferred human decision.
  • RESERVED scenarios (expected_detection.reserved: true, mallcoppro-db0) specify a must-fire outcome for a detector that may not exist yet — the REQUESTER's ground truth, authored independent of and prior to whoever eventually writes the detector. A reserved must_fire family with no REGISTERED detector (core/detect.Detectors()) grades as a TRACKED expected-miss (ExamDetectTotals.Reserved), not a hard failure — it still shows RED on the row, it just doesn't block the exam or CI. The day a detector implementing that family registers, grading reverts to the ordinary hard rule for it automatically — no re-flagging required.

grader.go — the DETERMINISTIC structural grader (portable-agent-architecture.md §4.4, §4.9). Lifted from cmd/mallcop-academy/grading.go: the LOGIC is the same (compare terminal chain_action to expected; substring mentions; tool/iteration counts), the CAMPFIRE TRANSPORT is dropped — this grades a ScenarioRun captured in-process, not a disposition chain reconstructed from campfire messages.

THE GRADER HAS NO LLM IN THE PASS/FAIL PATH. Pass/fail is the chain_action axis alone — string equality of the terminal action against expected.chain_action (plus the "escalate-or-stronger" token). §4.1's rubric-strictness study found every miss was on chain_action; mentions/tools/iterations contributed zero false failures. They are GRADED and REPORTED as structural axes (provenance for the classifier) but DO NOT gate the harness verdict. "Don't tune the rubric, tune the system."

harness.go — the end-to-end eval driver: load (SHA+count gate) → run the corpus N times → grade deterministically → report MEDIAN chain_action pass-rate with the §4.6 8pp noise band → classify failures (§4.8). MEDIAN-OF-N is mandatory: "single-run results lie" (§4.6) — a single run can swing ±10 pp on identical code, so the harness NEVER gates on one run.

recall.go — the RECALL-FIRST honest-measurement view over an ExamDetectReport.

THE HONEST DENOMINATOR THE SELF-HEAL LOOP AND THE OPERATOR BOTH NEED. The default exam-detect view reports one aggregate pass/fail per labeled scenario (Totals.Passed / Failed). That number blends two fundamentally different kinds of failure into one pile:

  • A MISSED ATTACK — an attack that MUST fire but did not. This is a FATAL production false-negative: the attack was never surfaced, the operator was never warned. In detection terms it is a RECALL failure.
  • A FALSE ALARM — a benign scenario that MUST stay silent but fired. This is noise: the operator gets paged for nothing. In detection terms it is a PRECISION failure.

Conflating them hides the number that actually matters. An exam at "90% pass" could be missing every real attack (0% recall) while acing the benign twins, or catching every attack while crying wolf on the benign set. The self-heal loop must know WHICH, because it grows detectors to close recall gaps and tunes them to kill false alarms — opposite moves.

This file splits the SAME labeled corpus by ground truth and reports the two numbers separately, with the failures NAMED:

  • RECALL = detected / must-fire, over every scenario whose expected_ detection carries a must_fire family (an attack that MUST be caught). The MISSED attacks are listed prominently — they are the fatal failures.
  • PRECISION = correct-silent / must-stay-silent, over every scenario with NO must_fire family (a benign case). The FALSE ALARMS are listed by name.

The split is derived ENTIRELY from the existing ExamDetectRow labels (MustFire / MustNotFire / Emitted) — it re-runs no detector and changes no grading. It is a second lens on the same graded rows, so it is exactly as deterministic and LLM-free as the report it reads.

reporoot.go — self-resolving repo root for the eval harness (portable-agent-architecture.md §3.5, §4 determinism via SetRepoRootForTest).

The harness reads the scenario corpus from exams/scenarios/ under the repo root. When nothing pins the root explicitly it walks UP from the binary's own location to a project marker — NOT from CWD (the runner relocates CWD). An EXPLICITLY-set MALLCOP_REPO_ROOT is an operator/orchestrator decision and takes precedence over that heuristic walk (explicit config beats discovery — see RepoRoot's doc for why the old walk-first order was a bug). A test-only override (SetRepoRootForTest) pins the root DETERMINISTICALLY so `go test -count=N -race` always resolves the same corpus regardless of where the toolchain places the test binary — the same flake-closing seam core/agent uses.

runner.go — the IN-PROCESS scenario runner (portable-agent-architecture.md §4).

Each scenario runs through the PORTABLE core (core/agent.ResolveFindingWith) IN THE SAME PROCESS — no subprocess, no campfire, no legion. The runner:

  1. Builds a core finding.Finding from the scenario's finding: block, with finding.Type = scenario.Detector so the data-driven PRE-LLM floor's hard-constraint routes (priv-escalation, injection-probe, log-format-drift, ...) fire BEFORE any model call — exactly as in production (§4.3: not every finding deserves agent inference).
  2. Projects the scenario's events + baseline into a deterministic ToolEvidence the cascade boxes and the structural gate scores (the eval's per-scenario baseline is what makes runs reproducible, §4.1).
  3. Drives the cascade through a CONTROLLABLE inference Client — a cannedbackend for the creds-free merge-gate, a real DirectClient for the parity run (the {base_url,key} pivot). The runner is parameterized over the Client; it never dials the network itself.
  4. CAPTURES A PER-SCENARIO TRANSCRIPT — every model request (system + boxed user prompt + advertised tools) and every model reply — via a recording Client wrapper. Transcript capture is NON-NEGOTIABLE (§4.7): it is how a silent-empty tool return or a bypassed channel becomes diagnosable.

The runner imports core/agent (the portable core) and pkg/finding ONLY. It does NOT import core/inference — the caller injects whatever Client it wants. That keeps the {cannedbackend ⇄ real} pivot in the caller's hands and keeps this file substrate-free.

runner_e2e.go — the END-TO-END scenario runner. Where RunScenario (runner.go) injects a SYNTHETIC finding built from the scenario's YAML finding: block and calls agent.ResolveFindingWith directly, RunScenarioE2E drives the SAME pipeline.Run the production `mallcop scan` command calls (cmd/mallcop/scan.go): connect → detect.Detect (the REAL detector fleet) → store findings → resolveAll → store resolutions. The finding the cascade resolves is WHATEVER core/detect emits over the scenario's raw events — NOT the YAML block. Tools come from the PRODUCTION core/toolrun.Runner over the same isolated store.

This is the only path that exercises connect+detect+toolrun+store together as a real scan does. Everything downstream of the verdict (Grade, median-of-N, the classifier, the corpus interlock) is SHARED with -mode real, so the e2e number is directly comparable to the -mode real number — but ONLY over the scenarios detect actually reproduces. The detect-fidelity accounting (detectfidelity.go) measures and surfaces that reproduction gap explicitly; it is the headline result e2e exposes and is NEVER silently passed or folded into the pass-rate.

PER-SCENARIO ISOLATION: each scenario gets a FRESH temp git store (os.MkdirTemp + gitInit + store.Open), torn down before the next scenario. toolrun.Runner re-reads the live store per finding, so a shared store would leak scenario A's events/findings into scenario B's tool reads. Workers:1 keeps the per-scenario resolve ordering deterministic.

core/eval is the harness layer (not the shipped runtime), so importing core/pipeline + core/toolrun + core/connect + core/store here is the intended seam — the import-lint bans only vendor SDKs / orchestration / dropped transport (see imports_test.go), none of which this adds.

scenario_tools.go — the PER-SCENARIO live ToolRunner that backs the ModeReal parity run with REAL telemetry (portable-agent-architecture.md §3.8, §4.1).

THE GAP THIS CLOSES: RunScenario drives the portable cascade (agent.ResolveFindingWith) which reaches tools ONLY through the injected agent.ToolRunner seam (CascadeOptions.Tools). Before this file the runner passed whatever single ToolRunner the caller set — nil for the merge-gate — so on ModeReal the live agent investigated with NO scenario evidence: search-events returned nothing scenario-specific, check-baseline returned nothing, and the parity number measured a model staring at an empty toolbox. That number is meaningless.

THE FIX: for EACH scenario, scenarioToolRunner seeds a per-scenario evidence source from the exam.Scenario (its events, baseline, finding) and dispatches tool calls to the REAL core/tools over THAT data — the SAME tool surface the agent gets in production:

  • search-events → tools.SearchEventsWrapped over a git-backed store seeded with the scenario's events, FOLDING the operator-decisions matched_rules (§3.8) keyed on the finding family + the finding's flat metadata.
  • check-baseline → tools.CheckBaseline over a baseline.Baseline reconstructed from the scenario's known_entities + frequency_tables (so the agent can answer "is this routine for this actor").
  • search-findings → tools.SearchFindings over the same store (seeded with the scenario's finding) so the deeper sweep has the finding stream.

Each scenario gets its OWN runner over its OWN store + baseline: scenario A's agent sees ONLY scenario A's telemetry. That per-scenario isolation is what makes the eval reproducible (§4.1) — the baseline the gate scores is the scenario's, not a shared/static fixture.

core/eval is the harness layer (not the shipped runtime), so importing the real core/tools + core/store here is the intended seam — the import-lint bans only vendor SDKs / orchestration / dropped transport, none of which this adds.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func RealClientFromEnv 

func RealClientFromEnv() (agent.Client, error)

RealClientFromEnv builds the real-model DirectClient from the environment. It is the {base_url,key} pivot's REAL leg: BaseURL=MALLCOP_INFERENCE_URL, Key=MALLCOP_API_KEY, Model=MALLCOP_MODEL (optional). It returns an error when either required var is unset — the harness REFUSES to run real mode without creds, which is why no real call happens in this environment.

func RepoRoot 

func RepoRoot() (string, error)

RepoRoot returns the project root that holds exams/scenarios.

Resolution order (§3.5, revised per PR #191 review):

  1. SetRepoRootForTest override (test-only seam, deterministic).
  2. MALLCOP_REPO_ROOT env override — an EXPLICIT pin set by a person or an orchestrating process (e.g. core/selfgate's runTreeExam, or core/investigate's run-eval self-exec) wins over the heuristic walk: explicit config beats discovery. The old walk-first order made the pin silently ignorable — whenever the binary happened to sit under ANY marker-bearing directory, the walk's answer shadowed the caller's explicit choice and eval graded the WRONG root (e.g. a silently-empty local scenarios/ union) with no error anywhere.
  3. The walk up from os.Executable() — the production default when nothing pins the root explicitly (the scaffolded GHA runtime path: no env var is set there, the pinned release binary is installed inside the deploy repo checkout, and the walk resolves it).

func SearchEventsEnvelopeFromSnapshot 

func SearchEventsEnvelopeFromSnapshot(events []tools.EventView, rules []tools.OperatorRule) tools.SearchEventsEnvelope

SearchEventsEnvelopeFromSnapshot reconstructs the minimal SearchEventsEnvelope writeSearchEvents renders — its Events and MatchedRules — from the frozen snapshot slices. It exists so RunTools renders the SAME compact transcript it always did while reading ONLY immutable snapshot data (no store re-read). Only the two fields writeSearchEvents consumes are populated; the rest of the envelope (FilterApplied, Notes) is not rendered by writeSearchEvents and is left at its zero value deliberately. The input slices are treated as read-only.

func SetRepoRootForTest 

func SetRepoRootForTest(dir string)

SetRepoRootForTest pins the repo root the harness resolves, taking precedence over the os.Executable() walk. It panics if dir is non-empty and does not contain exams/scenarios, so a typo fails loudly instead of silently resolving an empty corpus. Tests defer SetRepoRootForTest("") to clear it.

Production never calls this; the MALLCOP_REPO_ROOT env pin + the walk cover real deployments. Pinning the override (checked FIRST in RepoRoot) removes the non-determinism where the resolved root depends on binary placement — the reason the determinism harness (-count=10 + -race) needs an explicit seam.

func WriteArtifacts 

func WriteArtifacts(outDir string, report HarnessReport) (int, error)

WriteArtifacts persists the report, per-scenario results, per-scenario transcripts, and the classifier summary under outDir. Transcripts are read from each RunResult (the single source). Returns the number of transcript files written (one per scenario per run) for caller assertions.

Types

type AxisResult 

type AxisResult string

AxisResult is one structural axis outcome. Mirrors grading.go's vocabulary. 

const (
	AxisPass AxisResult = "pass"
	AxisFail AxisResult = "fail"
	AxisNA   AxisResult = "n/a"
)

type ChainStep 

type ChainStep struct {
	Seq   int    `json:"seq"`
	Model string `json:"model"`
	Reply string `json:"reply"`
	Err   string `json:"err,omitempty"`
}

ChainStep is one step of the captured chain (§4.3/§4.4: full chain = every sub-agent dispatch). Here, one model call = one step (model + reply snippet).

type ClassifierSummary 

type ClassifierSummary struct {
	Counts map[FailBin]int `json:"counts"`
	// PerScenario maps scenario id → its bin, for transcript pull (§4.8 step 3).
	PerScenario map[string]FailBin `json:"per_scenario"`
}

ClassifierSummary aggregates bins across the worst (most-failing) run so the loop sees the dominant cluster. Counts is bin → number of scenarios.

func Classify 

func Classify(runs []RunResult) ClassifierSummary

Classify bins the run with the LOWEST pass-rate (the worst case names the cluster to fix). Median-of-N reports the median rate; the classifier reports the worst run's failure shape so the loop targets the real weakness.

type Corpus 

type Corpus struct {
	// Scenarios are sorted lexicographically by RelPath — deterministic order so
	// the runner, the grader, and the manifest all agree.
	Scenarios []LoadedScenario
	// SHA is the corpus digest: sha256 of the canonical manifest. It equals the
	// pinned digest (Load fails otherwise).
	SHA string
	// Count is len(Scenarios) — equals the pinned count (Load fails otherwise).
	Count int
}

Corpus is the loaded, integrity-verified scenario set.

func Load 

func Load(repoRoot string) (Corpus, error)

Load scans the on-disk corpus under repoRoot, then HARD-VERIFIES it against the committed pin. This is the PRIMARY, disk-first entry point: dev and CI always resolve here, and an on-disk corpus always wins over the embed — editing a scenario on disk is picked up on the very next Load call, no rebuild required.

repoRoot is the directory holding exams/scenarios (the repo root). Callers resolve it via RepoRoot (walk up from the binary) or pin it in tests via SetRepoRootForTest — the same self-locating discipline §3.5 prescribes.

A repoRoot that resolves (a valid go.mod/.git marker) but carries no exams/scenarios directory — the shipped-binary-in-a-customer-repo case — returns a "corpus root not found" error; callers that want the embedded fallback in that case call LoadEmbedded explicitly (see LoadEmbedded's doc).

func LoadEmbedded added in v0.12.0 

func LoadEmbedded() (Corpus, error)

LoadEmbedded scans the corpus baked into the binary at build time (mallcop.ScenariosFS, corpusembed.go at the repo root) and HARD-VERIFIES it against the SAME committed pin Load checks — the embedded corpus.pin is byte-identical to the on-disk one (both are produced by the same //go:embed exams/scenarios directive, which embeds corpus.pin alongside the scenario files since "corpus.pin" does not begin with "_").

This is the FALLBACK entry point for a shipped mallcop binary running `eval` / exam-detect inside a customer deploy repo that has no exams/scenarios directory on disk at all: RepoRoot() still resolves there (the customer repo has its own go.mod or .git marker), so Load(repoRoot) fails with "corpus root not found" rather than an unresolvable-root error. LoadEmbedded lets that failure mode still produce a working eval, from the exact reference corpus the binary shipped with.

LoadEmbedded never reads the filesystem — it is unaffected by MALLCOP_REPO_ROOT and cannot pick up an edited on-disk scenario. Callers that want disk-first-with-embedded-fallback semantics call Load first and use LoadEmbedded only when Load fails to locate the corpus root — never the reverse, so dev edit-and-reload is always preserved.

func (Corpus) Manifest 

func (c Corpus) Manifest() string

Manifest returns the canonical manifest (the exact bytes that get hashed): one "<relpath><two spaces><filesha>\n" line per scenario, sorted by relpath. Exposed so the pin-regeneration helper and tests can recompute the digest from the same source of truth the loader uses.

type CorpusPin 

type CorpusPin struct {
	Count int
	SHA   string
}

CorpusPin is the committed integrity pin: the expected count + digest. A mismatch between the loaded corpus and this pin HARD-FAILS the run.

type DetectFidelity 

type DetectFidelity struct {
	Total      int `json:"total"`
	Reproduced int `json:"reproduced"`
	Mismatch   int `json:"mismatch"`
	DetectMiss int `json:"detect_miss"`
	// ReproductionRate = Reproduced / Total. Reported PROMINENTLY: a low rate means
	// the validated -mode real number does NOT yet transfer to live scan because the
	// detector fleet cannot produce most scenario findings.
	ReproductionRate float64 `json:"reproduction_rate"`
	// EndToEndPassRate is passes / Total over ALL scenarios where DETECT-MISS-on-
	// escalate and MISMATCH-on-escalate count as fails — the honest "live scan from
	// raw events" number. Far below the agent-reasoning rate when reproduction is low.
	EndToEndPassRate float64 `json:"end_to_end_pass_rate"`
	// E2ERecall is EndToEndPass / count over rows whose ExpectedAction demands
	// escalate (actionIsEscalate) — the TRUE live-scan attack-detection rate: a
	// DETECT-MISS or a MISMATCH that never escalates on an attack scenario counts
	// as a miss. Split out of EndToEndPassRate the same way harness.go's
	// RunResult.RecallRate splits the blended agent-reasoning PassRate (mallcoppro
	// C2; mirrors recall.go's exam-detect split at the detect-fidelity layer).
	E2ERecall float64 `json:"e2e_recall"`
	// E2EPrecision is EndToEndPass / count over rows whose ExpectedAction does NOT
	// demand escalate (the benign scenarios) — the TRUE live-scan rate of correctly
	// leaving benign activity alone (1.0 = no live-scan over-escalations).
	E2EPrecision float64 `json:"e2e_precision"`
	// Rows is the per-scenario fidelity detail.
	Rows []DetectFidelityRow `json:"rows"`
}

DetectFidelity is the report block aggregating per-scenario fidelity (§e2e).

type DetectFidelityRow 

type DetectFidelityRow struct {
	ScenarioID string `json:"scenario_id"`
	// ExpectedDetector is the scenario's YAML finding detector family (the family
	// the eval modes would inject).
	ExpectedDetector string `json:"expected_detector"`
	// ExpectedActor is the scenario's finding actor.
	ExpectedActor string `json:"expected_actor"`
	// ExpectedAction is the scenario's expected chain_action ("escalated" /
	// "resolved" / "escalate-or-stronger").
	ExpectedAction string `json:"expected_action"`
	// EmittedDetectors lists the (detector-family/actor) of every finding detect
	// emitted for this scenario — the visible drift.
	EmittedDetectors []string `json:"emitted_detectors"`
	// Outcome is the fidelity bucket.
	Outcome DetectOutcome `json:"outcome"`
	// MatchedFindingID is the id of the stored finding that matched (REPRODUCED) or
	// the closest-by-actor substitute (MISMATCH). Empty on DETECT-MISS.
	MatchedFindingID string `json:"matched_finding_id,omitempty"`
	// GradedOnSubstitute is true when the graded resolution belongs to a MISMATCH
	// substitute finding (provenance: the pass/fail did NOT come from a reproduced
	// finding). Always false for REPRODUCED.
	GradedOnSubstitute bool `json:"graded_on_substitute,omitempty"`
	// EndToEndPass is the TRUE "does live scan get the right answer from raw events"
	// verdict over ALL scenarios: DETECT-MISS / MISMATCH on an expected-escalate is a
	// FAIL; a DETECT-MISS on an expected-resolve is a PASS (correct: nothing flagged);
	// a REPRODUCED scenario inherits the chain_action grade. This is NOT the same as
	// the ScenarioResult.Pass (the agent-reasoning number over REPRODUCED only).
	EndToEndPass bool `json:"end_to_end_pass"`
	// NoFindingCorrect is true for the DETECT-MISS-on-expected-resolve case: nothing
	// was flagged and nothing should have been. Reported separately, never conflated
	// with an agent resolve.
	NoFindingCorrect bool `json:"no_finding_correct,omitempty"`
}

DetectFidelityRow is the per-scenario fidelity record surfaced in the report.

type DetectOutcome 

type DetectOutcome string

DetectOutcome is the per-scenario detect-fidelity bucket. 

const (
	// OutcomeReproduced — detect emitted a finding matching the expected
	// (actor, family). The matched resolution is the graded data point.
	OutcomeReproduced DetectOutcome = "REPRODUCED"
	// OutcomeMismatch — detect emitted finding(s) but none match the expected
	// family+actor (type/actor drift).
	OutcomeMismatch DetectOutcome = "MISMATCH"
	// OutcomeDetectMiss — detect emitted ZERO findings (no resolution to grade).
	OutcomeDetectMiss DetectOutcome = "DETECT-MISS"
)

type E2EOutcome 

type E2EOutcome struct {
	Result   ScenarioResult
	Fidelity DetectFidelityRow
	// Transcript is the captured model exchanges across all findings in this
	// scenario's run (§4.7). Carried so the harness writes artifacts from one source.
	Transcript []TranscriptEntry
}

E2EOutcome is the per-scenario result of an end-to-end run: the graded ScenarioResult (shared shape with -mode real, so the median/classifier spine works unchanged) plus the detect-fidelity record that says whether detect even reproduced the scenario's expected finding.

func RunScenarioE2E 

func RunScenarioE2E(ctx context.Context, client agent.Client, ls LoadedScenario, opts agent.CascadeOptions, repoRoot string) (E2EOutcome, error)

RunScenarioE2E runs ONE scenario through the REAL pipeline.Run over a fresh, isolated temp git store and returns the graded outcome + detect-fidelity row.

Steps (mirrors cmd/mallcop/scan.go's pipeline.Config, minus the file I/O):

  1. Fresh temp git store per scenario (seedScenarioStore's gitInit+store.Open pattern, but WITHOUT pre-seeding the finding — detect must produce it).
  2. baselineFromScenario reconstructs the typed baseline the prod detectors + toolrun read.
  3. memConnector returns the scenario's eventRecord-projected events.
  4. recordingClient wraps the supplied Client so the transcript is captured.
  5. pipeline.Run with Workers:1, ConsensusRuns threaded from opts, and the PROD toolrun.Runner over the same store + baseline.
  6. Read resolutions back from the store IN-PROCESS (store.Load), map the action string (resolve/escalate → resolved/escalated), classify detect fidelity, build a ScenarioRun, and Grade it.
  7. Tear down the temp store.

opts carries the per-tier model ids + ConsensusRuns (defaulted ON by harness.Run, identical to -mode real). repoRoot pins the §3.8 operator-decisions corpus for toolrun (pass the eval RepoRoot; "" lets the binary-walk resolve it).

type ExamDetectReport added in v0.8.0 

type ExamDetectReport struct {
	Rows   []ExamDetectRow  `json:"rows"`
	Totals ExamDetectTotals `json:"totals"`
	// Coverage is the per-detector-family coverage matrix (coverage.go) — a
	// pure re-aggregation of Rows that surfaces which families are thin
	// (few labels) or unsatisfied (missed / false_alarms), the corpus-expansion
	// steering artifact. Additive: populated by RunExamDetectExtra, omitted from
	// JSON when empty so a zero-row report's wire shape is unchanged.
	Coverage []FamilyCoverage `json:"coverage,omitempty"`
}

ExamDetectReport is the full run result: one row per labeled scenario (in corpus order — sorted by relpath, deterministic) plus totals.

func RunExamDetect added in v0.8.0 

func RunExamDetect(repoRoot string) (ExamDetectReport, error)

RunExamDetect loads the pinned corpus under repoRoot and grades the REAL core/detect.Detect output of every labeled scenario against its expected_detection ground truth. Offline, deterministic, LLM-free — no inference client is constructed anywhere on this path.

This is RunExamDetectExtra(repoRoot, "") — byte-identical to the prior behavior (no extra scenarios dir touched, no wire-shape change: Extra is always its zero value and thus omitted from JSON).

func RunExamDetectExtra added in v0.9.3 

func RunExamDetectExtra(repoRoot, extraScenariosDir string) (ExamDetectReport, error)

RunExamDetectExtra is RunExamDetect, additionally UNIONING the labeled scenarios found under extraScenariosDir (mallcoppro-f95) into the grading pass — a customer detector's OWN co-located efficacy scenarios (detectors/<name>/scenarios/*.yaml), loaded via LoadExtraScenarios (NO pin check, NEVER touching corpus.pin or the reference corpus's own digest). extraScenariosDir == "" reproduces RunExamDetect's exact prior behavior.

Every row this grades — reference or extra — runs through the IDENTICAL core/detect.Detect call over the SAME scenario/baseline projections; the only difference is provenance, carried on the row as Extra so callers (e.g. core/selfgate's customer-tree stage) can tell a detector's OWN proof scenarios apart from the reference corpus's.

func RunExamDetectOverCorpus added in v0.12.0 

func RunExamDetectOverCorpus(corpus Corpus, extraScenarios []LoadedScenario) ExamDetectReport

RunExamDetectOverCorpus grades corpus.Scenarios (Extra=false) UNIONED with extraScenarios (Extra=true) through the IDENTICAL core/detect.Detect grading loop RunExamDetectExtra uses — factored out as the corpus-SOURCE- AGNOSTIC seam (mallcoppro-bc2, `mallcop eval`'s C4 build) that lets a caller supply a corpus loaded any way it likes (LoadEmbedded, in particular: the shipped reference corpus baked into a customer deploy-repo binary, mallcop eval's default source) rather than only the on-disk pinned corpus Load(repoRoot) resolves. RunExamDetectExtra is now a thin wrapper: Load(repoRoot) + LoadExtraScenarios(dir), then this.

type ExamDetectRow added in v0.8.0 

type ExamDetectRow struct {
	// ScenarioID is the scenario's YAML id.
	ScenarioID string `json:"scenario_id"`
	// MustFire lists the detector family tokens that must appear among the
	// emitted findings' families (normalized lowercase).
	MustFire []string `json:"must_fire"`
	// MustNotFire lists the detector family tokens that must be absent.
	MustNotFire []string `json:"must_not_fire"`
	// Emitted lists the family token of every finding detect emitted, in
	// emission order (duplicates preserved — the visible multi-fire).
	Emitted []string `json:"emitted"`
	// Pass is true when every must_fire family is present and every
	// must_not_fire family is absent.
	Pass bool `json:"pass"`
	// Extra is true when this row came from an --extra-scenarios-dir UNION
	// (mallcoppro-f95) rather than the pinned reference corpus — a customer
	// detector's OWN co-located efficacy scenarios (detectors/<name>/scenarios/
	// *.yaml), graded through the identical real .wasm/detecthost path but
	// UNPINNED: they never touch corpus.pin and never count toward the
	// reference corpus's own integrity digest. Omitted (false) for every
	// ordinary reference-corpus row, so the wire shape is unchanged for every
	// existing caller that never passes an extra dir.
	Extra bool `json:"extra,omitempty"`
	// Reserved mirrors the scenario's expected_detection.reserved flag
	// (mallcoppro-db0) — an operator/requester-authored must-fire ground
	// truth for a detector that may not exist yet. Omitted (false) for every
	// ordinary row.
	Reserved bool `json:"reserved,omitempty"`
	// ReservedPending lists the normalized must_fire family tokens that are
	// STILL unregistered — no core/detect.Detectors() entry emits that family
	// in this process — on a Reserved row. Non-empty only when Reserved is
	// true. A row with ReservedPending is still Pass=false (the family
	// genuinely did not fire — it shows RED like any other unmet label) but
	// is EXCLUDED from ExamDetectTotals.Failed and instead counted in
	// ExamDetectTotals.Reserved: a TRACKED expected-miss, not a hard exam
	// failure. Every OTHER must_fire/must_not_fire label on the same row —
	// including a must_fire family whose detector HAS registered but still
	// doesn't fire, and any must_not_fire violation — is graded by the
	// ordinary hard rule and can still fail the row for real.
	ReservedPending []string `json:"reserved_pending,omitempty"`
}

ExamDetectRow is the per-labeled-scenario grading record.

type ExamDetectTotals added in v0.8.0 

type ExamDetectTotals struct {
	// Labeled is the number of scenarios carrying an expected_detection block.
	Labeled int `json:"labeled"`
	// Unlabeled is the number of scenarios WITHOUT the block — skipped from
	// grading but counted so coverage drift is visible.
	Unlabeled int `json:"unlabeled"`
	// Passed / Failed / Reserved partition the labeled set (Labeled ==
	// Passed+Failed+Reserved). Reserved counts rows whose ONLY reason for not
	// passing is a still-unregistered Reserved must_fire family
	// (ExamDetectRow.ReservedPending) — a TRACKED expected-miss, deliberately
	// excluded from Failed so a reserved-but-not-yet-authored detector never
	// hard-fails the exam or blocks CI (mallcoppro-db0).
	Passed   int `json:"passed"`
	Failed   int `json:"failed"`
	Reserved int `json:"reserved,omitempty"`
}

ExamDetectTotals aggregates the run.

type FailBin 

type FailBin string

FailBin is a classifier bin name. 

const (
	BinPass           FailBin = "PASS"
	BinNoInference    FailBin = "I3_no_inference"
	BinChainDrop      FailBin = "I2_chain_drop_no_terminal"
	BinShouldResolve  FailBin = "A1_invest_should_resolve_but_escalated"
	BinShouldEscalate FailBin = "A2_should_escalate_but_resolved"
	BinRubricAxisFail FailBin = "R_rubric_axis_fail"
)

type FalseAlarm added in v0.12.0 

type FalseAlarm struct {
	ScenarioID string   `json:"scenario_id"`
	Fired      []string `json:"fired_families"`
}

FalseAlarm names one must-stay-silent (benign) scenario that fired a family it was labeled to never fire — a precision failure. Fired lists the must_not_fire family tokens (normalized) that appeared among the emitted findings.

type FamilyCoverage added in v0.12.0 

type FamilyCoverage struct {
	// Family is the normalized detector family token (e.g. "priv-escalation").
	Family string `json:"family"`
	// AttackLabels is the number of graded rows that list this family in
	// must_fire — the recall denominator this family contributes.
	AttackLabels int `json:"attack_labels"`
	// BenignLabels is the number of graded rows that list this family in
	// must_not_fire — the precision denominator this family contributes.
	BenignLabels int `json:"benign_labels"`
	// Missed is the number of must_fire rows where this family did NOT appear
	// in the emitted findings and was not reserved-pending — a live recall gap.
	Missed int `json:"missed"`
	// FalseAlarms is the number of must_not_fire rows where this family DID
	// appear in the emitted findings — a live precision gap.
	FalseAlarms int `json:"false_alarms"`
	// Reserved is the number of must_fire rows where this family is
	// reserved-pending (no registered detector) — a tracked expected-miss, held
	// apart from Missed.
	Reserved int `json:"reserved"`
}

FamilyCoverage is one detector family's row in the coverage matrix.

func CoverageMatrix added in v0.12.0 

func CoverageMatrix(report ExamDetectReport) []FamilyCoverage

CoverageMatrix re-aggregates an exam-detect report into one FamilyCoverage row per detector family that appears in ANY row's must_fire or must_not_fire list. Families are returned sorted by token for a deterministic wire shape.

Pure and total: it never runs a detector or touches the corpus, and an empty report yields an empty (non-nil-safe, len 0) slice. A family that is only ever emitted incidentally (present in Emitted but never labeled must_fire/ must_not_fire on any row) is intentionally NOT a matrix row — the matrix tracks the LABELED coverage surface, not raw detector chatter.

type HarnessReport 

type HarnessReport struct {
	Mode           Mode              `json:"mode"`
	CorpusCount    int               `json:"corpus_count"`
	CorpusSHA      string            `json:"corpus_sha256"`
	Runs           []RunResult       `json:"runs"`
	MedianPassRate float64           `json:"median_pass_rate"`
	NoiseBandPP    float64           `json:"noise_band_pp"`
	WithinBand     bool              `json:"runs_within_band"`
	Classifier     ClassifierSummary `json:"classifier"`
	// DetectFidelity is the e2e-only block measuring how many scenario findings the
	// REAL detector fleet reproduced (vs mismatched/missed). Nil/zero for ModeCanned
	// and ModeReal (which inject the finding and so never exercise detect). It is the
	// headline result e2e exposes and must never be silently dropped.
	DetectFidelity *DetectFidelity `json:"detect_fidelity,omitempty"`
	// ReproducedPassRate is the e2e-only chain_action pass-rate over REPRODUCED
	// scenarios ONLY — the agent-reasoning number, directly comparable to ModeReal.
	// Computed as the median across runs of (passes among reproduced / reproduced
	// count). Zero for non-e2e modes.
	ReproducedPassRate float64 `json:"reproduced_pass_rate,omitempty"`
	// MedianRecallRate is the median across runs of RunResult.RecallRate — attacks
	// caught / attacks total. The honest attack-detection number, split out of the
	// blended MedianPassRate above (mallcoppro C2; mirrors recall.go's exam-detect
	// split at the agent-reasoning layer).
	MedianRecallRate float64 `json:"median_recall_rate"`
	// MedianPrecisionRate is the median across runs of RunResult.PrecisionRate —
	// benign scenarios correctly resolved / benigns total.
	MedianPrecisionRate float64 `json:"median_precision_rate"`
	// Note states, in the report itself, what the number means — so a reader can
	// never mistake the merge-gate's 100% for the accuracy number (§4.4), nor the
	// e2e all-scenario rate for the agent-reasoning rate. It also states, EXPLICITLY,
	// that MedianPassRate is BLENDED — the reader must consult MedianRecallRate /
	// MedianPrecisionRate (and, for e2e, DetectFidelity.E2ERecall/E2EPrecision) for
	// the honest attack-vs-benign split.
	Note string `json:"note"`
}

HarnessReport is the harness's full output: the corpus integrity facts, every run, the MEDIAN pass-rate + noise band, and the classifier summary.

func Run 

func Run(ctx context.Context, cfg RunConfig) (HarnessReport, error)

Run executes the harness: load → N passes → grade → median → classify.

The corpus integrity gate (count + SHA) runs FIRST via Load. A tampered or drifted corpus HARD-FAILS here and NOTHING runs (the eval interlock).

type LoadedScenario 

type LoadedScenario struct {
	// RelPath is the forward-slash path under exams/scenarios (e.g.
	// "auth/AF-01-fat-finger-benign.yaml"). Stable across OSes; used in the
	// manifest and as a provenance key.
	RelPath string
	// FileSHA is the lowercase-hex sha256 of the scenario file's bytes.
	FileSHA string
	// Scenario is the parsed, validated scenario (exam.Load).
	Scenario *exam.Scenario
}

LoadedScenario pairs a parsed scenario with its provenance: the corpus-relative path (stable id used in result filenames + the manifest) and the per-file content hash that fed the corpus digest.

func LoadExtraScenarios added in v0.9.3 

func LoadExtraScenarios(dir string) ([]LoadedScenario, error)

LoadExtraScenarios scans dir (any directory — e.g. a customer detector's own co-located detectors/<name>/scenarios/ sidecar, mallcoppro-f95) for scenario YAML files using the SAME leading-underscore skip and exam.Load parser scanCorpus applies to the pinned corpus, but performs NO pin verification and contributes NOTHING to any corpus.pin digest or count — this is the UNIONED, UNPINNED efficacy set RunExamDetectExtra grades alongside the reference corpus. It is never part of the reference corpus and never mutates it.

dir == "" (no extra scenarios shipped) returns (nil, nil) — the caller's union is then just the reference corpus, unchanged.

type MissedAttack added in v0.12.0 

type MissedAttack struct {
	ScenarioID string `json:"scenario_id"`
	// Missing is the must_fire family tokens (normalized) absent from the
	// emitted set — the detectors that should have fired and did not.
	Missing []string `json:"missing_families"`
	// Reserved is true when EVERY missing family is a tracked reserved-pending
	// gap (expected_detection.reserved: true and no detector for that family is
	// registered yet, mallcoppro-db0). The attack is genuinely not caught — so
	// it IS a recall miss, counted in the denominator like any other — but the
	// gap is KNOWN and awaiting an authored detector, not a regression. A miss
	// where any missing family has a registered detector reads as Reserved=false
	// so it surfaces as an ordinary fatal failure.
	Reserved bool `json:"reserved,omitempty"`
}

MissedAttack names one must-fire scenario whose attack was NOT fully detected — the fatal recall failure. Missing lists the must_fire family tokens (normalized) that did not appear among the scenario's emitted findings.

type Mode 

type Mode string

Mode selects the inference backend. 

const (
	// ModeCanned is the creds-free merge-gate: cannedbackend golden responses.
	ModeCanned Mode = "canned"
	// ModeReal is the parity run against a live model via DirectClient. WIRED but
	// not run here (no creds). ModeReal injects a SYNTHETIC finding built from the
	// scenario's YAML finding: block and calls agent.ResolveFindingWith directly —
	// the detector is bypassed.
	ModeReal Mode = "real"
	// ModeE2E is the END-TO-END scan run: it drives the REAL pipeline.Run (the same
	// function `mallcop scan` calls) so the scenario's raw events flow through
	// connect → detect.Detect (the PROD detector fleet) → store → resolve → store.
	// The finding the cascade resolves is whatever the REAL detectors emit, NOT the
	// YAML block. It uses the same {base_url,key} pivot client as ModeReal (or a
	// cannedbackend for the $0 canned-verification test). Its headline output is the
	// detect-fidelity block: how many scenario findings detect even reproduces.
	ModeE2E Mode = "e2e"
)

type PrecisionStat added in v0.12.0 

type PrecisionStat struct {
	// MustStaySilent is the denominator — scenarios with NO must_fire family (the
	// benign cases that must not raise their labeled must_not_fire families).
	MustStaySilent int `json:"must_stay_silent"`
	// CorrectSilent is the numerator — benign scenarios where none of the labeled
	// must_not_fire families fired.
	CorrectSilent int `json:"correct_silent"`
	// Rate is CorrectSilent/MustStaySilent, or 1.0 when there are no benign
	// scenarios.
	Rate float64 `json:"rate"`
	// FalseAlarms lists every benign scenario that fired a must_not_fire family,
	// named. Empty when precision is 100%.
	FalseAlarms []FalseAlarm `json:"false_alarms,omitempty"`
}

PrecisionStat is the precision side of the split: the benign scenarios and how many stayed correctly silent.

type RecallReport added in v0.12.0 

type RecallReport struct {
	Recall    RecallStat    `json:"recall"`
	Precision PrecisionStat `json:"precision"`
}

RecallReport is the recall-first view over an ExamDetectReport: recall (attacks caught) and precision (benign kept silent) reported separately, each with its failures named.

func RecallFromReport added in v0.12.0 

func RecallFromReport(report ExamDetectReport) RecallReport

RecallFromReport derives the recall-first split from a graded ExamDetectReport. Each labeled row is classified by its ground truth: a row with >=1 must_fire family is a MUST-FIRE (attack) scenario scored for recall; a row with no must_fire family is a MUST-STAY-SILENT (benign) scenario scored for precision. The split re-runs no detector — it reads only the labels and emitted families already on the rows — so it is as deterministic as the report it reads.

type RecallStat added in v0.12.0 

type RecallStat struct {
	// MustFire is the denominator — the number of scenarios labeled with at
	// least one must_fire family (the attacks that MUST be caught).
	MustFire int `json:"must_fire"`
	// Detected is the numerator — attack scenarios where EVERY must_fire family
	// appeared among the emitted findings.
	Detected int `json:"detected"`
	// Rate is Detected/MustFire, or 1.0 when there are no attack scenarios (a
	// vacuously perfect recall — avoids NaN in JSON).
	Rate float64 `json:"rate"`
	// Missed lists every attack that was NOT fully detected — the fatal failures,
	// named. Empty when recall is 100%.
	Missed []MissedAttack `json:"missed,omitempty"`
}

RecallStat is the recall side of the split: the attack scenarios and how many were fully caught.

type RunConfig 

type RunConfig struct {
	// Mode selects the backend: ModeCanned (merge-gate, golden responses) or
	// ModeReal (parity, live model). ModeReal requires creds (RealClientFromEnv).
	Mode Mode
	// N is the number of full-corpus passes for the median (§4.6). Default 3 when
	// <= 0 — single-run gating is forbidden.
	N int
	// Opts carries per-tier model ids + the (optional) live ToolRunner. The
	// merge-gate runs with a nil ToolRunner (golden responses need no live tools).
	Opts agent.CascadeOptions
	// RealClient is the live client for ModeReal. Ignored in ModeCanned (a fresh
	// golden cannedbackend is started per scenario). Must be non-nil for ModeReal.
	RealClient agent.Client
}

RunConfig configures one harness invocation.

type RunResult 

type RunResult struct {
	Index    int              `json:"index"`
	Results  []ScenarioResult `json:"results"`
	Passed   int              `json:"passed"`
	Total    int              `json:"total"`
	PassRate float64          `json:"pass_rate"`
	// Transcripts maps scenario id → the full captured transcript for this pass
	// (§4.7). Carried on the run so artifacts derive from a single source.
	Transcripts map[string][]TranscriptEntry `json:"-"`
	// Fidelity is the per-scenario detect-fidelity rows for this pass (ModeE2E
	// only; nil otherwise). detect is deterministic, so these are stable across
	// passes — the report aggregates run 0's rows.
	Fidelity []DetectFidelityRow `json:"-"`
	// ReproducedPassRate is the chain_action pass-rate over REPRODUCED scenarios
	// only for THIS pass (ModeE2E only) — the agent-reasoning number. Zero when no
	// scenario reproduced.
	ReproducedPassRate float64 `json:"reproduced_pass_rate,omitempty"`

	// --- recall/precision split (mallcoppro C2) -----------------------------
	//
	// PassRate above BLENDS two fundamentally different failure modes into one
	// number: a MISSED ATTACK (expected chain_action demands escalate, terminal
	// resolved — a fatal recall failure, the operator is never warned) and a
	// FALSE ALARM (expected resolved, terminal escalated — a precision failure,
	// noise). This mirrors recall.go's split of the exam-detect (detect-layer)
	// report, applied here to the harness's agent-reasoning ScenarioResult.Pass.
	//
	// Attacks/Benigns are the recall/precision denominators for THIS pass — a
	// scenario is an "attack" iff its expected chain_action demands escalate
	// (actionIsEscalate), else it is a "benign". The corpus is fixed, so these
	// counts are identical across every pass of a run (40 attacks / 18 benigns on
	// the shipped corpus) — but they are recorded per-run so a caller never has to
	// assume the split is constant.
	Attacks       int     `json:"attacks"`
	AttacksPassed int     `json:"attacks_passed"`
	RecallRate    float64 `json:"recall_rate"`
	Benigns       int     `json:"benigns"`
	BenignsPassed int     `json:"benigns_passed"`
	PrecisionRate float64 `json:"precision_rate"`
}

RunResult is one full corpus pass: the graded per-scenario results + the chain_action pass-rate for the pass.

type ScenarioResult 

type ScenarioResult struct {
	ScenarioID     string         `json:"scenario_id"`
	RelPath        string         `json:"rel_path"`
	FindingID      string         `json:"finding_id"`
	Category       string         `json:"category"`
	Difficulty     string         `json:"difficulty"`
	TerminalAction string         `json:"terminal_action"`
	ExpectedAction string         `json:"expected_chain_action"`
	TerminalReason string         `json:"terminal_reason"`
	ForceEscalated bool           `json:"force_escalated"`
	RouteID        string         `json:"route_id,omitempty"`
	ModelCalls     int            `json:"model_calls"`
	WallMillis     int64          `json:"wall_millis"`
	FullChain      []ChainStep    `json:"full_chain"`
	Structural     StructuralAxes `json:"structural"`
	// Pass is the harness verdict for this scenario: TRUE iff the chain_action
	// axis passed. The ONLY axis that gates (§4.4).
	Pass bool `json:"pass"`
}

ScenarioResult is the per-scenario, per-run result JSON (§4.4): scenario id, terminal action, the full chain, and the structural axes. This is the artifact every iteration tool (the classifier, the aggregator) consumes.

func Grade 

func Grade(run ScenarioRun) ScenarioResult

Grade turns a ScenarioRun into a graded ScenarioResult. DETERMINISTIC: identical inputs → identical output, no clock, no network, no LLM.

type ScenarioRun 

type ScenarioRun struct {
	Scenario       LoadedScenario
	TerminalAction string
	TerminalReason string
	ForceEscalated bool
	RouteID        string
	ModelCalls     int
	Transcript     []TranscriptEntry
	WallMillis     int64
	// SeedErr is non-empty when the per-scenario live ToolRunner could not be
	// seeded (a genuine store failure). On ModeReal this is a DEGRADED run — the
	// agent investigated without the scenario's live tools — and the harness can
	// surface it rather than silently scoring an empty-toolbox verdict.
	SeedErr string
}

ScenarioRun is the raw outcome of running ONE scenario through the core: the terminal Resolution, the captured transcript, and the call count. The grader (grader.go) turns this into a graded ScenarioResult. Kept separate so the runner stays free of grading policy.

func RunScenario 

func RunScenario(ctx context.Context, client agent.Client, ls LoadedScenario, opts agent.CascadeOptions, liveTools bool) ScenarioRun

RunScenario runs one scenario through the portable core IN-PROCESS against the supplied Client, capturing the full transcript. The Client is the {cannedbackend ⇄ real} pivot — RunScenario never decides which; the caller does.

LIVE TOOLS ARE WIRED PER SCENARIO when liveTools is true. opts carries the per-tier model ids; its Tools field (the ToolRunner) is OVERRIDDEN here with a per-scenario runner backed by the REAL core/tools over THIS scenario's events + baseline + finding. This is the whole point of the parity run: on ModeReal the live agent must investigate against the scenario's OWN telemetry — search-events returns the scenario's events (folding §3.8 matched_rules), check-baseline returns the scenario's frequencies, search-findings returns the scenario's finding stream. Each scenario sees ONLY its own data (per-scenario isolation, §4.1). A caller that pre-set opts.Tools is intentionally ignored: a shared/static runner would leak telemetry across scenarios and make the number meaningless.

liveTools is false for the MERGE-GATE (ModeCanned): golden responses prove the grader pipeline, not the model's tool use, and the live ToolEmpty fail-safe would inject model-independent escalations that have nothing to do with what a golden response is testing. On ModeReal liveTools is true and a real per-scenario runner is ALWAYS wired — the run NEVER silently investigates with an empty toolbox (the very gap this closes). If liveTools is true but the runner cannot be seeded (a genuine store failure), opts.Tools is left unchanged and the failure is recorded in SeedErr so the harness surfaces a degraded run rather than scoring an empty-toolbox verdict as if it were real.

The per-scenario store lives in a temp dir cleaned up before RunScenario returns.

type StructuralAxes 

type StructuralAxes struct {
	ChainAction  AxisResult `json:"chain_action"`
	TriageAction AxisResult `json:"triage_action"`
	Mentions     AxisResult `json:"mentions"`
	NoMentions   AxisResult `json:"no_mentions"`
	ToolsUsed    AxisResult `json:"tools_used"`
	Iterations   AxisResult `json:"iterations"`
}

StructuralAxes holds the per-axis grading for one scenario (§4.4). Only ChainAction gates pass/fail; the rest are reported provenance.

type TranscriptEntry 

type TranscriptEntry struct {
	// Seq is the 0-based call index within this scenario (triage=0,
	// investigate=1, escalate-formatter / deep-panel = 2+).
	Seq int `json:"seq"`
	// Model is the model id the cascade requested for this call (the tier).
	Model string `json:"model"`
	// System is the tier's system prompt (the ported POST.md).
	System string `json:"system"`
	// UserPrompt is the boxed user message text (USER_DATA-wrapped finding +
	// tool transcript) the model saw. This is where a planted injection is
	// visible — the audit value of §4.7.
	UserPrompt string `json:"user_prompt"`
	// Tools are the names of tools advertised on this call (the model's actual
	// API surface, §3.8).
	Tools []string `json:"tools,omitempty"`
	// Reply is the model's first text block — the verdict-carrying reply the
	// cascade parsed (or the escalate formatter's free-text alert).
	Reply string `json:"reply"`
	// Err is non-empty when the model call returned an error (the cascade
	// fail-safes such a call to escalate).
	Err string `json:"err,omitempty"`
}

TranscriptEntry is one model exchange the harness observed: the full request the cascade sent (model, system prompt, boxed user prompt, advertised tools) and the model's reply. §4.7 demands every tool call, input, output, and model message be captured — this is the unit.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.