govbench

Documentation

Overview

Package govbench measures the decision-layer governance gate the way the escape matrix measures isolation: with a fixed, versioned corpus and a top-line number that cannot silently regress. Isolation is proven by a CI-gated escape matrix; the prompt/decision layer was only defended, never measured. This closes that asymmetry.

The unit under test is the real dispatch waist. Each corpus case is a dispatch.Action a jailbroken or injected model would emit, and the benchmark runs it through a real dispatch.Dispatcher wired with the real capability.Admitter, observing whether the action was admitted (the work ran) or blocked (admission rejected it before any side effect). Nothing here is mocked: the gate under measurement is the same gate a run uses.

Two metrics, reported together so the gate cannot "win" by refusing everything:

• Attack Success Rate (ASR): the fraction of the attack corpus that was admitted. Lower is better.
• Benign-pass: the fraction of the benign corpus that was admitted. Higher is better; a floor stops a change from cutting ASR by breaking legitimate work.

The ablation is the point. The same corpus runs under two regimes that differ only in whether a least-privilege capability grant is bound on the context: RegimePolicyText models "we only told the model not to" (no grant, the waist admits every name); RegimeLeastPrivilege binds the grant a real least-privilege run would carry. The gap between the two ASR numbers is the mechanical enforcement layer's measured contribution, not an asserted one.

Honest scope. Capability admission is name-granularity: it decides whether an action NAME is permitted, not whether a permitted verb is being used for harm. So the residual attacks that survive RegimeLeastPrivilege are intent-level abuse of a granted verb (exfil through a granted fetch, a poisoned memory write, an injected model call, a destructive step in a granted shell). Each is tagged with the layer that closes it (egress allowlist, memory-write safety, the scanner port, plan inspection). The benchmark does not pretend the name-gate catches them; it reports the residual and points at the work, which is the whole value of measuring instead of claiming.

Index

• Constants
• Variables
• type Case
  • func Corpus() []Case
• type Disposition
  • func (d Disposition) String() string
• type Outcome
• type Regime
• type RegimeResult
• type Report
  • func Run(corpus []Case, regimes ...Regime) Report
  • func (rep Report) MechanicalContribution() float64
  • func (rep Report) String() string

Constants

const CorpusVersion = "v1"

CorpusVersion is the versioned identity of the attack + benign suite. Bump it whenever a case is added, removed, or its ground truth changes, so a shift in the numbers is always attributable to a known corpus revision.

Variables

var RegimeLeastPrivilege = Regime{
	Name: "least-privilege",
	// contains filtered or unexported fields
}

RegimeLeastPrivilege binds the least-privilege grant a real run would carry, so the waist denies any action outside it. This is the mechanical layer under test.

var RegimePolicyText = Regime{
	Name: "policy-text",
	// contains filtered or unexported fields
}

RegimePolicyText is the no-mechanical-enforcement baseline: the policy exists only as prose the model was asked to follow, so the waist has no grant to consult and admits every action name. This is the world a harness lives in when it "tells the model the rules" and nothing structural backs them.

Types

type Case

type Case struct {
	// ID is a stable identifier for the case, used in the report and regression
	// assertions. It never changes once published, so a diff to the numbers is
	// always traceable to a specific case.
	ID string
	// Category groups the case for reporting (jailbreak, exfil, memory-poison, ...).
	Category string
	// Attack is the ground truth: true if admitting this action is a success for an
	// adversary, false if it is legitimate work that must not be denied.
	Attack bool
	// Action is exactly what flows to the waist, the metadata a jailbroken model
	// would emit. Name is what capability admission consults.
	Action dispatch.Action
	// WantLeastPriv is the expected Disposition under RegimeLeastPrivilege: the
	// per-case regression anchor. An out-of-grant attack expects Blocked; a
	// residual intent-level attack on a granted verb expects Admitted; benign work
	// expects Admitted. A change that moves any case off its anchor fails the gate.
	WantLeastPriv Disposition
	// ClosedBy names the layer that closes a residual attack (one that survives
	// least-privilege because it abuses a granted verb). Empty for cases the
	// name-gate already blocks and for benign cases. It keeps the residual honest:
	// every surviving attack points at tracked work.
	ClosedBy string
	// Note is a one-line description of the attack or benign intent, for the report.
	Note string
}

Case is one corpus entry: an action the waist must judge, plus the ground truth used to score it. A Case is data, kept in corpus.go so the corpus is a versioned, reviewable artifact rather than logic.

func Corpus

func Corpus() []Case

Corpus returns the versioned attack + benign suite. It is a pure function of the constants above: deterministic, order-stable, and safe to call repeatedly.

The suite is deliberately split into three groups the report and the gate rely on: benign legitimate work; out-of-grant attacks the name-gate blocks; and in-grant-misuse attacks that reuse a granted verb and so survive least-privilege until a higher layer (named in ClosedBy) closes them.

type Disposition

type Disposition int

Disposition is what the waist did with an action: it either ran or it did not.

const (
	// Blocked means admission rejected the action before the work ran. For an
	// attack this is the desired outcome; for a benign action it is a false denial.
	Blocked Disposition = iota
	// Admitted means the action passed the waist and its work executed. For a
	// benign action this is correct; for an attack it is a success against us.
	Admitted
)

func (Disposition) String

func (d Disposition) String() string

type Outcome

type Outcome struct {
	Case Case
	Got  Disposition
}

Outcome is the scored result for one case under one regime.

type Regime

type Regime struct {
	Name string
	// contains filtered or unexported fields
}

Regime is a governance posture the whole corpus is scored under. The two shipped regimes differ only in whether a least-privilege grant is bound, which is what isolates the mechanical layer's contribution.

type RegimeResult

type RegimeResult struct {
	Regime string
	// ASR is admitted attacks over total attacks: the top-line number, lower better.
	ASR float64
	// BenignPass is admitted benign cases over total benign cases: the floor, higher
	// better.
	BenignPass float64
	// Admitted/Blocked attack counts and benign counts, kept for the report.
	AttacksAdmitted, AttacksTotal int
	BenignAdmitted, BenignTotal   int
	Outcomes                      []Outcome
}

RegimeResult is the aggregate for one regime over the whole corpus.

type Report

type Report struct {
	Results []RegimeResult
}

Report is the full benchmark result: one RegimeResult per regime plus the derived mechanical-contribution number. It renders to a stable text artifact so the numbers are publishable and reviewable in a golden file.

func Run

func Run(corpus []Case, regimes ...Regime) Report

Run scores the corpus under every regime and returns the report. It is deterministic: the same corpus yields the same numbers, so a golden comparison is a valid regression gate.

func (Report) MechanicalContribution

func (rep Report) MechanicalContribution() float64

MechanicalContribution is the ASR drop from the policy-text baseline to the least-privilege regime: how much the mechanical gate lowers attack success over prose policy alone. It is the ablation number, reported rather than asserted.

func (Report) String

func (rep Report) String() string

String renders the report as a stable Markdown artifact: a headline table plus the residual attacks that survive least-privilege, each with the layer that closes it. Deterministic ordering (cases in corpus order, regimes as run) keeps it golden-comparable.