loopgate

Documentation

Overview

loopbench measures what the witnessed exit-gate earns over a naive Ralph loop that terminates on the agent's own self-reported "done". It runs the SAME fixed task corpus through two policies — naive (accept the first self-report) and gated (accept only an externally witnessed done, re-arming otherwise via Adjudicate) — and reports the delta the gate removes: the false-done rate, the slop shipped under a false done, the wasted iterations, and the gate's own per-turn cost so the win is reported NET (the repo's net-true doctrine, docs/standards/net-true-value.md).

The corpus is a SIMULATED fixture (hand-authored turn traces representative of real loop runs); the report labels its provenance and names the live witness that remains `not yet`. Nothing here shells out or needs a host: the gated policy drives the real loopgate.Adjudicate decision against a witness adapter derived from each turn's ground truth, so the bench exercises the shipping gate rather than a strawman reimplementation of it.

Package loopgate adjudicates a loop turn's self-reported "done" claim against an external witness. The package owns no process spawning: callers provide the witness function, so unit tests use fixtures and production hosts can bind the request to dos commit-audit, dos verify, or another witness verb.

Index

• Constants
• type BenchTurn
• type Criterion
• type CriterionKind
• type Decision
  • func Adjudicate(ctx context.Context, turn Turn, witness WitnessFunc) Decision
• type Episode
  • func DefaultBenchCorpus() []Episode
• type LoopStats
• type NetTrue
• type Report
  • func CompareLoops(corpus []Episode) Report
  • func (r Report) JSON() ([]byte, error)
• type Request
  • func (r Request) Argv() []string
• type Turn
• type Verdict
• type WitnessFunc
• type WitnessOutcome
• type WitnessResult
  • func CommitAuditResultFromJSON(data []byte) (WitnessResult, error)
  • func GenericWitnessResultFromJSON(data []byte) (WitnessResult, error)
  • func TestWitnessResultFromJSON(data []byte) (WitnessResult, error)
  • func VerifyResultFromJSON(data []byte) (WitnessResult, error)

Constants

const (
	// ReasonDoneUnwitnessed is the re-arm reason for a done claim that no
	// external witness corroborated. It is declared in dos.toml so loop drivers
	// can surface a closed reason token rather than prose.
	ReasonDoneUnwitnessed  = "LOOP_DONE_UNWITNESSED"
	ReasonSchemaUnreadable = "SCHEMA_UNREADABLE"
)

Types

type BenchTurn

type BenchTurn struct {
	// ClaimedDone is whether the agent self-reported "done" this turn — the
	// only signal a naive Ralph loop consumes.
	ClaimedDone bool `json:"claimed_done"`
	// Witnessed is whether an external witness (commit-audit OK / verify
	// shipped) would corroborate the done claim this turn — the ground truth
	// the gate checks and the naive loop ignores.
	Witnessed bool `json:"witnessed"`
	// SlopUnits is the slop-scorecard delta the turn's commit introduces if it
	// is shipped as the loop's accepted answer.
	SlopUnits int `json:"slop_units"`
	// TurnTokens is the cost of running this turn — used to price the rework a
	// false done pushes downstream.
	TurnTokens int `json:"turn_tokens"`
	// GateCostTokens is the dos-adjudication overhead the gate pays when it
	// adjudicates this turn's claim.
	GateCostTokens int `json:"gate_cost_tokens"`
}

BenchTurn is one turn's ground truth in a benchmarked loop episode.

type Criterion

type Criterion struct {
	Kind      CriterionKind
	Ref       string
	Plan      string
	Phase     string
	Source    string
	Subject   string
	Baseline  string
	Candidate string
}

Criterion is the goal's witness requirement. Empty Kind defaults to commit-audit over the turn ref.

type CriterionKind

type CriterionKind string

CriterionKind names the witness surface that should adjudicate the turn.

const (
	CriterionCommitAudit     CriterionKind = "commit-audit"
	CriterionVerify          CriterionKind = "verify"
	CriterionTestWitness     CriterionKind = "test-witness"
	CriterionCitationResolve CriterionKind = "citation-resolve"
	CriterionWitness         CriterionKind = "witness"
	CriterionMetric          CriterionKind = "metric"
)

type Decision

type Decision struct {
	Verdict Verdict `json:"verdict"`
	Reason  string  `json:"reason,omitempty"`
	Summary string  `json:"summary,omitempty"`
	Request Request `json:"request,omitempty"`
	Witness string  `json:"witness,omitempty"`
}

Decision is the gate's full typed result.

func Adjudicate

func Adjudicate(ctx context.Context, turn Turn, witness WitnessFunc) Decision

Adjudicate maps a turn's done claim and witness criterion to the loop exit verdict. A done claim is accepted only on OutcomeWitnessed. Unwitnessed claims re-arm with ReasonDoneUnwitnessed. Malformed criteria or witness adapter failures terminate as structured refusals.

type Episode

type Episode struct {
	ID    string      `json:"id"`
	Turns []BenchTurn `json:"turns"`
}

Episode is one fixed task's turn trace. Turns run in order; the loop reads one turn per iteration.

func DefaultBenchCorpus

func DefaultBenchCorpus() []Episode

DefaultBenchCorpus is the fixed, SIMULATED task set. It mixes honest episodes (the agent's "done" coincides with the witnessed done) with trap episodes (a premature self-reported "done" the witness refutes), so the naive loop's false-done rate is non-zero and separable from the gate's. The numbers are a fixture, not a measurement — see Report.Provenance.

type LoopStats

type LoopStats struct {
	Policy string `json:"policy"`
	// Episodes is the corpus size scored.
	Episodes int `json:"episodes"`
	// AcceptedDone is the count of episodes the loop terminated on a "done".
	AcceptedDone int `json:"accepted_done"`
	// FalseDone is the count of episodes whose accepted "done" no witness
	// corroborated (the naive loop's CLAIM_UNWITNESSED / NOT_SHIPPED stops).
	FalseDone int `json:"false_done"`
	// FalseDoneRate is FalseDone / AcceptedDone — the headline number.
	FalseDoneRate float64 `json:"false_done_rate"`
	// WitnessedDoneReached is the count of episodes the loop drove to a
	// genuinely witnessed done.
	WitnessedDoneReached int `json:"witnessed_done_reached"`
	// SlopShipped is total slop units of the commits the loop accepted as done.
	SlopShipped int `json:"slop_shipped"`
	// ItersToAcceptedDone is total turns run before each accepted "done"
	// (1-based; the accepted turn counts).
	ItersToAcceptedDone int `json:"iters_to_accepted_done"`
	// WastedIterations is total turns spent shipping-broken after a false done
	// before the true witnessed done is reached — the rework the false stop
	// pushed downstream.
	WastedIterations int `json:"wasted_iterations"`
	// ReworkTokens prices WastedIterations: the turn tokens the downstream
	// rework costs because the loop shipped a false done.
	ReworkTokens int `json:"rework_tokens"`
	// GateCostTokens is the dos-adjudication overhead this policy paid (zero
	// for the naive loop, which never adjudicates).
	GateCostTokens int `json:"gate_cost_tokens"`
}

LoopStats is one policy's outcome over the corpus.

type NetTrue

type NetTrue struct {
	// ReworkTokensAvoided is the downstream rework the naive loop pays that the
	// gate prevents by refusing the false done.
	ReworkTokensAvoided int `json:"rework_tokens_avoided"`
	// GateCostTokens is what the gate spent adjudicating to prevent it.
	GateCostTokens int `json:"gate_cost_tokens"`
	// NetTokens is ReworkTokensAvoided − GateCostTokens; positive means the
	// gate earns its keep on this corpus.
	NetTokens int `json:"net_tokens"`
	// GateEarnsKeep is NetTokens > 0.
	GateEarnsKeep bool `json:"gate_earns_keep"`
}

NetTrue reports the gate's win net of its own cost, per docs/standards/net-true-value.md (criterion 2: net of the cost the change itself adds).

type Report

type Report struct {
	Benchmark string `json:"benchmark"`
	// Provenance labels the corpus — SIMULATED here (a fixture), never quoted
	// as a measured live-agent number.
	Provenance string `json:"provenance"`
	// Corpus is the fixed task set, named so a re-run is comparable.
	Corpus string `json:"corpus"`
	// Naive and Gated are the two policies' stats over the same corpus.
	Naive LoopStats `json:"naive"`
	Gated LoopStats `json:"gated"`
	// FalseDoneRateDelta is Naive.FalseDoneRate − Gated.FalseDoneRate — the
	// rate the exit-gate removes. Positive is the claim of the epic.
	FalseDoneRateDelta float64 `json:"false_done_rate_delta"`
	// SlopShippedDelta is Naive.SlopShipped − Gated.SlopShipped.
	SlopShippedDelta int     `json:"slop_shipped_delta"`
	NetTrue          NetTrue `json:"net_true"`
	// Finding is a one-line human summary — including the honest "corpus too
	// easy to separate them" reading if the delta is zero.
	Finding string `json:"finding"`
	// NotYet names the live witness this fixture bench does not yet provide.
	NotYet string `json:"not_yet"`
}

Report is the full verified-vs-naive comparison.

func CompareLoops

func CompareLoops(corpus []Episode) Report

CompareLoops runs the corpus through both policies and folds the four metrics, net of gate cost.

func (Report) JSON

func (r Report) JSON() ([]byte, error)

JSON renders the report as the stable, re-runnable bench artifact.

type Request

type Request struct {
	Kind      CriterionKind `json:"kind"`
	Ref       string        `json:"ref,omitempty"`
	Plan      string        `json:"plan,omitempty"`
	Phase     string        `json:"phase,omitempty"`
	Source    string        `json:"source,omitempty"`
	Subject   string        `json:"subject,omitempty"`
	Baseline  string        `json:"baseline,omitempty"`
	Candidate string        `json:"candidate,omitempty"`
	Claim     string        `json:"claim,omitempty"`
}

Request is the normalized witness call the loop host must satisfy.

func (Request) Argv

func (r Request) Argv() []string

Argv returns the dos CLI argv corresponding to this request, excluding the leading "dos" binary. It is a convenience for hosts that bind the gate to the CLI; Adjudicate itself never shells out.

type Turn

type Turn struct {
	ClaimedDone bool
	Claim       string
	HeadRef     string
	Criterion   Criterion
}

Turn is the loop turn state visible to the gate.

type Verdict

type Verdict string

Verdict is the exit-gate decision a loop driver consumes.

const (
	VerdictWitnessed Verdict = "WITNESSED"
	VerdictNotYet    Verdict = "NOT_YET"
	VerdictRefused   Verdict = "REFUSED"
)

type WitnessFunc

type WitnessFunc func(context.Context, Request) (WitnessResult, error)

WitnessFunc satisfies one normalized witness request.

type WitnessOutcome

type WitnessOutcome string

WitnessOutcome is the normalized outcome returned by the witness adapter.

const (
	OutcomeWitnessed WitnessOutcome = "witnessed"
	OutcomeNotYet    WitnessOutcome = "not_yet"
	OutcomeRefused   WitnessOutcome = "refused"
)

type WitnessResult

type WitnessResult struct {
	Outcome    WitnessOutcome `json:"outcome"`
	Reason     string         `json:"reason,omitempty"`
	Detail     string         `json:"detail,omitempty"`
	RawVerdict string         `json:"raw_verdict,omitempty"`
	Rung       string         `json:"rung,omitempty"`
}

WitnessResult is the adapter's evidence summary.

func CommitAuditResultFromJSON

func CommitAuditResultFromJSON(data []byte) (WitnessResult, error)

CommitAuditResultFromJSON folds `dos commit-audit --json` into the normalized witness outcome. A range is witnessed only when at least one row is OK and no row reports an unwitnessed claim.

func GenericWitnessResultFromJSON

func GenericWitnessResultFromJSON(data []byte) (WitnessResult, error)

GenericWitnessResultFromJSON folds `dos witness --json` best-effort. The generic verb is plugin-shaped, so this parser accepts the stable fields the built-in renderer exposes while failing closed on unknown shapes.

func TestWitnessResultFromJSON

func TestWitnessResultFromJSON(data []byte) (WitnessResult, error)

TestWitnessResultFromJSON folds `dos test-witness --json`.

func VerifyResultFromJSON

func VerifyResultFromJSON(data []byte) (WitnessResult, error)

VerifyResultFromJSON folds `dos verify --json`.