Documentation
¶
Overview ¶
Package guardtrace is the end-to-end test/replay harness for `fak guard`.
`fak guard` is the productized form of fak's whole reason for existing: it stands up the gateway on loopback, points a child agent at it, and adjudicates EVERY tool call the agent proposes against a capability floor — allowing benign calls, denying the danger classes (rm -rf, sudo, curl-pipe-sh, writes into .ssh/.git), recording each verdict to a hash-chained decision journal, and surfacing the per-turn token / cache economy. Until now that whole path had no end-to-end exercise: the unit tests drove an in-process stub planner one turn at a time and never fired the floor while also asserting on the journal and the token numbers.
guardtrace closes that gap with a hermetic, no-API-key, no-GPU loop:
- A FIXTURE TRACE (Fixture) is an ordered script of turns. Each turn is one upstream model response carrying tool_use / tool_calls blocks AND a usage block; each call names the verdict the floor MUST reach (allow / deny + reason).
- A FAKE UPSTREAM (FakeUpstream) is an httptest.Server that replays the fixture turn-by-turn in a provider-correct wire shape (Anthropic /v1/messages or OpenAI /v1/chat/completions). Pointing the gateway's Config.BaseURL at it runs the REAL proxy planner + parse path — not a stub.
The gateway end-to-end test (internal/gateway) and the `fak guard --replay-trace` operator surface (cmd/fak) both build on this package, so the thing the operator watches in the terminal is the SAME loop the test asserts on.
Index ¶
- Constants
- func BuildInboundRequest(provider, model string, t Turn) ([]byte, error)
- func InboundRoute(provider string) string
- func LoadSessionTrace(path string) (t *bench.Trace, cas *engine.Cassette, engineID string, err error)
- func PostTurn(client *http.Client, gatewayURL, traceHeader, trace, provider, model string, ...) ([]byte, int, error)
- func SessionEngineID(raw []byte) string
- type CacheMetrics
- type Call
- type Expectation
- type FakeUpstream
- type Fixture
- type RequestMessage
- type ResponseAdjudication
- type Turn
- type Usage
Constants ¶
const SessionTracePrefix = "session:"
SessionTracePrefix marks a bench.Trace built FROM A CAPTURED SESSION (as opposed to a checked-in testdata suite like "tau2-smoke"). ablate's report provenance carries the trace's SliceID verbatim, so a report bound to a captured session is visibly distinct from the frozen mock suite at a glance (acceptance #1 of #1846: "provenance/workload hash is bound to real captured traffic, not tau2-smoke").
Variables ¶
This section is empty.
Functions ¶
func BuildInboundRequest ¶
BuildInboundRequest renders the per-turn request the CLIENT sends INTO the gateway, in the wire shape the route expects. Fixtures can provide a full client-side history in Turn.Messages; older fixtures get a minimal but well-formed default request with a system turn, one user turn, and the tool declarations for every tool the upcoming response will call. The gateway forwards this to the fake upstream, whose scripted response for this turn supplies the tool calls + usage.
func InboundRoute ¶
InboundRoute is the gateway HTTP route a wire posts its turns to.
func LoadSessionTrace ¶ added in v0.37.0
func LoadSessionTrace(path string) (t *bench.Trace, cas *engine.Cassette, engineID string, err error)
LoadSessionTrace loads a Fixture from path and returns the bench.Trace + the Cassette to register as its engine — the whole read-side pipeline `fak ablate --from-session` drives. engineID is derived from the fixture's own bytes (SessionEngineID) so the caller can register it deterministically.
func PostTurn ¶
func PostTurn(client *http.Client, gatewayURL, traceHeader, trace, provider, model string, t Turn) ([]byte, int, error)
PostTurn posts one turn's inbound request to the gateway at gatewayURL and returns the raw response body. traceHeader/trace, when non-empty, pin the session trace id so every turn shares one journal/session — the same X-Trace-Id a real wrapped agent would carry.
func SessionEngineID ¶ added in v0.37.0
SessionEngineID derives a stable, collision-resistant engine id for a captured session's cassette, so registering it (abi.RegisterEngine) never clashes with the built-in "mock"/"cassette"/"inkernel" ids or with another session's registration in the same process. Two loads of the SAME fixture bytes get the SAME id (content-addressed), so re-running --from-session against an unchanged file is idempotent instead of leaking a fresh registry entry each time.
Types ¶
type CacheMetrics ¶ added in v0.41.0
type CacheMetrics struct {
SliceID string `json:"slice_id"`
// Turns is the number of turns that carried a positive prompt (the turns cacheobs
// actually attributed) — normally every turn in the fixture.
Turns int `json:"turns"`
// PromptTokens is the summed resident prompt across the transcript (fresh input +
// cache_read + cache_creation per turn) — the denominator of the reuse ratio.
PromptTokens int `json:"prompt_tokens"`
// ReusedTokens is the summed cache_read across the transcript — the prefix tokens
// the provider served warm, i.e. the numerator of the reuse ratio.
ReusedTokens int `json:"reused_tokens"`
// CacheCreationTokens is the summed cache_creation across the transcript (prompt
// tokens freshly written to the cache), surfaced so a golden diff shows which axis moved.
CacheCreationTokens int `json:"cache_creation_tokens"`
// ShedTokens is the total resident prompt REMOVED from the window across the
// transcript: the sum of each turn's positive drop in resident prompt versus the turn
// before it (the internal/ablate ShedTokens sense). An append-only trajectory never
// sheds (0); a compaction/prune fire that collapses the window sheds the collapse.
ShedTokens int `json:"shed_tokens"`
// ReuseRatio is the aggregate realized cache-hit ReusedTokens/PromptTokens as
// computed by cacheobs, rounded to 6 decimals so the golden is float-stable across
// platforms and Go versions.
ReuseRatio float64 `json:"reuse_ratio"`
// FrozenTurns / PartialTurns / ColdTurns bucket the turns by per-turn reuse ratio
// (cacheobs FrozenFloor / ColdCeil) — the regime shape a single aggregate ratio hides.
FrozenTurns int `json:"frozen_turns"`
PartialTurns int `json:"partial_turns"`
ColdTurns int `json:"cold_turns"`
}
CacheMetrics is the frozen shed/reuse projection of ONE canonical transcript. It is the row the golden regression pins: every field is a deterministic fold of the fixture's recorded per-turn usage, so two folds of the same fixture are byte-identical and any change to the derivation (or to the recorded transcript) shifts a field and reds the golden until it is regenerated with -update.
type Call ¶
type Call struct {
ID string `json:"id"`
Tool string `json:"tool"`
Args json.RawMessage `json:"args"`
// Class is the disposition the guard floor MUST reach for this call: "allow" (the
// call survives to the caller) or "deny" (the floor drops it). Reason, when set,
// is the closed-vocabulary refusal code the deny must carry (e.g. POLICY_BLOCK,
// SELF_MODIFY) — asserted so a deny for the WRONG reason is caught, not just any deny.
Class string `json:"class"`
Reason string `json:"reason,omitempty"`
}
Call is one proposed tool call in a turn, with the verdict the floor must reach.
func (Call) ArgPreview ¶
ArgPreview is a short, single-line form of the call's most salient argument for a human report line (the command for Bash, the file_path for a write). It never returns the whole args blob, so a replay report line stays one row.
func (Call) ArgString ¶
ArgString renders the call's args as the compact JSON the model would emit — the form the upstream wire carries (Anthropic tool_use.input object; OpenAI tool_calls function.arguments string).
func (Call) ExpectAllow ¶
ExpectAllow reports whether this call must survive the floor.
type Expectation ¶
type Expectation struct {
TotalCalls int
Allowed int
Denied int
// ByReason counts the deny reasons the fixture declares (e.g. POLICY_BLOCK -> 2).
ByReason map[string]int
// CacheReadTokens / CacheCreationTokens / InputTokens are the summed provider usage
// axes across every turn — the token economy the summary surfaces.
CacheReadTokens int
CacheCreationTokens int
InputTokens int
}
Expectation folds a fixture into the aggregate counts the gateway's AdjudicationSummary must report after the whole trace runs, so a caller can assert the roll-up the exit banner prints in one comparison.
func (Expectation) Reasons ¶
func (e Expectation) Reasons() []string
Reasons returns the declared deny reasons in stable order (for a deterministic report).
type FakeUpstream ¶
FakeUpstream is an httptest.Server that stands in for a real model provider. It replays a fixture turn-by-turn: each inbound completion request pops the next turn and answers with a provider-correct response carrying that turn's tool_use / tool_calls blocks AND its usage block. Pointed at by the gateway's Config.BaseURL, it drives the REAL proxy planner + parse path, so the gateway adjudicates genuine upstream-shaped tool calls and accounts genuine provider-reported token usage — "a trace that leads to token work", with no API key and no GPU.
func NewFakeUpstream ¶
func NewFakeUpstream(provider, model string, f *Fixture) *FakeUpstream
NewFakeUpstream starts a fake provider upstream for the given wire ("anthropic" or "openai") that replays f's turns. The caller closes it via Close().
func (*FakeUpstream) Hits ¶
func (u *FakeUpstream) Hits() int
Hits is the number of completion requests served so far (for diagnostics).
type Fixture ¶
Fixture is a parsed guard trace: an ordered list of turns the fake upstream replays.
func LoadFixture ¶
LoadFixture reads and parses a guard trace fixture from disk, validating that every call names a known class so a typo fails loud at load rather than silently passing the assertions. It is the one parse path the test and the CLI share.
func ParseFixture ¶
ParseFixture parses fixture bytes (the path-free core of LoadFixture).
func (*Fixture) CacheMetrics ¶ added in v0.41.0
func (f *Fixture) CacheMetrics() CacheMetrics
CacheMetrics folds the fixture's recorded per-turn usage into the frozen shed/reuse row. The reuse ratio + regime buckets come from a fresh cacheobs.Observer (the canonical metric, per-fixture isolated so it never touches the process-global Default); the shed count is the summed positive resident-prompt drop turn over turn.
func (*Fixture) Expect ¶
func (f *Fixture) Expect() Expectation
Expect computes the aggregate expectation over the whole fixture.
func (*Fixture) ToBenchTrace ¶ added in v0.37.0
ToBenchTrace converts a Fixture into a *bench.Trace: one bench.Call per fixture call, in turn order, flattened across all turns. The trace's SliceID is SessionTracePrefix + the fixture's SliceID (or "captured" if the fixture left it blank), so WorkloadHash — and every report/provenance field derived from it — is bound to the captured session, never confusable with a suite trace.
func (*Fixture) ToCassette ¶ added in v0.37.0
ToCassette builds an engine.Cassette that answers every call in the fixture with the REAL per-turn usage the session recorded — carrying the input / cache_read / cache_creation / output split onto whichever CassetteEntry matches that call's (tool, args), so an ablate arm replayed with --engine cassette:<id> (see RegisterSessionEngine) reports actual billed usage, not a synthesized count (acceptance #2 of #1846).
A caveat inherent to the cassette's (tool, args) keying: if the SAME (tool, args) pair repeats across turns with DIFFERENT usage (e.g. a cache_read that grows turn over turn), the cassette can only bind ONE usage value per key — the LAST turn's usage for that pair wins. This does not affect the aggregate token totals bench.RunArm sums (each trace call still contributes once), only which turn's per-call usage a duplicate (tool,args) reports if inspected individually.
type RequestMessage ¶ added in v0.37.0
type RequestMessage struct {
Role string `json:"role"`
Content string `json:"content"`
Repeat int `json:"repeat,omitempty"`
}
RequestMessage is one client-side history span posted into the gateway during replay.
func (RequestMessage) ExpandedContent ¶ added in v0.37.0
func (m RequestMessage) ExpandedContent() string
type ResponseAdjudication ¶
ResponseAdjudication is the per-call verdict the gateway returned in its `fak` extension, decoded from EITHER wire's response into one shape so the test and the CLI read verdicts uniformly.
func DecodeAdjudications ¶
func DecodeAdjudications(raw []byte) ([]ResponseAdjudication, error)
DecodeAdjudications extracts the gateway's per-call verdicts from a turn response on either wire. Both the Anthropic and the OpenAI response carry the same `fak` extension shape, so one decoder serves both.
type Turn ¶
type Turn struct {
Note string `json:"note,omitempty"`
// Messages, when present, is the client history the replay posts into the gateway for
// this turn. Older fixtures omit it and get the compact default request; context
// fixtures use it to drive ctx-view over the same HTTP path as guard/serve.
Messages []RequestMessage `json:"messages,omitempty"`
Usage Usage `json:"usage"`
Calls []Call `json:"calls"`
}
Turn is one upstream model response: the tool calls it proposes plus the token usage the provider would report for that turn.
type Usage ¶
type Usage struct {
InputTokens int `json:"input_tokens"`
OutputTokens int `json:"output_tokens"`
CacheReadInputTokens int `json:"cache_read_input_tokens"`
CacheCreationInputTokens int `json:"cache_creation_input_tokens"`
}
Usage is the provider-reported token accounting for one turn. Field names match BOTH the Anthropic Messages usage object and what the OpenAI usage object is mapped from, so one fixture drives both wires.
Source Files