Documentation
¶
Overview ¶
Package conformance replays recorded adapter transcripts.
Boundary: one replay engine, two consumers. `recall doctor --conformance` drives an adapter binary through a suite and diffs what it says against what was recorded; the evaluation runner replays the same recordings in place of a live source so a model-backed or network-backed adapter produces reproducible benchmark runs. Both read the same transcripts through Load, mask the same declared-volatile fields, and speak the same lockstep discipline. Nothing here ranks, fuses, or interprets a locator, and nothing here records: a transcript is evidence about an adapter, and the tool that writes one lives beside the adapter that produced it.
The format is normative in docs/adapter-protocol.md#conformance, with cmd/recall-stream/conformance/FORMAT.md as the worked example. Three rules are load-bearing enough to restate, because each one is why a transcript is replayable at all rather than a behavior that could be tuned:
- ${FIXTURE} and ${WORKDIR} are substituted textually, before parsing. Without them a transcript is bound to the absolute paths of the machine that recorded it. Deadlines are never substituted: every recorded request states a fixed far-future one, so the harness never rewrites a request field.
- Under flow "lockstep" a request waits for its own response before the next request is sent, and a notification is sent immediately. The second half is what makes a cancellation case recordable: waiting for the response to the search being cancelled would deadlock.
- The manifest's response count is enforced. An adapter that stops answering fails on the count rather than passing on a short list that happened to match as far as it went.
Index ¶
Constants ¶
const ( // DefaultResponseTimeout bounds the wait for one reply. It is generous: a // case that hits it has hung, and the difference between two seconds and // twenty does not change that. DefaultResponseTimeout = 20 * time.Second // DefaultDrainTimeout bounds the read that follows closing stdin. A process // that exits closes stdout and the drain ends immediately; the timeout is // for one that does not. DefaultDrainTimeout = 2 * time.Second // DefaultStopGrace is how long a clean exit has after stdin closes, before // the harness kills the process itself. DefaultStopGrace = 2 * time.Second )
Defaults for one replay.
const ( ManifestFile = "manifest.json" RequestFile = "request.jsonl" ResponseFile = "response.jsonl" FixtureDir = "fixture" )
The file names inside one case directory, fixed by docs/adapter-protocol.md#conformance.
const FlowLockstep = "lockstep"
FlowLockstep is the only dispatch discipline the format defines. A manifest naming anything else is refused rather than replayed on a guess: a harness that fell back to lockstep for an unknown flow would report a pass for a case it never ran the way its author meant.
Variables ¶
This section is empty.
Functions ¶
func Redact ¶
Redact masks a case's declared-volatile fields for RECORDING.
A transcript is committed, read in review, and diffed by people. A recorder that wrote frames verbatim put whatever the recording machine happened to hold into the repository: wall-clock timestamps, and worse, absolute paths naming the home directory of whoever ran it. Because Compare masks those same fields on both sides, such a transcript replays correctly everywhere and the leak stays invisible until somebody opens the file — which is exactly the kind of defect a recorded transcript exists to prevent, committed by the tool that records them.
Masking at record time makes the committed bytes say what the check actually asserts: this field is not compared. A frame that cannot be parsed is passed through unchanged, because a recorder is not the place to start rejecting an adapter's output — the replay that follows is.
Types ¶
type Bindings ¶
type Bindings struct {
// Fixture binds ${FIXTURE}.
Fixture string
// Workdir binds ${WORKDIR}.
Workdir string
}
Bindings are the machine-specific values a transcript's placeholders stand for. Both are absolute paths; Workdir must be fresh, writable, and empty for each case and each run, because reusing one would let a second replay observe a warm index the recording never had.
type Difference ¶
type Difference struct {
// Case is the case directory name.
Case string
// Response is the 1-based index of the frame, or 0 for a statement about
// the exchange as a whole.
Response int
// Pointer is an RFC 6901 JSON Pointer from the root of the frame. It is
// empty when the difference is about the frame itself rather than a value
// inside it.
Pointer string
// Want and Got are the recorded and replayed values, JSON-encoded and
// truncated for reading. Both are empty when Detail says everything.
Want string
// Got is the replayed value.
Got string
// Detail explains a difference that is not a plain value mismatch: a
// missing member, an unexpected one, a frame that never arrived.
Detail string
}
Difference is one place a replayed frame departed from the recording.
Pointer names the value, not the frame: a conformance failure is almost always one field, and a report that printed two 4KB frames and left the reader to spot it would be why nobody runs the suite.
func Compare ¶
func Compare(name string, want, got [][]byte, volatile []string) []Difference
Compare masks the declared-volatile fields on both sides and reports every place the replayed frames departed from the recorded ones.
Masking is applied to both sides rather than skipping the comparison, so a declaration covers a value's unpredictability without excusing its absence: a volatile /result/checked_at that the adapter stopped sending still fails.
func (Difference) String ¶
func (d Difference) String() string
type Manifest ¶
type Manifest struct {
// Case is the case directory name.
Case string `json:"case"`
// Description explains the behavior the transcript proves.
Description string `json:"description"`
// Flow is the transcript dispatch discipline.
Flow string `json:"flow"`
// Placeholders documents each ${NAME} token the requests carry and what a
// harness must bind it to. It is prose for a reader; [Transcript.Bind] is
// what enforces that every token was actually bound.
Placeholders map[string]string `json:"placeholders"`
// Volatile lists RFC 6901 JSON Pointers from the root of a response frame,
// with the segment "*" matching every element of an array and every member
// of an object. Both sides are masked before comparison.
Volatile []string `json:"volatile"`
// Responses is how many frames the adapter is expected to write, drain
// included.
Responses int `json:"responses"`
}
Manifest is <case>/manifest.json: what the case is, and how to replay it.
The protocol document fixes the file names and nothing else. Three things about a recorded exchange are specific to the machine that recorded it — the paths in its requests, the order its lines may be dispatched in, and the fields no adapter can reproduce — and this is where all three are settled.
type Options ¶
type Options struct {
// Workdir is the parent of each case's ${WORKDIR}. A fresh empty directory
// is created inside it per case and per run; an empty value uses the system
// temporary directory. Nothing is reused between cases, because an adapter
// that found a warm index would be replaying against state its recording
// never had.
Workdir string
// KeepWorkdir leaves each case's workdir behind for inspection.
KeepWorkdir bool
// ResponseTimeout bounds the wait for each response frame.
ResponseTimeout time.Duration
// DrainTimeout bounds the final read after stdin closes.
DrainTimeout time.Duration
}
Options tune one replay. The zero value uses the defaults above and a fresh temporary workdir per case.
type Player ¶
type Player struct {
// contains filtered or unexported fields
}
Player answers a live request stream from a recorded transcript.
This is the second thing a recording is for. A conformance run drives an adapter and checks what it says; an evaluation run needs the opposite — model-backed and network-backed adapters replay their recorded responses instead of running live, so a benchmark is reproducible. Both directions read the same transcripts through Load, which is why they are one package: a separate playback implementation would be free to disagree with the replayer about what a transcript means.
Matching is by position and method, and by nothing else. Params legitimately differ between the recording and the run — the workdir is new, and every deadline is an absolute instant — so comparing them would refuse every replay. A method arriving out of the recorded order is refused rather than answered from the nearest recorded frame: a source that hands back evidence for a question nobody asked is worse than a source that is unavailable.
func NewPlayer ¶
func NewPlayer(tr *Transcript) (*Player, error)
NewPlayer prepares a transcript for playback.
The correlation between recorded requests and recorded responses is checked here rather than mid-stream, because a transcript missing an answer is a defect in the pack and the run that discovers it should never have started.
func (*Player) Serve ¶
Serve answers requests read from in by writing the recorded frames to out. It returns when the stream ends, or when the caller asks for something the recording cannot answer.
Recorded ids are not echoed: the response carries the live request's id, so a caller numbering its own requests correlates them normally. Everything else — result, error code, timestamps — is the recording verbatim, which is the point of a fixture run.
type Process ¶
type Process struct {
// Stdin receives request frames from the replay engine.
Stdin io.WriteCloser
// Stdout supplies response frames to the replay engine.
Stdout io.Reader
// Stderr returns whatever the adapter has logged so far. It is attached to
// a failure report and never interpreted as protocol.
Stderr func() string
// Stop releases the process and must guarantee it is gone, and Stdout with
// it, when it returns. The format lets a case assume the adapter exits on
// its own after recall/shutdown or after stdin closes; a harness must still
// kill one that does not, and a Stop that left Stdout open would strand the
// reader waiting on it.
Stop func()
}
Process is one running adapter under replay.
It is stdio and nothing else, because that is all the transport is: the harness writes request lines to Stdin, reads frames from Stdout, and never parses Stderr. Stderr and Stop may be nil for a target that has neither.
type Result ¶
type Result struct {
// Case is the transcript case name.
Case string
// Dir is the absolute case directory.
Dir string
// Responses are the frames the adapter wrote, in the order it wrote them.
Responses [][]byte
// Volatile is the case's declared-volatile pointer list, carried here so a
// recorder can mask them with [Redact] before committing a transcript
// rather than writing a machine's clock and home directory into the tree.
Volatile []string
// Stderr is the adapter's free-form logging, captured for the report and
// never parsed.
Stderr string
// Stopped says why the harness stopped driving before it had sent every
// line, and is empty when it sent them all. An adapter that stops answering
// still fails on the response count; this is the reason to print beside it.
Stopped string
// Differences records every mismatch from the recorded response.
Differences []Difference
}
Result is what one case did.
func Replay ¶
Replay drives one transcript against target and diffs what came back.
The error return is for a harness failure — a workdir that cannot be created, a process that cannot be started. Everything the adapter did wrong is in the result, because a conformance run reports on adapters and only fails on itself.
func Verify ¶
Verify replays every case under root against target.
It returns a result per case, in suite order, and an error only when the suite itself could not be run. A failed case is a Result that is not OK, not an error: a caller reporting conformance wants every case's verdict, not the first one that went wrong.
type Target ¶
Target starts the adapter under test.
It is a function rather than a command string so the same engine can drive a binary, a wrapper around one, or an in-process server. The evaluation runner needs the last of those; a conformance run uses Command.
func Command ¶
Command runs an adapter binary with no arguments beyond the ones given, which is what a case may assume about how it is started.
The pipes are ours rather than exec's: exec's StdoutPipe closes on Wait, which races a reader still draining a frame. Owning the descriptors means the read side closes when the harness says so.
type Transcript ¶
type Transcript struct {
// Dir is the absolute case directory.
Dir string
// Manifest declares how the case is replayed.
Manifest Manifest
// Requests are the recorded request lines.
Requests [][]byte
// Recorded are the expected response lines.
Recorded [][]byte
}
Transcript is one loaded case directory.
Requests and Recorded hold the raw lines, not decoded frames. Replay sends the recorded bytes rather than a re-encoding of them, so nothing the harness does to a request — key order, number formatting, string escaping — can change what the adapter under test actually receives.
func Load ¶
func Load(dir string) (*Transcript, error)
Load reads one case directory and checks it against the format.
The checks here are about the transcript, not the adapter: a manifest that names a different case than its directory, or declares a response count its own recording does not have, is a defect in the suite, and finding it at load time is the difference between a confusing replay failure and a clear one.
func LoadSuite ¶
func LoadSuite(root string) ([]*Transcript, error)
LoadSuite reads every case directory under root, in name order.
A root holding no cases is an error. A suite that quietly lost its cases would otherwise report a clean pass, which is the one answer a conformance run must never give for having checked nothing.
func (*Transcript) Bind ¶
func (t *Transcript) Bind(b Bindings) ([][]byte, error)
Bind substitutes the placeholders into every request line.
A token left over after substitution is an error rather than a line sent as written: an adapter handed a literal "${WORKDIR}" would create a directory by that name and the case would fail somewhere far from the cause.
func (*Transcript) Fixture ¶
func (t *Transcript) Fixture() string
Fixture is the case's source data directory.
Source Files