replay

package
v0.0.10 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 2, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package replay serves a recorded session as a model and as tools, so hooks, transforms and fronts are tested against real traffic with nothing behind them.

Model serves the session's recorded model calls in path order: each response entry on the path is one call, served with the output items recorded before it, item entries naming its response ID and, from agentturn v0.0.15, custom entries marked with it, which hold an output item the filter kept from the model; each compaction entry is the fold that preceded the call after it, and each agentturn:compaction_failed entry is a fold that failed there, one call per attempt it made. In strict mode a request whose hash differs from the one recorded on the entry about to be served is refused with ErrDiverged, a fold's own request included, so a summary prompt, a budget or a filter that changed is not signed off as neutral; a call the record carries no hash for is refused with ErrUnverifiable rather than served unchecked, unless AllowUnhashed says to serve it. The exception is a fold through the compaction endpoint, which sends no request the format hashes and so is served unchecked in every mode: see Model.Compact. In lenient mode the calls are served by position and the hashes are reported through the observer. Tools wraps a tool list so that a call matching a recorded function call returns its recorded output instead of running.

A model and its tools are not the whole of what a call was made under. Model.BeforeModelCall serves back the instructions and the tool list in force at each recorded call, so a product whose layers rebuild them every turn from a store, a skill set or a memory block is replayed against the run rather than against what those layers would build today; Model.Settings holds the rest of what each call was made under. 

s, _ := store.Open(ctx, id)
model, _ := replay.NewModel(s, replay.Strict())
cfg.Model = model
cfg.BeforeModelCall = model.BeforeModelCall
cfg.Tools = replay.Tools(s, cfg.Tools, replay.Strict())

Index

Constants

This section is empty.

Variables

View Source 
var ErrCallIDRepeated = fmt.Errorf("%w: the path repeats a call ID", ErrUnverifiable)

ErrCallIDRepeated is returned by a strict NewModel for a path on which two function calls share a call ID, as a session recorded by agentturn v0.0.11 or earlier holds when its provider numbered calls per response. agentturn from v0.0.12 gives the second call an ID of its own, so the request after it cannot hash to what the recording sent, and a strict replay would diverge there naming two hashes and nothing about call IDs. It wraps ErrUnverifiable, and no option serves such a path strictly; a lenient replay serves it, and Tools serves the repeated call's recorded output.

View Source 
var ErrDiverged = errors.New("replay: diverged from the recording")

ErrDiverged is returned by a strict model when the request it is handed does not hash to the value recorded on the entry it would serve, and by strict tools for a call with no recorded output.

View Source 
var ErrExhausted = errors.New("replay: no further recorded call on the path")

ErrExhausted is returned when the path holds no further recorded call of the kind requested.

View Source 
var ErrSubstituted = fmt.Errorf("%w: the path changes workspace", ErrUnverifiable)

ErrSubstituted is returned by a strict NewModel for a path on which an env entry after a response names another workspace than the one in force before it, which before the first env entry is none: RFC 0001 calls that a substitution, and a reader that holds the environment fixed treats the path from it on as not verifiable against what came before it. The recorded outputs after it came from another file system than those before it, so a pass on one half says nothing about the other. It wraps ErrUnverifiable: nothing was measured to differ, and the record cannot be held to one environment. AllowSubstitution serves the path anyway.

View Source 
var ErrUnverifiable = errors.New("replay: the record cannot say what was sent")

ErrUnverifiable is returned when the record carries no hash for a call a strict replay would serve, so nothing can be checked against what the recording sent. It is not ErrDiverged: nothing was measured to differ, and the record simply does not say. NewModel returns it for a path whose responses are not all hashed, and a strict fold whose compaction entry recorded no fold hash returns it at the call; AllowUnhashed serves both unchecked instead. A path whose workspace was substituted is refused with it too, as ErrSubstituted.

Functions

func DefaultFoldText 

func DefaultFoldText(summary openresponses.Item) string

DefaultFoldText is the default WithFoldText: the text of a message summary with compact.SummaryMessage's prefix removed, or the text of any other item.

func Tools 

func Tools(s *agentsession.Session, tools []agenttool.Tool, opts ...Option) []agenttool.Tool

Tools wraps tools so that a call whose call_id, or whose name and arguments, match a function call with an output on the path returns that output instead of running. A call or an output the filter in force kept from the model, which agentturn v0.0.15 writes as a custom entry marked with session.ResponseIDMember, counts as the item it holds. Arguments are compared canonically, RFC 8785 over the JSON, the rule the request hash uses, so a serialiser that reorders members still matches; arguments that are not JSON compare byte for byte. A call with no recorded output runs the real tool, or fails with ErrDiverged when Strict. Each tool is wrapped with agenttool.Wrap, so it is the original in every way but Execute: its name, description and schema, and every property it declares, confinement, resource, annotations and replay safety among them, so a policy decides a replayed call as it decided the live one.

A served result carries the record its call's result carried as its Details, as a Record or as the type DetailsAs names, so a wrapper that acts on Details acts on a served call as it did on the live one. The record is the last custom entry naming the call in its call_id before the call's output, outside agentturn's own namespaces and other than a nested call's record, or the last in a namespace DetailsAs names when there is one. A record the tool wrote while it ran, through agenttool.WriteRecord, names the call too and cannot be told from its result's; a tool whose Details are not recordable and that wrote one is served that one. A session written before agentsession/0.6, which defined call_id, names no call; there the record is taken by position, when one call alone was waiting for its output, so a call of a parallel batch, or any call after one that never got an output, is served without one.

A session recorded by agentturn v0.0.11 or earlier may repeat a call ID its provider numbered per response. There an output belongs to the latest call before it with its ID, as RFC 0001 tells a reader, and every call is served: the loop now renames the repeat, so it is matched on its name and arguments rather than running its tool.

A session with no path to the leaf named holds no recordings, so every call falls through, or fails when Strict. From and AfterBase serve the outputs recorded after the entry they name, and one that is not on the path, or a base the header does not name, is the same as no path: nothing is served.

func Unverifiable added in v0.0.2 

func Unverifiable(s *agentsession.Session, leaf string, opts ...Option) error

Unverifiable reports whether a strict replay of the path to leaf, or to the session's current leaf when leaf is empty, could check every call it serves. It returns nil when it could, and an error wrapping ErrUnverifiable when it could not: one naming how many responses are unhashed, one wrapping ErrSubstituted naming the env entry that changed workspace, one wrapping ErrCallIDRepeated naming the call ID and both its calls' entries, or those that apply joined. An error that does not wrap ErrUnverifiable is the failure to resolve leaf to a path at all, which says nothing either way about the record. AllowUnhashed and AllowSubstitution among opts leave out what they allow, as they do for NewModel, and From or AfterBase counts the unhashed responses among the steps served; the other options are ignored.

The recorder writes a response without a request hash when the path it wrote does not rebuild that request's input: what a compacting configuration whose folds were never reported produces, and what a transform or a hook that edits the request produces. There is nothing to check such a call against, so a strict replay either refuses it or serves it unchecked, and which of those it does is AllowUnhashed.

This is the rule NewModel applies to the path it builds, exported so a caller can ask before building a model or running a suite rather than find out at the call.

Types

type Failure added in v0.0.8 

type Failure struct {
	// N is the step, numbered as [Served.N] is, and EntryID the entry
	// the failure was served from.
	N       int
	EntryID string
	Err     *openresponses.Error
	Cause   error
}

Failure is the error a replay model serves for a failed attempt the record holds: an agentturn:model_retry entry, a fold's summary call that failed, or a response entry that recorded a failed response, the last attempt of a run whose retries ran out. It names the step and the entry it was served from, so a run that ends on it, under a configuration that gives up where the recording retried, says it ended on a recorded failure rather than reading as a provider outage. It unwraps to Err, the openresponses error its recorded text spells or a 503 carrying the text, so a Retry.Retryable that reads the status decides as it did, and to Cause, the transport failure the text names when it names one, so a Retryable that retries only transport failures retries it too. The texts recognised, as the whole text or its last colon-separated part, are those of openresponses.ErrTruncatedStream, io.ErrUnexpectedEOF, io.EOF, os.ErrDeadlineExceeded ("i/o timeout"), context.DeadlineExceeded, and syscall.ECONNRESET, ECONNREFUSED, EPIPE, ETIMEDOUT, ENETUNREACH and EHOSTUNREACH, each served as itself; "no such host", served as a *net.DNSError; and a net/http client timeout, served as a net.Error whose Timeout is true. A text of the shape net/http's client gives every error it returns, `<Op> "<url>": <rest>`, is served as a *url.Error with that Op and URL whose Err is the error rest names, or, when rest names none, a net.Error whose Timeout reports whether rest says timeout, as net/http's response-header and TLS-handshake timeouts, which it names only by text, do. Each is a net.Error or one of the errors agentturn's DefaultRetryable names. A Retryable that declines every openresponses error before asking about the transport declines it: the record keeps the text alone, and the 503 is what gives it Retry-After. A failed response has no Cause and is served as the error it recorded.

func (*Failure) Error added in v0.0.8 

func (f *Failure) Error() string

func (*Failure) Unwrap added in v0.0.8 

func (f *Failure) Unwrap() []error

Unwrap returns Err, and Cause when there is one.

type Kind 

type Kind string

Kind says what a Served value describes. 

const (
	// KindResponse is a model call served from a response entry.
	KindResponse Kind = "response"
	// KindFold is a fold served from a compaction entry.
	KindFold Kind = "fold"
	// KindFailure is a failed attempt served from an
	// agentturn:model_retry entry, as the error the loop retries.
	KindFailure Kind = "failure"
	// KindFailedFold is one summary call of a fold that failed, served
	// from an agentturn:compaction_failed entry so the fold fails
	// again as it did.
	KindFailedFold Kind = "failed_fold"
	// KindCall is a tool call served from a recorded output.
	KindCall Kind = "call"
)

Kinds of served value.

type Model 

type Model struct {
	// contains filtered or unexported fields
}

Model is an openresponses.Streamer, and a compact.Compactor, served from a recorded session. It is safe for concurrent use, though a loop calls it from one goroutine at a time.

func NewModel 

func NewModel(s *agentsession.Session, opts ...Option) (*Model, error)

NewModel builds a model over the path from the root to the leaf named by WithLeaf, or to the session's current leaf, serving the steps after the entry From or AfterBase names when one is given. A strict model over a path a strict replay could not check is refused here with ErrUnverifiable, rather than at the call it could not check: see Unverifiable, and AllowUnhashed and AllowSubstitution to serve it anyway.

func (*Model) BeforeModelCall added in v0.0.2 

func (m *Model) BeforeModelCall(_ context.Context, req *openresponses.Request) error

BeforeModelCall serves the recorded settings of the call about to be served: it replaces the request's instructions and tool list with the ones in force at that response on the path. Chain it from the configuration under test, last, so the hook order stays the product's: 

inner := cfg.BeforeModelCall
cfg.BeforeModelCall = func(ctx context.Context, req *openresponses.Request) error {
	if inner != nil {
		if err := inner(ctx, req); err != nil {
			return err
		}
	}
	return model.BeforeModelCall(ctx, req)
}

Without it a strict replay measures the layers as they are today rather than the run: a product whose instructions are rebuilt each turn from a store, a skill set or a memory block diverges at the first call, and the error names two hashes and no layer. The other recorded settings are left alone, and Model.Settings holds them for a caller that wants to serve more.

func (*Model) Compact 

func (m *Model) Compact(_ context.Context, req openresponses.CompactRequest) (*openresponses.CompactResponse, error)

Compact serves the next fold, which must be next on the path, as a compaction response carrying the recorded summary item.

A fold through the compaction endpoint is served unchecked, in every mode, AllowUnhashed or not. It is the one call a strict model does not check and cannot: the endpoint takes a openresponses.CompactRequest rather than a request the format hashes, so the record holds nothing to compare it against and its absence is not ErrUnverifiable but the shape of the endpoint. The step order is still enforced. Served.Recorded carries the fold hash when the entry has one, which is a session recorded through a local fold and replayed through the endpoint; it is reported rather than checked, for the same reason.

func (*Model) Create 

func (m *Model) Create(ctx context.Context, req openresponses.Request) (*openresponses.Response, error)

Create serves the next recorded call as a complete response.

func (*Model) CreateStream 

func (m *Model) CreateStream(_ context.Context, req openresponses.Request, sink openresponses.EventSink) error

CreateStream serves the next recorded call. When that is a fold, the request must be a local fold's summary call and the compaction entry's summary is served as the answer; when it is a response, the response entry is served after its hash check in strict mode; when it is an attempt that failed, its failure is returned for the loop to retry, unchecked, since the record hashes only the attempt that answered.

func (*Model) Reset 

func (m *Model) Reset()

Reset makes the model serve from the start of the path again.

func (*Model) Served 

func (m *Model) Served() int

Served returns how many of them have been served.

func (*Model) Settings added in v0.0.2 

func (m *Model) Settings() []agentsession.Settings

Settings returns the settings in force at each recorded step, in path order and indexed as Served.N less one: the model, instructions, reasoning, text format, tools and passthrough members the config entries on the path say that call was made under. A judge, or a product checking what it sent, reads them here rather than replaying the config entries itself.

func (*Model) SettingsAt added in v0.0.2 

func (m *Model) SettingsAt(n int) (agentsession.Settings, bool)

SettingsAt returns the settings of step n, numbered as Served.N is. It reports false for a step the path does not hold.

func (*Model) Steps 

func (m *Model) Steps() int

Steps returns how many recorded calls, responses, folds and failed attempts, the path holds.

type Option 

type Option func(*options)

Option configures a Model or Tools.

func AfterBase added in v0.0.10 

func AfterBase() Option

AfterBase is From over the session header's Base, and wins over From when both are given: it serves the steps a fork recorded after the base it was forked at. A task forked through Runner.Header replays through the runner with it, since the runner seeds the fork's agent at the base. NewModel returns an error for a session whose header names no base; Tools then serves no recorded output, as for a From entry not on the path.

func AllowSubstitution added in v0.0.6 

func AllowSubstitution() Option

AllowSubstitution lets a strict model serve a path whose workspace was substituted part way, which it otherwise refuses with ErrSubstituted. The calls are still checked against their hashes; what the caller accepts is that the tool outputs before and after the substitution came from different file systems, as when a session recorded in a container was resumed on a laptop and the replay is meant to cover both halves.

func AllowUnhashed added in v0.0.2 

func AllowUnhashed() Option

AllowUnhashed lets a strict model serve a call whose entry recorded no hash: by position, and unchecked. It turns the guarantee off for those calls rather than relaxing it. What the caller gives up is the whole of what Strict is for — that every served call was checked against the request the recording made — for every call the record is silent about, and the replay's own result — that it finished without ErrDiverged — no longer distinguishes a call that was checked and matched from one that was never checked. Only an observer subscribing to Served can tell them apart afterwards: a model call was checked exactly when its Recorded and Got are both set.

It exists for recordings the format cannot describe: one made before agentturn v0.0.6, which wrote no fold member, and one whose requests a transform or a hook edited. Reach for it to replay an old session at all, not to quiet a failure on a current one.

func DetailsAs added in v0.0.5 

func DetailsAs[T agenttool.Recordable]() Option

DetailsAs serves the record of a call in T's namespace as a T, decoded from its JSON, where it would be a Record: a wrapper that acts on a result's Details by its type, such as one that grants a skill's tools when the Details are the skill's read, then sees what the live tool returned. A call with records in several namespaces is served the last one in a namespace given here. The namespace is the one T reports for its zero value, or for a new value when T is a pointer; it panics when T is an interface or reports no namespace, since it would match nothing. Several may be given, one per namespace. A record that does not decode as a T fails the call. A recorder on the replayed run writes the T as json.Marshal encodes it, which may differ from the recorded bytes in members T does not hold.

func From added in v0.0.10 

func From(entryID string) Option

From serves only the steps after the entry named, which must be on the path: for NewModel the model calls recorded after it, for Tools the outputs recorded after it. The settings still accumulate from the whole path, so a config entry before it is in force at the steps served, and a strict model checks substitution and repeated call IDs over the whole path too; only the unhashed responses it refuses are counted among the steps served. It is how a session whose first part another agent sent is replayed from an agent seeded with that part: a fork, whose agent starts at the base and never sends the base's requests, is AfterBase. NewModel returns an error for an entry that is not on the path; Tools, which has no error to return, then serves no recorded output, so under Strict every call diverges.

func Strict 

func Strict() Option

Strict makes a model refuse a request whose hash differs from the recorded one, and tools refuse a call with no recorded output. A fold is checked against the hash its compaction entry recorded for it as a response is against its own.

An entry that recorded no hash is refused rather than served unchecked, with ErrUnverifiable: NewModel refuses the whole session when a response on the path carries no request hash, before a call is served, and a fold whose compaction entry recorded no fold hash is refused at the call, because whether that matters depends on the compactor the replay is run with. AllowUnhashed serves them instead, which is how a recording made before agentturn v0.0.6 replays.

One call is served unchecked whatever is set: a fold through the compaction endpoint, which sends no request the format hashes, so there is nothing for strict mode to check and nothing for AllowUnhashed to allow. See Model.Compact.

The default is lenient: the model serves by position and reports the hashes through the observer, and an unmatched call runs the real tool.

func WithFoldText 

func WithFoldText(fn func(summary openresponses.Item) string) Option

WithFoldText says how the summary item of a compaction entry becomes the text the model answered a local fold with. The default undoes compact.SummaryMessage; a configuration that folds with its own WithSummaryItem passes the inverse here.

func WithLeaf 

func WithLeaf(id string) Option

WithLeaf names the path to serve. The default is the session's current leaf, which after judging is an outcome entry rather than a model output; a branched session has several leaves and a replay names the one it wants.

func WithObserver 

func WithObserver(fn func(Served)) Option

WithObserver sets a function called for everything served.

type Record added in v0.0.5 

type Record struct {
	NS   string
	Data json.RawMessage
}

Record is the Details of a served result whose recorded call carried a record: the namespace and data of the custom entry the recorder wrote beside the call from the result's agenttool.Recordable Details. It is itself Recordable, marshalling to the data as recorded, so a recorder on the replayed run writes the same record beside the served call. A wrapper that wants its own type back asks for it with DetailsAs.

func (Record) MarshalJSON added in v0.0.5 

func (r Record) MarshalJSON() ([]byte, error)

MarshalJSON returns the data as recorded.

func (Record) RecordNS added in v0.0.5 

func (r Record) RecordNS() string

RecordNS returns the namespace the recorded tool chose.

type Served 

type Served struct {
	Kind Kind
	// N is the 1-based position among the served calls of the model,
	// or of the tools.
	N int
	// EntryID is the entry served: the response, compaction or
	// model_retry entry for the model, the output's item entry for a
	// tool call.
	EntryID string
	// Recorded and Got are the hash recorded for the call and the hash
	// of the request received: the response entry's own hash, or, for
	// a fold, the one on its compaction entry's fold member. An entry
	// written before agentturn v0.0.6 carries no fold member at all,
	// and a response entry carries no hash when the path does not
	// rebuild that request's input; both leave Recorded empty, and a
	// strict model reports them only under [AllowUnhashed], having
	// otherwise refused the call.
	//
	// A fold served through [Model.Compact] is the exception, and the
	// one case where an empty Recorded on a strict replay does not mean
	// the caller opted out: the compaction endpoint sends no request
	// the format hashes, so that fold is served unchecked in every
	// mode, and Got is always empty there whatever Recorded holds.
	//
	// A failed attempt has no Recorded: the recorder hashes only the
	// request of the attempt that answered, and that one is checked
	// when it is served. Got is the hash of what the failed attempt
	// sent.
	//
	// Both are empty for a tool call, which is matched on its call ID
	// or its arguments rather than on a hash.
	//
	// Match reports whether the two agree, and is false when there was
	// nothing to compare. A model call — [KindResponse], [KindFold] or
	// [KindFailedFold] —
	// was checked exactly when Recorded and Got are both set, so Match
	// on its own is not a statement that it was. A tool call is the
	// other way round: it is reported only when a recorded output was
	// found, so its Match is always true and its hashes are never
	// consulted.
	Recorded, Got string
	Match         bool
	// CallID and Name describe a served tool call, and ByID reports
	// whether it matched on call ID rather than on name and arguments.
	CallID, Name string
	ByID         bool
}

Served is what the model and the tools report through the observer for each thing they serve.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.