replay

Documentation

Overview

Package replay compares the behaviour of two recorded agent sessions.

It answers the triage question "did the agent do something different this time, and where did it first diverge?" — the thing you actually want to know when a task that worked yesterday does not work today.

Behaviour, not prose

Comparison is over the sequence of tool calls, not over the assistant's text. Model output is nondeterministic: two runs of the same task almost always word things differently while doing exactly the same work, so diffing prose reports a difference on essentially every comparison and is useless as a signal. The tool calls are what changed the world, so they are what is compared.

Assistant text is still carried on each Turn so a reporter can show what was said around a divergence; it just does not decide whether a divergence happened.

Delegated work

A turn taken by a sub-agent is still a turn the run took, so sub-sessions are walked in place: their turns appear in sequence where the delegation happened. Skipping them would report "identical behaviour" for two runs whose sub-agents did entirely different things — the precise wrong answer to the question this package exists for.

First divergence only

Once two runs differ, everything after that point is downstream of the difference and comparing it produces noise, not information. So the comparison stops at the first divergence and reports where it was.

Index

• func PrintResult(out io.Writer, r Result, nameA, nameB string)
• type Divergence
• type Kind
• type Result
  • func Compare(a, b []Turn) Result
  • func CompareSessions(a, b *session.Session) Result
  • func (r Result) Identical() bool
• type ToolCall
• type Turn
  • func TurnsOf(sess *session.Session) []Turn

Functions

func PrintResult

func PrintResult(out io.Writer, r Result, nameA, nameB string)

PrintResult writes a human-readable comparison.

Types

type Divergence

type Divergence struct {
	Kind Kind `json:"kind"`
	// TurnIndex is the 0-based turn at which the runs first differ.
	TurnIndex int `json:"turn_index"`
	// A and B are the diverging turns. Either may be nil when one run had no
	// turn at this index.
	A *Turn `json:"a,omitempty"`
	B *Turn `json:"b,omitempty"`
}

Divergence is the first behavioural difference found.

type Kind

type Kind string

Kind classifies how two runs differ.

const (
	// KindToolCalls means both runs made a turn at this index but called
	// different tools, or the same tools with different arguments.
	KindToolCalls Kind = "tool_calls"
	// KindExtraTurn means the second run kept going after the first stopped.
	KindExtraTurn Kind = "extra_turn"
	// KindMissingTurn means the second run stopped before the first did.
	KindMissingTurn Kind = "missing_turn"
)

type Result

type Result struct {
	TurnsA int `json:"turns_a"`
	TurnsB int `json:"turns_b"`
	// TurnsMatched is how many leading turns behaved identically.
	TurnsMatched int `json:"turns_matched"`
	// Divergence is nil when both runs behaved identically throughout.
	Divergence *Divergence `json:"divergence,omitempty"`
}

Result is the outcome of a comparison.

func Compare

func Compare(a, b []Turn) Result

Compare walks two runs' turns and reports the first behavioural divergence.

func CompareSessions

func CompareSessions(a, b *session.Session) Result

CompareSessions is TurnsOf on both sides followed by Compare.

func (Result) Identical

func (r Result) Identical() bool

Identical reports whether the two runs behaved the same the whole way.

type ToolCall

type ToolCall struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCall is the comparable part of a tool invocation.

type Turn

type Turn struct {
	// Index is the turn's 0-based position in the session.
	Index int `json:"index"`
	// Agent is the agent that produced the turn, so a divergence in a
	// multi-agent run can be attributed.
	Agent string `json:"agent,omitempty"`
	// Content is the assistant text. Carried for reporting; never compared.
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Turn is one assistant turn: what it said, and what it called.

func TurnsOf

func TurnsOf(sess *session.Session) []Turn

TurnsOf extracts the assistant turns from a session, in order, including those taken by delegated sub-agents.

Only assistant messages are turns: user messages are inputs and tool messages are results, neither of which is a decision the model made. A nil session yields no turns.