Documentation
¶
Overview ¶
Package eval is a benchmark for agentic memory and handoff.
Why another benchmark ¶
LongMemEval and its relatives measure one thing well: given a question, can the store surface the session holding the answer. That is recall, and recall is the easy half. It says nothing about the question this project exists to answer — an agent stopped mid-task, a *different* agent arrives, and the user re-explains nothing. Nobody scores that, so nobody optimises for it.
So this suite measures continuity: what survives the boundary between two agents. The metric that matters most is the one no recall benchmark has a slot for — whether the approaches already ruled out make it across. Anyone can restate a goal. The expensive knowledge is the three things that didn't work, and it is exactly what dies when a session ends.
The honest half ¶
A benchmark whose author picks the categories is a benchmark its author wins. The counterweight is deliberate: this suite includes families brain is known to be bad at, several of them found by an agent criticising brain's own output during a live handoff — stale context presented without its age, circular sourcing where prose restates a claim the data contradicts, and abstention, which the LongMemEval harness in memory/bench.go explicitly filters out before scoring. Those cases are marked Expect: Fail below. If a change makes them pass, that is progress; if a change makes a passing case fail, the suite says so.
What is compared ¶
Every system under test implements Adapter — write events, read back context for a task. That common denominator is the point. brain has checkpoint and resume as primitives; a store with only add() and search() must flatten a checkpoint into prose. The asymmetry is not a handicap imposed by the harness, it is the finding.
Index ¶
- func Composition(suite []Scenario) string
- func ErrNoSourceOfTruth(err error) bool
- func Report(results []Result, verbose bool) string
- func Tokens(s string) int
- type Adapter
- type Aggregate
- type Durable
- type Event
- type Fact
- type Gold
- type Kind
- type Known
- type Options
- type Query
- type Response
- type Result
- type Scenario
- type Score
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Composition ¶
Composition summarises what the suite contains, so a reader can see the balance of strengths to weaknesses before reading any score.
func ErrNoSourceOfTruth ¶
ErrNoSourceOfTruth reports whether a score failed because the system had nothing left after its cache was dropped.
func Report ¶
Report renders the comparison.
Fidelity leads because it is the number that cannot be gamed by returning more: it rises only when the required facts arrive and the wrong ones stay out. Density sits beside it for the opposite reason — a system can buy recall with tokens, and the column shows the price.
func Tokens ¶
Tokens estimates cost at four characters per token.
A real tokenizer would be a model-specific dependency answering a question that only needs to be roughly right, and — more to the point — every adapter is measured with this same function. Consistency across systems matters more here than absolute accuracy against any one vocabulary.
Types ¶
type Adapter ¶
type Adapter interface {
Name() string
Reset() error
Write(Event) error
Read(Query) (Response, error)
Close() error
}
An Adapter is a memory system under test. Reset must leave it as empty as a fresh install: the suite runs every scenario against a clean store, so a system cannot score by accumulating context across cases.
type Aggregate ¶
type Aggregate struct {
Adapter string
Group string
N int
Recall float64
Leakage float64
Signal float64
Fidelity float64
Density float64
// PassRate is the share of scenarios that met every bar they set. It is the
// honest summary for families where fidelity does not apply — an abstention
// case has no facts to carry, so it scores full recall by definition and
// only the signal says whether the system did the right thing.
PassRate float64
Tokens int // mean
OverRuns int // responses that exceeded their budget
Errors int
}
An Aggregate rolls scores up for one adapter over one slice of the suite.
type Durable ¶
type Durable interface {
DropDerived() error
}
Durable is implemented by systems that distinguish a source of truth from a rebuildable cache. DropDerived deletes everything the system considers disposable; whatever survives is what the system really owns.
Not implementing this is itself an answer, and the suite reports it as one: a store whose only copy lives in its own index has no source of truth, and scores zero on the durability family rather than being excused from it.
type Event ¶
type Event struct {
TS int64 // unix seconds; scenarios use real offsets so staleness is testable
Actor string // "user", "claude", "cursor" — who produced this
Kind Kind
Project string
Title string // for KindDoc
Text string
// Checkpoint fields. Adapters with a checkpoint primitive should map these
// onto it; the rest get them via Flatten.
Task string
Decisions []string
Failed []string // the approaches already ruled out
Questions []string
Next string
}
An Event is one thing that happened, in the order it happened. Scenarios are built from these, and every adapter receives exactly the same sequence.
func (Event) Flatten ¶
Flatten renders an event as the plain prose a store without structured primitives would have to swallow. It is deliberately generous — everything is present, nothing is abbreviated — so that a system scoring badly on this suite cannot blame the harness for withholding information.
type Fact ¶
type Fact struct {
Label string // what to call this in the report
Any []string // at least one must appear
All []string // every one must appear (for facts that are only right together)
}
A Fact is something the harness looks for in a response. Matching is substring, case-insensitive, over whitespace-collapsed text.
Surface variants are the scenario author's job, not the matcher's: write Any: {"71 percent", "71%"} rather than hoping a cleverer matcher guesses. A fuzzy matcher that silently accepts near-misses is how a benchmark starts flattering everyone equally.
type Gold ¶
type Gold struct {
// Carry must appear. These are the facts the arriving agent cannot do the
// job without.
Carry []Fact
// Avoid must NOT appear — a superseded price, a cancelled plan, a value the
// user corrected. Carrying a stale fact forward is worse than carrying
// nothing, because the agent will act on it.
Avoid []Fact
// Signal is meta-knowledge about the answer rather than the answer: that the
// information is old, that the store does not know, that a finding came from
// a particular agent. Scored separately because almost every system alive
// scores zero here, and folding that into the main number would say more
// about the state of the art than about any one system.
Signal []Fact
}
Gold is what a correct answer looks like.
type Kind ¶
type Kind string
Kind classifies an event, because systems with richer primitives are allowed to use them. An adapter that has no notion of a checkpoint is expected to flatten one into ordinary text — see Event.Flatten.
const ( // KindMessage is a conversational turn. KindMessage Kind = "message" // KindFact is a durable statement about the user, the sort a memory tool // stores verbatim. KindFact Kind = "fact" // KindDoc is a document that exists in the world — a spec, a spreadsheet // export, a project page. KindDoc Kind = "doc" // KindNote is a line of working progress, written while the work happens. KindNote Kind = "note" // KindCheckpoint is where an agent stopped: state, decisions, what failed, // what is next. KindCheckpoint Kind = "checkpoint" )
type Known ¶
type Known string
Known is what brain is expected to do on a case today. Recording the prediction in the suite is what turns a weakness into a tracked one: a "weakness" that starts passing is progress worth noticing, and a "strength" that starts failing is a regression the aggregate numbers would otherwise average away.
type Options ¶
type Options struct {
// Only runs the scenarios whose id, family or skill contains this string.
Only string
// Progress is called before each scenario.
Progress func(adapter, scenario string, done, total int)
// Trace, if set, receives the raw response alongside its score. The suite is
// only as good as its gold labels, and the only way to tell a real pass from
// a substring that happened to appear is to read what the system actually
// returned.
Trace func(sc Scenario, resp Response, score Score)
}
Options controls one benchmark run.
type Query ¶
type Query struct {
Task string
Project string
Agent string // who is asking now — for handoff cases, not whoever wrote the events
Budget int
Now int64 // the clock at read time, so staleness has a reference point
}
A Query is what the arriving agent asks for. Budget is a token ceiling: a system may return less, but returning more is measured and counted against it, because context that overflows the window is context nobody reads.
type Response ¶
type Response struct {
Text string
// Err is set when the system declined or failed to answer. A failed read is
// scored as an empty response rather than aborting the run — a memory layer
// that errors on an unfamiliar question is one an agent learns to avoid, and
// that should show up in the numbers rather than in a stack trace.
Err error
}
A Response is what the system would put into the arriving agent's context window. Text is scored; Tokens is measured by the harness, not by the adapter, so nobody can under-report their own cost.
type Scenario ¶
type Scenario struct {
ID string
Family string // continuity | memory | durability
Skill string // the specific capability under test
Why string // one line, for the report — what this case is really asking
Known Known // brain's expected outcome, so regressions are visible
Setup []Event
Query Query
Gold Gold
// DropDerived wipes every rebuildable artifact between writing and reading.
// Systems that keep no source of truth outside their own index return
// nothing after this, which is the intended result.
DropDerived bool
}
A Scenario is one case: a history, a task, and what a correct answer carries.
type Score ¶
type Score struct {
Scenario string
Family string
Skill string
Known Known
Adapter string
CarryHit, CarryTotal int
LeakHit, LeakTotal int
SignalHit, SignalTotal int
Tokens int
Budget int
Missed []string
Leaked []string
Unsignalled []string
Err error
}
A Score is one adapter's result on one scenario.
func Run ¶
Run puts one adapter through a suite and returns a score per scenario.
Every scenario starts from Reset, so no system can score by carrying state between cases — a store that answered the last question well should get no help on the next one. Setup events are delivered in timestamp order, exactly as written, to every adapter alike.
func (Score) Density ¶
Density is required facts carried per 1000 tokens spent. The axis on which "put the whole history in the window" loses: it scores well on recall by brute force and terribly here, and the window it burns is the window the actual conversation needed.
func (Score) Fidelity ¶
Fidelity is the headline: carrying what is needed while not carrying what is wrong. Multiplicative rather than averaged, because a response that carries every required fact alongside a superseded price is not "half right" — the agent will act on the stale number, and the correct facts sitting next to it do not undo that.
func (Score) Pass ¶
Pass is whether the scenario was satisfied on every axis it declares.
Fidelity alone is not enough, and the staleness case is why: its gold labels ask for the note (carried) and for its age (a signal). Judged on fidelity it scored 100% while doing precisely the thing the case was written to catch. A scenario passes when it meets every bar it set, not the most flattering one.
func (Score) Recall ¶
Recall is the share of required facts that survived.
A scenario that requires nothing scores 1, not 0. The abstention cases have no Carry labels by design — the correct answer is an admission of ignorance, and there is no fact to fetch. Dividing zero by zero and calling it a miss marked every system down for declining to invent one.
Source Files
Directories