Documentation
¶
Overview ¶
Package engine scores code-search tools against each other: corpora and their pinned revisions, labelled query sets, the expected span for each query, the two-granularity metric, and the subprocess adapter that drives a tool from a JSON description.
Every tool reaches the scoring function through one interface and is graded by one loop, so the only thing that differs between two rows is the system that produced the ranking. What a tool cannot do is declared in its Capabilities and reported on the row; nothing is inferred on its behalf.
No code here branches on any tool's identity, and none should: the moment a harness knows which engine it is scoring, its rows stop meaning what they say.
Index ¶
- Constants
- Variables
- func Answers(h Hit, g Gold) bool
- func DefaultBenchDir(root string) string
- type Capabilities
- type Command
- type Corpus
- type Cost
- type Diagnoser
- type Difficulty
- type Engine
- type ExecEngine
- type ExecSpec
- type Gold
- type Hit
- type IndexStats
- type Metrics
- type Miss
- type ParseSpec
- type Query
- type Result
- type SetupStatus
- type Span
- type Spec
- type Tier
Constants ¶
const CorporaEnv = "BENCH_CORPORA"
CorporaEnv is the environment variable read when -corpora-root is not given.
Deliberately not named for any tool that this benchmark scores. A reader with one competitor's variable already exported would otherwise get a corpora root the command line does not mention.
Variables ¶
var Ks = []int{5, 10, 20}
Ks are the recall cutoffs every run reports. 20 is the coverage cutoff (G1): a target missing from the top 20 was destroyed or diluted by chunking, not merely mis-ranked.
var PerFile = []int{1, 2, 3}
PerFile are the chunks-per-file settings the harness scores. 1 reproduces the protocol's metric and exists to prove the wide path has not moved it.
Functions ¶
func DefaultBenchDir ¶
DefaultBenchDir is where indexes and the embed cache live under a corpora root. It mirrors the committed spec's own layout (a `.bench` sibling of the checkouts), so -corpora-root alone is enough to relocate a whole run.
Types ¶
type Capabilities ¶
type Capabilities struct {
// Spans reports whether hits carry line information. False means chunk
// granularity is not measurable for this engine and the chunk columns are
// reported as file columns, flagged.
Spans bool
// PerFile lists the chunks-per-file settings the engine can serve. Nil
// means only 1. Settings outside this list are not scored, so the wide
// table shows a competitor's real curve rather than a flat line implying
// it answered a question it was never asked.
PerFile []int
// Tokens reports whether hits carry the text they would show. False means
// the token column is omitted rather than reported as zero.
Tokens bool
// TokensFromCorpus marks a token figure the harness derived by reading the
// engine's reported spans out of the corpus, because the engine returns
// coordinates but not text. The number is real, but it assumes a caller
// receives the span the engine points at — so it is labelled everywhere it
// is printed rather than passing as the engine's own accounting.
TokensFromCorpus bool
}
Capabilities is what an engine can be asked for. It exists so a comparison degrades *visibly* rather than silently: an engine that cannot report line spans is graded at file granularity and says so on the row, instead of scoring a quiet zero at chunk granularity and looking beaten.
func (Capabilities) Supports ¶
func (c Capabilities) Supports(n int) bool
Supports reports whether the engine can serve n chunks per file.
type Command ¶
type Command struct {
Command []string `json:"command"`
Dir string `json:"dir,omitempty"`
Timeout string `json:"timeout,omitempty"`
// Stdin, if set, is written to the process (templated). Some tools take the
// query on stdin rather than argv, which avoids every quoting question.
Stdin string `json:"stdin,omitempty"`
}
Command is one invocation: argv (templated), an optional working directory, and a timeout.
type Corpus ¶
type Corpus struct {
Name string `json:"name"`
Repo string `json:"repo"` // clone URL
Rev string `json:"rev"` // pinned tag
Commit string `json:"commit"` // pinned commit the tag resolved to
Lang string `json:"lang"` // the language this corpus grades
Path string `json:"path"` // local clone, absolute or relative to the spec file
Queries string `json:"queries"` // query-set file, relative to the spec file
Subdirs []string `json:"subdirs,omitempty"` // if set, index only these subtrees
QuerySet []Query `json:"-"`
}
Corpus is one evaluation repository: where it is, what revision it is pinned to, and the labeled queries that grade it.
func (*Corpus) IndexRoot ¶
IndexRoot is the directory an external engine should be pointed at, plus the prefix its results need in order to land back in the corpus-relative frame gold uses.
It exists because the two sides of a comparison were not being shown the same tree. A corpus may declare only part of its checkout as the code under test; an engine handed the whole thing instead ranks the expected answer against far more candidates. On redis that is 593 files against 1610, including trees the query set never points at. A difference that large is not a detail of the integration, it is the result: it makes the row a statement about who was given the bigger haystack rather than about retrieval.
Exact is false when a corpus names several subtrees, because no single directory expresses that and a one-directory tool cannot be handed it. The caller reports that rather than quietly comparing unequal trees.
func (*Corpus) Strata ¶
func (c *Corpus) Strata() map[Difficulty]int
Strata counts the query set by difficulty, so a run can report whether the set still meets the roughly one-third split the protocol calls for.
func (*Corpus) Validate ¶
Validate checks a query set against the corpus on disk: every referenced path must exist at the pinned revision, and every span must lie inside its file. A query set that points at files a refactor moved silently deflates recall, so the harness refuses to score one it cannot verify.
type Cost ¶
type Cost struct {
IndexSeconds float64 `json:"index_seconds"`
FilesPerSec float64 `json:"files_per_sec"`
ChunksPerSec float64 `json:"chunks_per_sec"`
PeakRSSMB float64 `json:"peak_rss_mb"`
QuerySeconds float64 `json:"query_seconds"` // mean latency per query
}
Cost is the G6 accounting for one corpus.
type Diagnoser ¶
Diagnoser is the optional root-cause half of the protocol: an engine that can say what its index actually holds for an expected span it missed.
It is an interface assertion rather than a method on Engine because a tool driven as a subprocess cannot answer the question — it has no way to look inside itself from out here. An engine embedded in a program that *does* have that access can implement it, and Score will use it; anything else reports the miss without a cause, which is honest rather than absent.
type Difficulty ¶
type Difficulty string
Difficulty labels how much lexical help a query gives the retriever.
- easy: names the target symbol or a distinctive identifier from it.
- medium: describes the behavior in domain words that partly overlap the target's vocabulary.
- hard: describes intent or effect with no shared identifier at all, so only the semantic ranker can find it.
The split exists to keep the set discriminative: a query set that scores near 100% everywhere cannot tell two rankings apart.
const ( Easy Difficulty = "easy" Medium Difficulty = "medium" Hard Difficulty = "hard" )
type Engine ¶
type Engine interface {
// Name identifies the engine on the scoreboard row.
Name() string
// Caps declares what the engine can be asked for.
Caps() Capabilities
// Prepare indexes a corpus and returns what that cost. It is called once
// per corpus, before any Search.
Prepare(c Corpus) (IndexStats, error)
// Search returns up to topK ranked slots, one per file, each carrying up to
// perFile spans, best first.
Search(query string, topK, perFile int) ([]Hit, error)
// Close releases whatever Prepare acquired.
Close() error
}
Engine is one retrieval system under test.
The interface is the whole point of the comparison: every tool is graded by the *same* loop over the *same* expected spans, so the only thing that differs between two rows is the system that produced the ranking. Anything an engine cannot do is declared in Capabilities and shows up on the row; nothing is inferred or filled in on its behalf.
type ExecEngine ¶
type ExecEngine struct {
Spec *ExecSpec
Dir string // where per-corpus state directories are created
Log func(string, ...any)
// Reindex empties the state directory before indexing, so the tool builds
// from nothing.
//
// An incremental indexer handed a surviving state directory reports the cost
// of noticing it has nothing to do. That is not a slow-path bug, it is a
// silent one — one tool measured here re-enables a warm index in 0.1s against
// 22s cold, and the harness cannot tell those apart from the outside. A cost
// comparison where one side is cold and the other warm is worse than none,
// because it still prints a number.
Reindex bool
// contains filtered or unexported fields
}
ExecEngine runs a third-party tool as a subprocess.
func NewExecEngine ¶
NewExecEngine prepares an engine from a spec. dir is where the harness gives each corpus a scratch directory, handed to the tool as {{.State}}.
func (*ExecEngine) Caps ¶
func (e *ExecEngine) Caps() Capabilities
func (*ExecEngine) Close ¶
func (e *ExecEngine) Close() error
func (*ExecEngine) Name ¶
func (e *ExecEngine) Name() string
func (*ExecEngine) Prepare ¶
func (e *ExecEngine) Prepare(c Corpus) (IndexStats, error)
Prepare runs the tool's index command over the corpus and times it.
The index is built into a scratch directory the harness owns, not into the corpus checkout, so a run leaves the pinned corpora byte-identical for the next engine. Whatever the tool writes there is its own business.
type ExecSpec ¶
type ExecSpec struct {
Name string `json:"name"`
// Mode is "oneshot" (run the search command per query, the default) or
// "persistent" (start it once and speak JSON lines over stdin/stdout).
//
// The choice is a measurement decision, not a convenience: a oneshot engine
// pays process startup on every query, and that can be larger than anything
// about retrieval itself. Persistent mode is the fairer comparison where a
// tool supports it; where it does not, the cost is the integration's and the
// latency column should be read as such.
Mode string `json:"mode,omitempty"`
Index Command `json:"index"`
Search Command `json:"search"`
// Parse says how to read a search command's output.
Parse ParseSpec `json:"parse"`
// Candidates scales the result depth requested from the tool. A tool that
// ranks chunks rather than files may need many results to yield K distinct
// files, so this is how it asks for depth. It only widens what is requested —
// the harness still grades the top K files.
Candidates int `json:"candidates,omitempty"`
// Spans, PerFile and Tokens declare what the tool can do; they become the
// engine's Capabilities. Declaring Spans false is not a penalty — the row
// is graded and reported at file granularity instead of scoring zero at a
// granularity the tool never claimed to serve.
Spans bool `json:"spans"`
PerFile []int `json:"per_file,omitempty"`
Tokens bool `json:"tokens,omitempty"`
// TextFromCorpus prices a tool that reports line numbers but does not print
// the text at them: the harness reads those lines out of the corpus itself.
//
// This is dereferencing the tool's own answer, not inventing one — the
// coordinates are the tool's and the bytes at them are fixed. It exists
// because the alternative is worse in both directions: a tool whose CLI
// prints a truncated preview either gets no token column at all, and its
// recall is read with no idea what was spent to buy it, or the preview is
// priced and its context cost reads as near zero.
//
// It rests on an assumption the row must carry: that a caller consuming this
// tool receives the span it points at. Where that is false the number is
// wrong, so Result.TokensFromCorpus marks every row derived this way and no
// such row should be published without saying so.
TextFromCorpus bool `json:"text_from_corpus,omitempty"`
// StripPrefix is removed from every returned path before matching. Gold
// paths are corpus-relative; a tool that echoes absolute paths needs the
// checkout root taken off, and one that prefixes a collection name needs
// that taken off. Absolute paths under {{.Root}} are relativized
// automatically, so this is for anything stranger.
StripPrefix string `json:"strip_prefix,omitempty"`
// LineBase is the first line number the tool uses. Set 0 for a tool that
// counts lines from zero; the harness converts to the 1-based convention
// the gold spans use. An off-by-one here silently halves span recall, so it
// is explicit rather than guessed.
LineBase *int `json:"line_base,omitempty"`
// Env is added to the tool's environment, as "KEY=value" entries. Templated
// like the commands, so a tool configured by environment rather than by flag
// can still be pointed at {{.State}} — which is how a tool that would
// otherwise write its index into the corpus is kept out of it.
Env []string `json:"env,omitempty"`
}
ExecSpec describes how to drive a retrieval tool from the command line, so a binary this project has never seen can be scored by the same protocol as every other.
It is a *description*, not code, because the thing being compared is the tool's retrieval quality and nothing else. Everything the harness needs from an engine — index this directory, answer this query, here is where the answer lives — is a command template plus a way to read the output. A tool whose interface does not fit is wrapped in a shell script that does; the harness asks only that the script speak one of the formats below.
Templates are expanded with these fields:
{{.Root}} the corpus checkout on disk (absolute)
{{.Name}} the corpus name ("ripgrep")
{{.Lang}} the corpus language
{{.State}} a per-corpus scratch directory the harness owns
{{.Query}} the query text (search only)
{{.K}} how many files are wanted (search only)
{{.N}} K × Candidates, the raw result depth to ask for
{{.PerFile}} spans wanted per file (search only)
func LoadExecSpec ¶
LoadExecSpec reads an engine description.
type Gold ¶
type Gold struct {
Path string `json:"path"`
Symbol string `json:"symbol,omitempty"`
Start int `json:"start,omitempty"`
End int `json:"end,omitempty"`
}
Gold is one expected result. Path alone grades file granularity; Start/End (1-based, inclusive) additionally grade chunk granularity — a retrieved chunk counts only if it actually overlaps the lines that answer the query, which is what makes the chunker, not the ranker, accountable for the chunk numbers.
type Hit ¶
Hit is one slot in a ranked result list: a file, and the spans of it the engine chose to show. One slot per file is the protocol's unit of grading and protocol's unit of grading — "was the right file returned, and did the piece that came back with it contain the answer?" — so an engine that ranks raw chunks must collapse them per file before answering.
type IndexStats ¶
type IndexStats struct {
Files int
Chunks int // 0 when the engine does not expose a chunk count
Cost Cost
// Tree is the subtree the engine indexed, relative to the corpus checkout
// ("" or "." meaning the whole thing), and Unequal marks that it could not
// be given the same one as the other engine. Both travel onto the Result:
// which haystack each side searched is part of what a row means.
Tree string
Unequal bool
}
IndexStats is what preparing a corpus cost and produced.
type Metrics ¶
type Metrics struct {
Queries int `json:"queries"`
Recall map[int]float64 `json:"recall"` // keyed by cutoff
MRR float64 `json:"mrr"`
ZeroMiss float64 `json:"zero_miss"`
}
Metrics is one scoreboard row's worth of retrieval quality.
type Miss ¶
type Miss struct {
Query string `json:"query"`
Difficulty Difficulty `json:"difficulty"`
Gold []Gold `json:"gold"`
FileRank int `json:"file_rank"` // rank of the gold file, 0 if absent
ChunkRank int `json:"chunk_rank"` // rank of an overlapping chunk, 0 if absent
// NearestChunk describes the closest chunk the index holds for the gold
// span, so a miss can be attributed to a boundary rather than guessed at.
NearestChunk string `json:"nearest_chunk,omitempty"`
}
Miss records a query whose gold set never appeared in the top 20 at chunk granularity — a G1 failure, the kind the protocol requires be root-caused to a specific chunk before the next iteration starts.
type ParseSpec ¶
type ParseSpec struct {
Format string `json:"format"`
Results string `json:"results,omitempty"`
Path string `json:"path"`
Start string `json:"start,omitempty"`
End string `json:"end,omitempty"`
Text string `json:"text,omitempty"`
Score string `json:"score,omitempty"`
// Regex is used when Format is "regex". Named groups path, start, end,
// text and score are read; the rest are ignored.
Regex string `json:"regex,omitempty"`
}
ParseSpec says how to turn a search command's output into ranked results.
Format is one of:
"json" one JSON document; Results is a dotted path to the array "jsonl" one JSON object per line, in rank order "regex" one match per line, with named groups
For the JSON formats the field names are dotted paths into each result object, so nested shapes ({"chunk": {"file": ...}}) need no wrapper script.
type Query ¶
type Query struct {
Query string `json:"query"`
Difficulty Difficulty `json:"difficulty"`
Subsystem string `json:"subsystem"`
Relevant []Gold `json:"relevant"`
}
Query is one labeled evaluation query.
func LoadQueries ¶
LoadQueries reads one labeled query set.
type Result ¶
type Result struct {
// Engine names the retrieval system that produced this row. Every result
// file and every scoreboard line carries it, because a number without the
// engine beside it is the one mistake a comparison harness cannot recover
// from.
Engine string `json:"engine"`
Corpus string `json:"corpus"`
Lang string `json:"lang"`
Tier Tier `json:"tier"`
Files int `json:"files"`
Chunks int `json:"chunks"`
// SpanlessHits counts results an engine returned with no line information
// despite declaring that it reports spans. Such a result can never satisfy
// a spanned gold entry, so counting them keeps a parsing bug in an adapter
// from being read as a quality deficit in the tool it wraps.
SpanlessHits int `json:"spanless_hits,omitempty"`
// TokensFromCorpus marks WideTokens as priced from spans the harness read
// out of the corpus rather than text the engine returned. See
// ExecSpec.TextFromCorpus: the coordinates are the engine's, the assumption
// that a caller receives them is the harness's, and a row must say so.
TokensFromCorpus bool `json:"tokens_from_corpus,omitempty"`
// Slots records how many distinct files the engine actually filled, per
// query, after the one-slot-per-file collapse. A tool asked for too few raw
// results yields fewer than the cutoff and loses recall to the adapter
// rather than to its retrieval — SlotsMin below the deepest cutoff is that
// bug, and it is otherwise invisible.
// SlotsRecorded distinguishes "no queries were run" from "every query
// returned nothing" — omitempty would otherwise make a legitimate zero
// minimum indistinguishable from a result file that predates the field.
SlotsMin int `json:"slots_min"`
SlotsMean float64 `json:"slots_mean"`
SlotsRecorded bool `json:"slots_recorded,omitempty"`
// IndexedTree is the tree this engine was actually pointed at, relative to
// the corpus checkout ("." for the whole thing). UnequalTrees marks a row
// where the two engines could not be given the same one.
//
// These are on the row because their absence is what let a real confound
// survive: one engine was pointed at a corpus's declared subtree while another
// got the whole checkout, so on redis one ranked the answer against 1610 files
// and the other against 593 — and *nothing in any output said so*. A
// comparison that cannot state which haystack each side searched is not
// reporting the thing that decides it.
IndexedTree string `json:"indexed_tree,omitempty"`
UnequalTrees bool `json:"unequal_trees,omitempty"`
// FileGranularityOnly marks a row whose engine cannot report line spans.
// The Chunk metrics then repeat the File ones rather than reading zero, and
// the row is flagged wherever it is printed: an engine that answers a
// coarser question has not lost the chunk comparison, it did not enter it.
FileGranularityOnly bool `json:"file_granularity_only,omitempty"`
File Metrics `json:"file"`
Chunk Metrics `json:"chunk"`
ByDiff map[Difficulty]Metrics `json:"by_difficulty"` // chunk granularity
Misses []Miss `json:"misses"`
Cost Cost `json:"cost"`
// Wide holds chunk-granularity metrics when a file carries more than one
// chunk, keyed by how many (see PerFile). It is deliberately a *separate*
// field rather than a replacement for Chunk: the protocol's metric is one
// chunk per file, because that is what a caller receives today, and
// iteration 0's harness correction is on the scoreboard because a metric
// was changed underneath a comparison. Wide[1] is reported alongside Chunk
// as a consistency check — the two must agree.
Wide map[int]Metrics `json:"wide,omitempty"`
// WideTokens is the mean estimated token count of everything a query's
// top-10 results carry, keyed the same way. Widening is not free: it buys
// recall with the caller's context budget, and a recall number without this
// one beside it is only half the trade.
WideTokens map[int]float64 `json:"wide_tokens,omitempty"`
}
Result is a full evaluation of one corpus: metrics at both granularities, the per-difficulty breakdown, and the queries that missed.
func Score ¶
Score runs the protocol against one engine on one corpus.
This is the function the whole comparison rests on. Every engine reaches it through the same interface, is asked the same queries in the same order at the same cutoffs, and is graded by the same two functions against the same expected spans. Everything an engine cannot do is declared in its Capabilities and reported on the row; nothing is inferred on its behalf, and no engine gets a scoring path of its own.
type SetupStatus ¶
type SetupStatus struct {
Corpus string
Path string
State string // "ok", "cloned", "checked out", or an error description
Err error
}
SetupStatus is what Setup did, or would have to do, for one corpus.
func Setup ¶
func Setup(corpora []Corpus, log func(string, ...any)) []SetupStatus
Setup makes the pinned corpora present on disk at the exact commits the spec names, cloning what is missing and checking out what is on the wrong revision.
The corpora are ~5 GB and are deliberately not in the repository, so without this the evaluation harness cannot be reproduced on a fresh machine at all — every number in docs/scoreboard.md would be unverifiable by anyone who did not happen to have the same clones lying around. Pinning to the resolved commit rather than the tag is what makes a re-clone comparable: tags move.
type Span ¶
Span is a stretch of lines an engine chose to put in front of the reader, 1-based and inclusive — the same convention as Gold, because the two are compared directly.
Text is what the engine would actually show for that span. It is optional (an engine that returns only coordinates leaves it empty) and is used for one thing: pricing a result set in tokens. A recall number without the context cost beside it describes half a trade.
type Spec ¶
type Spec struct {
Corpora []Corpus `json:"corpora"`
}
Spec is the corpus list: the whole evaluation protocol in one file, so a run is reproducible from a single path.
func LoadSpec ¶
LoadSpec reads a corpus spec and every query set it references, resolving relative paths against the spec file's directory.
func (*Spec) Rebase ¶
Rebase moves every corpus under root, keeping the directory name the spec gave it.
The committed spec pins each corpus to an absolute path, because a run has to be reproducible from one file and a relative path would depend on where the harness was invoked. That makes the spec machine-specific — the checked-in paths are one contributor's Linux home — and the corpora are ~5 GB of pinned clones nobody should have to re-path by hand to score a row on a different machine. Rebasing is the override: the *layout* the spec describes is what matters, and it is the same layout wherever the parent directory sits.
Only Path moves. Repo, Rev and Commit are the protocol and are untouched, so a rebased run still grades the same trees at the same commits — `bench setup` clones into the new root and `bench validate` gates it exactly as before.
Source Files