codesearch-bench

README

codesearch-bench

Scores code-search tools against each other on pinned open-source repositories, using one protocol and one scoring function.

Every tool is described by a JSON file and driven as a subprocess. There is no built-in engine and no privileged one — a tool measured by different code than its competitor is not being compared with it.

go build ./cmd/bench          # writes ./bench

./bench setup    -corpora-root ~/corpora     # clone the corpora — full history, ~10 GB, tens of minutes
./bench validate -corpora-root ~/corpora     # every expected path and span still resolves
./bench probe    -engine protocol/engines/grep.json -corpus ripgrep -corpora-root ~/corpora
./bench compare  -engine protocol/engines/grep.json \
                 -engine protocol/engines/yours.local.json \
                 -corpus ripgrep,leveldb -corpora-root ~/corpora -reindex

No dependencies beyond the Go standard library. Run from the repository root: -spec and the engine specs default to paths relative to it.

Bring your own tool. grep.json is the only spec shipped, and it works out of the box because you already have grep. Writing a spec for the tool you actually want to score is the normal case, not an advanced one.

Anything matching protocol/engines/*.local.json is ignored by git. Keep your specs there: a spec names a binary, its flags and sometimes its quirks, and those are not always yours to publish — a tool shared with you privately certainly is not. It is also why no spec for any particular product is committed here. A benchmark that ships one tool's adapter and not another's has taken a side before the first query runs.

Budget. setup clones full history (a pinned commit cannot be checked out of a shallow clone), so the seven corpora are ~10 GB, kubernetes most of it. Scoring runs the query set once for the headline metric and once more for each per_file setting a tool supports — up to four passes per tool per corpus.

The protocol here is versioned by this repository and nothing else. kubernetes declares plugin/ alongside pkg/ and staging/, because a query asking where a user's permission to act is decided is answered by the RBAC authorizer that lives there — a corpus that excludes it makes that query unanswerable. Numbers produced against a differently-scoped corpus of the same name are not comparable to numbers produced here, whoever produced them.

Run probe first, always

Every cross-tool comparison fails the same way the first time: the adapter mis-reads a path or a line base, the tool scores near zero, and the number gets read as a verdict on the tool. probe prints what the adapter made of one query's output next to the expected answer, before an hour of indexing.

The first probe of a corpus has to index it, so it is quick only from the second run on. probe keeps its state under <dir>/probe, separate from compare, so that the step you are told to run first cannot warm the index compare then measures.

Two things on a compare row mean fix the adapter, not the tool is bad, and both are printed with !!:

• results arriving with no line spans — the span fields are wrong;
• fewer file slots filled than the cutoff — the tool was asked for too few raw results and is losing recall to the request depth, not to its retrieval.

What is scored

Two questions, from a single retrieval:

• file recall — was the right file returned?
• chunk recall — did the piece returned with that file contain the answer?

Chunk recall is at most file recall by construction. The gap between them is where a tool finds the right file and shows you the wrong part of it.

Each query names a symbol; the expected answer is the line span that symbol's own grammar assigns it, so no tool is graded against its own boundaries. Recall is reported at cutoffs 5/10/20 with MRR and zero-miss, and every tool is collapsed to one slot per file in first-appearance order before grading.

A chunk hit is an overlap. A tool showing 100 lines around the answer scores the same hit as one showing the 8 lines that are the answer, and hits more often — so read chunk recall next to the token column, never alone.

Writing an engine spec

grep.json and the script beside it are the worked example: the floor any real tool must clear, and the smallest thing that can be driven. Read them together — the script's four arguments are the four template fields the spec passes it. The shape below covers what the rest of a real spec needs.

{
  "name": "acme",
  "spans": true,
  "per_file": [1, 2, 3],
  "candidates": 3,
  "env":    ["ACME_STATE={{.State}}"],
  "index":  { "command": ["acme", "index", "{{.Root}}"] },
  "search": { "command": ["acme", "query", "-n", "{{.N}}", "{{.Query}}"] },
  "parse":  { "format": "json", "results": "hits",
              "path": "file", "start": "from", "end": "to", "text": "body" }
}

Templates: {{.Root}} (the tree to index), {{.State}} (a scratch directory the harness owns — point your tool's index at it), {{.Name}}, {{.Lang}}, and in search also {{.Query}}, {{.K}}, {{.N}}, {{.PerFile}}.

Field	Meaning
spans	Results carry line numbers. False ⇒ graded at file granularity, and the row says so
per_file	Chunks-per-file settings the tool can serve. Anything unlisted is not asked for
tokens	Results carry the text they would show, so context cost can be priced
text_from_corpus	The tool reports line numbers but no text: price it by reading those lines. Marked * on the row
candidates	{{.N}} = {{.K}} × this. A tool ranking chunks needs depth to yield K distinct files
strip_prefix, line_base	Path prefix to drop; set line_base: 0 for a tool counting from zero
mode	persistent keeps the tool alive and speaks JSON lines, instead of a process per query — see below
name	Names the tool on every row, and keys its state directory. Two engines may not share one
index / search	{"command": [...], "timeout": "2h", "dir": "...", "stdin": "..."}. Default timeouts are 6h for index and 2m for search; both shipped specs override them
env	KEY=value entries, templated. The way to point a tool configured by environment at {{.State}}

text_from_corpus does nothing unless spans is true, and candidates defaults to 1 when absent.

parse.format is json, jsonl, or regex. Field names are dotted paths, so nested shapes need no wrapper.

When the tool has no machine-readable output

Use regex, with named groups path, start, end, and optionally text and score. It is applied per line; lines that do not match are skipped in silence, so a wrong pattern yields zero results rather than an error — which is exactly why probe exists.

Anchor on something only a result line has. Tools that print a block per hit — a header line followed by continuation or preview lines — will otherwise match the continuations too, and those parse into plausible-looking spans that are not results. Matching ^(?P<path>\S+):(?P<start>\d+)-(?P<end>\d+) alone is rarely enough; requiring a distinctive suffix that only the header carries is.

Pair it with text_from_corpus when the tool prints a truncated preview rather than the text it would return: the harness then prices the lines the tool points at, and marks the figure * because that assumes a caller receives the whole span.

Persistent mode

"mode": "persistent" starts the search command once and speaks JSON lines over stdin/stdout: one request object per line, one response line back, parsed by the same parse spec. Each request carries query, k, n, per_file, root, corpus, and state. Worth implementing if you care about the latency column — a tool run per query pays process startup on every one, and none of the shipped examples uses this mode.

Declared, not assumed

Anything a tool cannot do is declared in its spec and reported as not entered:

• no line spans ⇒ graded at file granularity, flagged — not a zero at a granularity it never claimed;
• no result text and no text_from_corpus ⇒ no token column, because a zero there reads as free;
• per_file settings it cannot serve ⇒ blank rows, not a flat line implying it answered a question nobody asked;
• no machine-readable file count ⇒ blank files/chunks and no size tier.

Comparing the same candidate set

A corpus may declare subtrees (redis/src), and every engine is pointed at the same one. This is not a detail: pointing one tool at redis/src (593 files) and another at the whole checkout (1,610) produces a difference in the results that looks like retrieval quality and is not. The tree each engine received is printed as tree= and stored as indexed_tree; where a corpus names several subtrees and a one-directory tool cannot be given them all, the row is flagged unequal_trees rather than compared silently.

Reading a comparison honestly

• Index cost is not comparable unless every engine rebuilt from nothing. -reindex clears each engine's state directory, but a tool with its own content-addressed cache elsewhere may still skip work.
• Query latency is a property of the integration. A tool run per query pays process startup; one in persistent mode does not.
• Peak RSS measures different things for a long-lived process and a per-index child.
• The corpora and the query sets came from somewhere. Whoever wrote them chose what to ask, and a tool developed against them has an advantage that no scoring function removes. Say so when publishing a row.
• A failed engine leaves a gap, not a zero. compare keeps going, reports which combinations produced no row, still writes -json, and exits non-zero.