bench

Documentation

Overview

Package bench measures the project's core bet — token efficiency — the same headline the upstream codebase-memory-mcp reports (10x fewer tokens, 2.1x fewer tool calls on structural questions; see docs/UPSTREAM.md).

For each structural question ("who calls X?") it compares three strategies:

graph         — one callers/callees query, returns compact refs (1 tool call)
baseline-win  — grep X, then read a ±window around each match (1+N calls)
baseline-file — grep X, then read each whole matched file (1+N calls)

The graph wins because it has already resolved the enclosing caller of every match and filtered out definitions/imports/homonyms — exactly the work grep dumps onto the agent. The reported number is a RATIO (baseline/graph), so the rough token estimate cancels out. We do NOT measure answer quality here: that needs an LLM-as-judge harness and would be a romantic number, not an engineered one. This file measures only what is deterministic.

Index

• func EstimateTokens(s string) int
• type Corpus
  • func LoadCorpus(root string) (*Corpus, error)
• type Cost
• type Outcome
  • func RunOne(eng *query.Engine, c *Corpus, q Question) (Outcome, error)
• type Question
  • func QuestionsFromHubs(hubs []graph.Node) []Question
• type Summary
  • func Summarize(outs []Outcome) Summary

Functions

func EstimateTokens

func EstimateTokens(s string) int

EstimateTokens approximates LLM token count as bytes/4 (the classic ~4 chars/token rule). The absolute value is rough, but the same function meters both sides of every comparison, so the RATIO — the only number we report — is stable under the approximation.

Types

type Corpus

type Corpus struct {
	// contains filtered or unexported fields
}

Corpus is the repo's source loaded once, reused across every question's grep.

func LoadCorpus

func LoadCorpus(root string) (*Corpus, error)

LoadCorpus reads every discoverable source file into memory (the same files the indexer sees), so the baseline's grep is apples-to-apples with the graph.

type Cost

type Cost struct {
	Tokens int
	Calls  int
}

Cost is tokens + tool-calls for one strategy answering one Question.

type Outcome

type Outcome struct {
	Question     Question
	Graph        Cost
	BaselineWin  Cost // grep + read ±window around each match
	BaselineFile Cost // grep + read whole file per match
	MatchFiles   int  // distinct files the grep baseline had to open
	GraphResults int  // nodes the graph returned
}

Outcome is the full three-way comparison for one Question.

func RunOne

func RunOne(eng *query.Engine, c *Corpus, q Question) (Outcome, error)

RunOne benchmarks a single question across all three strategies.

type Question

type Question struct {
	Kind string // "callers" | "callees"
	QN   string // qualified name to query
	Name string // short symbol name the grep baseline searches for
}

Question is one structural query to benchmark.

func QuestionsFromHubs

func QuestionsFromHubs(hubs []graph.Node) []Question

QuestionsFromHubs turns call hubs into "who calls X" questions.

type Summary

type Summary struct {
	N                  int
	MedianRatioWin     float64
	MedianRatioFile    float64
	TotalRatioWin      float64
	TotalRatioFile     float64
	CallRatioWin       float64 // baseline-win tool-calls / graph tool-calls (total)
	GraphTokens        int
	BaselineWinTokens  int
	BaselineFileTokens int
	GraphCalls         int
	BaselineWinCalls   int
	BaselineFileCalls  int
}

Summary aggregates outcomes. Two aggregate ratios are reported because they answer different questions: the MEDIAN per-query ratio (typical case, robust to one common-named outlier) and the TOTAL/TOTAL ratio (the "10x" headline: total tokens an agent spends across the whole question set).

func Summarize

func Summarize(outs []Outcome) Summary

Summarize folds outcomes into aggregate ratios.