membench

Documentation

Overview

Package membench evaluates memcode's session-memory retrieval against public conversational-memory benchmarks (LongMemEval, LoCoMo) with ZERO model calls.

Each benchmark question's chat history is ingested as real .memcode session logs (one events.jsonl per benchmark session, timestamps preserved), then retrieval adapters rank content for the question and are scored against the benchmark's own evidence labels (answer_session_ids / evidence dia_ids). Metrics: recall@k and nDCG@k. The point is to measure the retrieval layer exactly as the product exercises it, before any answer-mode run.

Index

• func CacheDir() string
• func GenerateFacts(ds *Dataset, limit int) error
• func Ingest(root string, docs []SessionDoc) error
• type Adapter
  • func Adapters(name string) []Adapter
• type BM25Adapter
  • func (a BM25Adapter) Name() string
  • func (a BM25Adapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)
• type Dataset
  • func Load(name, dataPath string) (*Dataset, error)
  • func SplitQuestions(ds *Dataset, split string) (*Dataset, error)
• type Fact
• type Granularity
• type HybridAdapter
  • func NewHybridAdapter() (*HybridAdapter, error)
  • func (h *HybridAdapter) Name() string
  • func (h *HybridAdapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)
• type LegacyAdapter
  • func (LegacyAdapter) Name() string
  • func (LegacyAdapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)
• type ProductAdapter
  • func (a ProductAdapter) Name() string
  • func (a ProductAdapter) Prepare()
  • func (ProductAdapter) Rank(root string, q Question, _ []SessionDoc, k int) ([]string, error)
• type QAResult
  • func RunQA(ds *Dataset, ad Adapter, workDir string, limit, topK int) (*QAResult, error)
  • func (r *QAResult) Accuracy() float64
  • func (r *QAResult) Render() string
• type Question
• type QuestionResult
• type RunResult
  • func Run(ds *Dataset, ad Adapter, workDir string, limit int, keep bool) (*RunResult, error)
  • func (r *RunResult) Render() string
  • func (r *RunResult) WriteLog(dir string) (string, error)
• type SessionDoc
• type Turn

Functions

func CacheDir

func CacheDir() string

CacheDir is where datasets and retrieval logs live; never committed.

func GenerateFacts

func GenerateFacts(ds *Dataset, limit int) error

GenerateFacts extracts facts for every session in the first `limit` questions' haystacks (0 = all) and attaches them to the SessionDocs in place. Identical sessions across overlapping haystacks generate once; results persist in a content-addressed cache so reruns are free.

func Ingest

func Ingest(root string, docs []SessionDoc) error

Ingest writes each benchmark session as a real .memcode session log: .memcode/sessions/<SessionDoc.ID>/events.jsonl via the production sessionlog.Writer, with Record.TS carrying the benchmark timestamps and the file mtime pinned to the session time (recency ordering in sessionlog is mtime-based). Turn identity rides Record.Slug — unused by real chat sessions and excluded from Search's match haystack, so it labels without polluting retrieval.

Types

type Adapter

type Adapter interface {
	Name() string
	Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)
}

Adapter ranks turns for a question over an ingested root and returns turn IDs, best first. The scorer aggregates turns to sessions when the dataset is session-granular.

func Adapters

func Adapters(name string) []Adapter

Adapters returns the named adapter set ("all" = every adapter).

type BM25Adapter

type BM25Adapter struct {
	// Time-aware scoring (the bm25+time adapter is this struct with TimeAware
	// set): explicit date cues in the question define a window that boosts
	// in-window turns and dampens out-of-window ones, and knowledge-update
	// questions get a recency tiebreak so the latest version of a changed
	// fact outranks the original.
	TimeAware bool
	// V2 layers the accuracy work on top: light stemming, bigram matching,
	// category-aware query expansion, and relative-date windows. Kept as a
	// flag so v1 numbers stay reproducible for the before/after table.
	V2 bool
}

BM25Adapter tokenizes the question and every ingested turn and ranks turns by BM25. Reads the turns back from the ingested session logs (not the in-memory dataset) so the whole file pipeline is exercised.

func (BM25Adapter) Name

func (a BM25Adapter) Name() string

func (BM25Adapter) Rank

func (a BM25Adapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)

type Dataset

type Dataset struct {
	Name      string
	Gran      Granularity
	Questions []Question
}

Dataset is a fully parsed benchmark.

func Load

func Load(name, dataPath string) (*Dataset, error)

Load fetches (or reads from cache / an explicit path) and parses a dataset. Supported names: "longmemeval-s", "locomo".

func SplitQuestions

func SplitQuestions(ds *Dataset, split string) (*Dataset, error)

SplitQuestions carves a tune/holdout split by question index parity so knob tuning ("even") never reads the reporting set ("odd"). "all"/"" is identity.

type Fact

type Fact struct {
	Fact     string   `json:"fact"`
	Entities []string `json:"entities"`
	Source   string   `json:"source,omitempty"` // benchmark turn id the fact is drawn from
}

Fact is one extracted atomic fact attached to a SessionDoc.

type Granularity

type Granularity int

Granularity of the ranked unit a dataset is scored at.

const (
	BySession Granularity = iota
	ByTurn
)

type HybridAdapter

type HybridAdapter struct {
	HybridWeight float64
	// contains filtered or unexported fields
}

HybridAdapter blends the v2 lexical score with embedding cosine similarity: score = bm25/bm25max + HybridWeight*cosine. Turns are read back from the ingested logs like every other adapter.

func NewHybridAdapter

func NewHybridAdapter() (*HybridAdapter, error)

NewHybridAdapter fails fast when no embedding key is configured.

func (*HybridAdapter) Name

func (h *HybridAdapter) Name() string

func (*HybridAdapter) Rank

func (h *HybridAdapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)

type LegacyAdapter

type LegacyAdapter struct{}

LegacyAdapter is the stub that stands in when the binary is built WITHOUT `-tags membench`: the real pre-2026-07-25 substring scan (legacy.go) is bench-only history and must not ship in release binaries. The type keeps the exported surface (Adapters, cmd/bench) compiling; running it says why it isn't available instead of silently benching nothing.

func (LegacyAdapter) Name

func (LegacyAdapter) Name() string

func (LegacyAdapter) Rank

func (LegacyAdapter) Rank(root string, q Question, docs []SessionDoc, k int) ([]string, error)

type ProductAdapter

type ProductAdapter struct {
	Variant string
	Opts    *sessionlog.RankingOptions
}

ProductAdapter runs the production session-search code path exactly as the memcode{session,search} tool runs it. Since the Phase B promotion this is the ranked search; the pre-promotion substring behavior lives on as LegacyAdapter for the before/after table. Opts installs a ranking-feature variant (via Prepare, before workers spawn) so the ablation table isolates each layer's lift; nil keeps whatever is installed.

func (ProductAdapter) Name

func (a ProductAdapter) Name() string

func (ProductAdapter) Prepare

func (a ProductAdapter) Prepare()

func (ProductAdapter) Rank

func (ProductAdapter) Rank(root string, q Question, _ []SessionDoc, k int) ([]string, error)

type QAResult

type QAResult struct {
	Dataset  string
	Adapter  string
	Model    string
	Total    int
	Correct  int
	ByType   map[string][2]int // type -> {correct, total}
	Failures []string          // question ids judged incorrect (capped)
}

QAResult aggregates answer-mode accuracy.

func RunQA

func RunQA(ds *Dataset, ad Adapter, workDir string, limit, topK int) (*QAResult, error)

RunQA answers questions with retrieval-trimmed context and judges them. topK is how many retrieved sessions ride the prompt.

func (*QAResult) Accuracy

func (r *QAResult) Accuracy() float64

func (*QAResult) Render

func (r *QAResult) Render() string

type Question

type Question struct {
	ID       string
	Type     string
	Text     string
	Answer   string // gold answer text (QA mode)
	Date     time.Time
	Gold     map[string]bool
	Haystack []SessionDoc
	Gran     Granularity // stamped from the dataset by Run; adapters may branch on it
}

Question is one scored instance. Gold holds the evidence ids at the benchmark's native granularity: session ids for LongMemEval, dia ids for LoCoMo. Haystack is the history visible to this question.

type QuestionResult

type QuestionResult struct {
	QuestionID string
	Type       string
	Ranked     []string
	RecallAtK  map[int]float64
	NDCGAtK    map[int]float64
}

QuestionResult is the scored outcome for one question under one adapter.

type RunResult

type RunResult struct {
	Dataset   string
	Adapter   string
	Questions int
	Skipped   int // no-evidence questions (abstention / adversarial)
	Recall    map[int]float64
	NDCG      map[int]float64
	ByType    map[string]typeAgg
	PerQ      []QuestionResult
}

RunResult aggregates one adapter's scores over a dataset.

func Run

func Run(ds *Dataset, ad Adapter, workDir string, limit int, keep bool) (*RunResult, error)

Run ingests each question's haystack into a throwaway .memcode root, ranks with the adapter, and scores against the gold labels. Workers bound the ingest cost; every root is deleted unless keep is set.

func (*RunResult) Render

func (r *RunResult) Render() string

Render prints the result table.

func (*RunResult) WriteLog

func (r *RunResult) WriteLog(dir string) (string, error)

WriteLog persists per-question ranked ids for later parity runs against the benchmarks' own scoring scripts.

type SessionDoc

type SessionDoc struct {
	ID    string
	TS    time.Time
	Turns []Turn
	Facts []Fact
}

SessionDoc is one benchmark chat session. Facts are attached by GenerateFacts and ingested as KindFacts records, mirroring what the product's cognition loop appends to real sessions.

type Turn

type Turn struct {
	Role     string // "user" | "assistant"
	Text     string
	ID       string
	Evidence bool
}

Turn is one benchmark utterance. ID is the benchmark's identity for the turn (LongMemEval: "<session>#<idx>"; LoCoMo: the dia_id) and rides the ingested Record so ranked results map straight back to evidence labels.