docs

Documentation

Overview

Package docs is Recall's built-in document corpus adapter: Markdown chunked by heading, retrieved from an adapter-owned lexical index, expanded live from the original file and line range.

Boundary: everything here is source-local. The adapter owns chunking, tokenization, BM25 scoring, its index generations, and its locator format; it owns no identity, no prior, and no cross-source comparison. It writes only inside the workdir supplied at handshake.

The corpus boundary is configuration rather than a rule. exclude_dirs and exclude_nested_repos decide which directories are walked; both are in the settings digest, so changing one rebuilds the index instead of answering over the boundary the old generation was built under; and every directory the walk refused is counted in health beside coverage. An exclusion nobody can see is a source reporting complete coverage over content inside the root the operator named.

Retrieval is lexical only. Semantic retrieval is a later layer that has to earn its place in evaluation, and an embedding model in this package would make the first document adapter unevaluatable against the plain baseline it is supposed to beat. Query analysis keeps a bounded set of English grammatical function words from serving as the only evidence for a match. Every non-exact candidate must itself contain a retained content term. Once it does, the full query remains its ranking input. Quoted terms, exact identifiers, all-function-word short queries, negation, and every non-English token are preserved. The analyzer version is reported in diagnostics and index_config so a ranking change cannot be invisible.

Three rules here are invariants rather than tunable behavior:

• source_record_id is the identity of the DOCUMENT, not of the chunk. Every chunk of one file carries the same value, because corroboration collapses on source_uid plus source_record_id and a per-chunk value would let this source corroborate itself with two halves of one paragraph.
• An index generation is built into a fresh directory and published by renaming a pointer, only after the corpus walk completed. A build that fails or is killed leaves the previous generation readable, and health reports it stale. Nothing partially written is ever loadable: a generation without its trailer record is rejected.
• exact_identifier is emitted only for a path-shaped identifier or a declared alias matched at token boundaries. Prose that happens to contain a file's words is a lexical match and nothing more.

Index

• type Adapter
  • func New() *Adapter
  • func (a *Adapter) Close() error
  • func (a *Adapter) Expand(ctx context.Context, req recall.ExpandRequest) (recall.ExpandResponse, error)
  • func (a *Adapter) Health(ctx context.Context) (recall.Health, error)
  • func (a *Adapter) Initialize(ctx context.Context, cfg adapter.Config) (recall.Manifest, error)
  • func (a *Adapter) Refresh(ctx context.Context, p protocol.RefreshParams) (recall.Health, error)
  • func (a *Adapter) Search(ctx context.Context, req recall.SearchRequest) (recall.SearchResponse, error)
• type Settings
  • func DefaultSettings() Settings

Types

type Adapter

type Adapter struct {
	// contains filtered or unexported fields
}

Adapter is a document corpus source: Markdown chunked by heading, ranked by BM25 from an adapter-owned index, expanded live from the file.

One Adapter owns one workdir. Concurrent searches and expansions are safe: the published generation is immutable, and a rebuild swaps a pointer rather than mutating what a search is reading.

func New

func New() *Adapter

New returns an unconfigured document adapter. Everything it needs — corpus root, workdir, settings, source name — arrives at the handshake, because a built-in adapter and an external one must be configurable the same way.

func (*Adapter) Close

func (a *Adapter) Close() error

Close releases the adapter. The published generation stays on disk: it is a projection the next handshake reopens, not state that belongs to a process.

func (*Adapter) Expand

func (a *Adapter) Expand(ctx context.Context, req recall.ExpandRequest) (recall.ExpandResponse, error)

Expand retrieves evidence behind a locator, live from the file.

func (*Adapter) Health

func (a *Adapter) Health(ctx context.Context) (recall.Health, error)

Health reports the index against the live corpus.

This is the one operation that walks the corpus, because it is the only one whose answer depends on what the source looks like NOW rather than on what the last complete boundary saw. A recent build is not health: the watermarks are what say whether the published generation still describes the corpus.

func (*Adapter) Initialize

func (a *Adapter) Initialize(ctx context.Context, cfg adapter.Config) (recall.Manifest, error)

Initialize negotiates the protocol version, validates the settings block, and makes sure a generation exists.

The first handshake builds one synchronously. The alternative is a source that answers "unavailable" until something else happens to trigger a build, which is indistinguishable from a broken source to everyone downstream.

func (*Adapter) Refresh

func (a *Adapter) Refresh(ctx context.Context, p protocol.RefreshParams) (recall.Health, error)

Refresh indexes the corpus into a new generation and publishes it.

Indexing is an operation someone schedules — at handshake, from a maintenance command, after an edit — and a build that can only be triggered as a side effect of something else cannot be tested for what happens when it fails.

A build that fails is reported through the returned health and not as an error. That is not softening the failure: health carries the stale watermark, a non-healthy status, and the reason in diagnostics, which is strictly more than an error conveys. The error return means the refresh could not run.

func (*Adapter) Search

func (a *Adapter) Search(ctx context.Context, req recall.SearchRequest) (recall.SearchResponse, error)

Search ranks chunks against the query, from the published generation only.

It does not walk the corpus. Staleness is a property of the source over time and belongs to Health, which the core probes on its own cadence; scanning the corpus on every query would make every search pay for a fact that changes far more slowly than queries arrive. Ranking therefore reads the index alone; the documents behind the results it is about to return are read once each, to cut each excerpt around the span that matched.

type Settings

type Settings struct {
	// Root is the corpus root. Empty means the instance's configured location,
	// which is the normal case; Root exists for an instance whose location
	// names something broader than the directory to index.
	Root string

	// Extensions are the file suffixes that count as documents.
	Extensions []string

	// MaxFileBytes rejects a file too large to be a document. It is a record
	// failure, not a build failure: one pathological file must not cost the
	// corpus its index.
	MaxFileBytes int64

	// ExcludeDirs are glob patterns matched against a directory's own name.
	// A directory that matches is not walked at all.
	//
	// This is declared configuration rather than a rule compiled into the walk
	// because it decides what the corpus IS, exactly as Extensions does. An
	// exclusion nobody can see is a source answering "nothing found, coverage
	// complete" over content inside the root the operator named.
	ExcludeDirs []string

	// ExcludeNestedRepos keeps the walk out of a directory that holds a .git
	// entry of its own.
	//
	// A nested checkout is the failure the dot-class exclusion was really
	// aimed at: `.claude/worktrees/agent-1/` is a whole second copy of the
	// corpus, so the same document is indexed twice under distinct paths.
	// Lineage groups on source_record_id, those copies carry different ones,
	// and one document arrives as several independent lineage roots that
	// corroborate each other. Nothing downstream can detect it, because by
	// then the copies genuinely are distinct records.
	ExcludeNestedRepos bool

	// ExamplesQuoteQueries declares that this corpus quotes realistic user
	// queries as worked examples.
	//
	// Documentation does this deliberately and should: the relevance section of
	// docs/adapter-protocol.md is easier to read because it argues over "make a
	// dentist appointment" rather than over foo and bar. The cost is that a
	// corpus quoting user queries matches those queries, and half the answer to
	// `recall query dentist` was the retrieval system discussing the retrieval
	// of dentists.
	//
	// Where it is set, an occurrence inside a quotation does not count toward
	// how much a chunk is ABOUT the query. It stays indexed and stays findable:
	// somebody asking how relevance is defined must still reach that section,
	// and a chunk withheld for a low relevance is reported as a suppression
	// rather than dropped. What changes is that the chunk stops competing, on a
	// query about the thing it was quoting, with records that are that thing.
	//
	// It is declared per source rather than inferred because only the corpus
	// knows: a note quoting a decision IS the decision, and the same rule
	// applied there would discount the strongest evidence a personal corpus
	// holds.
	ExamplesQuoteQueries bool

	// Aliases are declared stable names for a document, keyed by its
	// corpus-relative path. They are the only single-word identifiers that may
	// produce an exact_identifier signal, because a person wrote them down for
	// this corpus.
	Aliases map[string][]string
}

Settings is the adapter-owned settings block for one document source instance. Every field here changes an explained code path; a field that configured nothing would be a defect.

func DefaultSettings

func DefaultSettings() Settings

DefaultSettings is the configuration an instance gets when its settings block is empty.