retrieve

Documentation

Overview

Package retrieve holds FTS5 keyword recall: search, the linear-scan fallback, cross-project the cross-project search, and the anchor helpers (lineage walk + ranked anchor messages) that the view layer composes into bookend windows.

Index

• Constants
• func LineageRoot(con *sql.DB, sid string) string
• func SearchExplained(dbp, q string, limit int, p SearchParams) (out []Hit, explains []ScoreExplain)
• type AllHit
  • func SearchAll(q string, limit int, p SearchParams, pathPred func(cwd string) bool) []AllHit
• type Anchor
  • func MatchAnchors(con *sql.DB, q string, fetch int, p SearchParams) []Anchor
• type ExplainInputs
• type Hit
  • func LinearFallback(transcriptDir, q string, limit int, p SearchParams) []Hit
  • func Search(dbp, q string, limit int, p SearchParams) []Hit
• type ScoreExplain
  • func Explain(covs []int, in ExplainInputs) []ScoreExplain
• type SearchParams

Constants

const (
	MethodBM25         = "bm25 only"
	MethodBM25Coverage = "bm25 + coverage re-rank"
	MethodSortOverlay  = "sort overlay (recency)"
)

Ranking-regime labels reported by ScoreExplain.Method — the honest name of the rule that decided a hit's position. These mirror the three branches in Search/MatchAnchors exactly (no invented blend):

• bm25 only: single-term relevance — pure FTS5 bm25 order (rank, m.id).
• bm25 + coverage re-rank: multi-term relevance — stable re-sort by the count of distinct query terms matched; bm25 is the tiebreak.
• sort overlay (newest/oldest): a recency sort REPLACES relevance entirely; bm25 and coverage do not influence position.

Functions

func LineageRoot

func LineageRoot(con *sql.DB, sid string) string

LineageRoot walks parent_id to the conversation root (capped at 64 hops).

func SearchExplained

func SearchExplained(dbp, q string, limit int, p SearchParams) (out []Hit, explains []ScoreExplain)

SearchExplained runs the same ranking as Search and returns the top-`limit` Hits alongside a parallel, honest ScoreExplain per hit (explains[i] explains out[i]). It is the clean entrypoint the cli calls behind --debug-search; no extra query, no behavior change — the order is byte-identical to Search.

Types

type AllHit

type AllHit struct {
	Hit
	Project string
	Hits    int
}

AllHit is one cross-project cross-project result: a Hit plus the project label and the per-project hit count.

func SearchAll

func SearchAll(q string, limit int, p SearchParams, pathPred func(cwd string) bool) []AllHit

SearchAll is cross-project discovery: search every project, surface each matching project's most-recent hit, ordered by recency. `pathPred` (may be nil) filters which projects are touched.

type Anchor

type Anchor struct {
	ID        int
	SessionID string
	UUID      string // source message uuid — the stable external read-ref handle
	Role      string // empty for a vector-only synthesized anchor
	ISO       string
	Parent    string
	Snip      string
	Cov       int

	// OnlyCopySince is the session's only_copy_since watermark: >0 when the backing
	// source file was deleted by the CLI but the row is retained (RawClaw is now the only copy).
	// Surfaced so the only-copy state is clear. 0 = present upstream.
	OnlyCopySince float64

	// Attached by the fusion / discovery layers (zero until set):
	Fused   float64 // RRF score (semantic.Fuse)
	Topic   string  // topic-layer label for this anchor's segment (Fuse / TopicForMessage)
	Root    string  // lineage root session id (LineageRoot)
	Project string  // project label
	DBP     string  // db path this anchor came from
	Rank    int     // original keyword rank (tiebreak)
	Routine bool    // true when the session has an effective routine verdict
}

Anchor is one ranked anchor message from MatchAnchors — a candidate the view layer expands into a bookend window. Carries (id, session_id, role, iso, parent, snip, cov) plus the fields the view/discovery layers attach downstream.

func MatchAnchors

func MatchAnchors(con *sql.DB, q string, fetch int, p SearchParams) []Anchor

MatchAnchors runs FTS5 recall and returns ranked Anchor messages (the OR/coverage logic of Search, returning message ids for the view layer). `fetch` is the overfetch ceiling.

type ExplainInputs

type ExplainInputs struct {
	Terms    []string // distinct lowercased query terms (from lowerSet)
	Multi    bool     // true when the query OR-expanded (>1 term) — gates coverage re-rank
	Sort     string   // p.Sort: "" relevance, else "newest"/"oldest" overlay
	Routines []bool   // routine flag per hit, parallel to covs
}

ExplainInputs are the real ranking inputs an explainer needs, captured at the same point Search/MatchAnchors compute them. A caller that already ran the search passes the ordered Cov values it observed.

type Hit

type Hit struct {
	ISO        string
	SessionID  string
	Role       string
	IsSubagent bool
	Parent     string
	Snippet    string
	Routine    bool
}

Hit is one flat search result returned by Search / LinearFallback: (iso, sid, role, is_sub, parent, snip).

func LinearFallback

func LinearFallback(transcriptDir, q string, limit int, p SearchParams) []Hit

LinearFallback is the FTS5-absent linear scan over a project's JSONL, honoring the same flags + phrase (substring/adjacency) semantics.

NOTE: modernc.org/sqlite always has FTS5, so this path is dead in practice — kept for parity with the FTS5 path.

func Search

func Search(dbp, q string, limit int, p SearchParams) []Hit

Search runs the FTS5 keyword query against one project's db and returns up to `limit` ranked Hits (OR/coverage re-rank for multi-term plain queries; a single-term query is byte-identical to a plain FTS5 MATCH).

type ScoreExplain

type ScoreExplain struct {
	BM25Rank int      `json:"bm25_rank"` // ordinal in bm25 order; -1 when a sort overlay ordered the hit
	Coverage int      `json:"coverage"`  // distinct query terms matched (the real re-rank key)
	Recency  float64  `json:"recency"`   // 1 = a recency sort overlay set the order, else 0
	Final    int      `json:"final"`     // 0-based ordinal position in the returned results
	Method   string   `json:"method"`    // ranking regime: one of the Method* constants
	Terms    []string `json:"terms"`     // the lowercased distinct query terms scored against
	Tier     string   `json:"tier"`      // sort tier: "normal" or "routine"
}

ScoreExplain is the LLM-free, honest breakdown of WHY one hit landed at its rank. It carries only what the real ranking actually uses — it does NOT fabricate a composite scalar score, because RawClaw has none.

Honesty notes (read these before trusting a field):

• BM25 is NOT selected as a scalar by the live query (the SQL orders by the opaque FTS5 `rank` and never reads its value). So BM25Rank is the hit's ORDINAL position in bm25 order, not the bm25 number. -1 means "bm25 did not order this result" (a recency sort overlay was in force).
• Coverage is the REAL integer the re-rank uses: distinct query terms found in the hit's coverage haystack. Always 1 for a single-term query.
• Recency is a BOOL-as-float flag, not a weight: 1 when a sort overlay set the order, 0 otherwise. There is no recency term blended into relevance.
• Final is the hit's 0-based ordinal in the returned slice (its actual position), NOT a computed score. Method names the rule that produced it.

func Explain

func Explain(covs []int, in ExplainInputs) []ScoreExplain

Explain builds one honest ScoreExplain per (already-ordered) coverage value, in result order. `covs[i]` is the coverage the ranker computed for the hit at position i (Anchor.Cov, or scoredHit.cov). The returned breakdowns describe the REAL regime — no invented blend, no fake BM25 scalar.

Under a sort overlay, bm25 did not order the hits, so BM25Rank is -1 and Recency is 1. Under relevance, BM25Rank is the hit's bm25 ordinal: for the single-term (bm25-only) regime it equals the result position; for the multi-term regime it is unknown after the stable coverage re-sort, so it is reported as -1 (honest: we cannot recover the pre-resort bm25 ordinal here).

type SearchParams

type SearchParams struct {
	Role             string // "" = any; else "user"/"assistant"
	Sort             string // "" = relevance; "newest"/"oldest"
	IncludeTools     bool
	IncludeSubagents bool
	Since            string // "" = no bound; else YYYY-MM-DD inclusive
	Before           string // "" = no bound; else YYYY-MM-DD inclusive
	RawMatch         string // "" = plain path; else explicit FTS5 expr (boolean query)
	MinMessages      int    // 0 = no minimum

	// Scope narrowing for the one store, where project and source are columns
	// rather than a choice of which database to open. Both empty searches the
	// whole corpus. Projects is an exact-match list the caller has already
	// resolved (a path pattern is matched in Go, never pushed into SQL).
	Projects   []string
	SourceTool string
}

SearchParams groups the many optional filters shared by Search / MatchAnchors (keeps signatures under the 4-param guideline). A zero value = no filters.