index

package
v0.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 25, 2026 License: MIT Imports: 32 Imported by: 0

Documentation

Overview

Package index owns ingest orchestration over the on-disk SQLite/FTS5 store: schema ensuring (over internal/store's DDL), file fingerprinting, incremental reindexing, and corpus stats. Pure-Go via modernc.org/sqlite (no cgo).

Package index (this file): reaching RETAINED sessions for `rawclaw delete`.

lifecycle.Delete's matchSessions walks the live projects tree only — it can never see a row whose backing .jsonl the source tool already purged. That is exactly the row set durable retention creates (only_copy_since set, content still indexed), so without this the feature's own contract — "explicit delete is the ONLY way retained history dies" — was unreachable in practice (live-verified: `delete --project <purged>` reported "no sessions match"). lifecycle cannot import this package back (index already imports lifecycle, for LoadTombstones) so the retained-side scan lives here and the CLI composes the two searches.

Index

Constants

View Source 
const (
	MetaLastIngestTime         = "last_ingest_time"
	MetaLastIngestCatalogMTime = "last_ingest_catalog_mtime"
)

Meta keys for index-level freshness watermarks.

View Source 
const ArchiveDBPrefix = "archive-"

ArchiveDBPrefix namespaces the cache dbs of ARCHIVE-replica scopes (foreign machines' sessions pulled through the transcript archive). The prefix is what keeps those read-only replicas out of the local orphan-db discovery and out of the delete path's retained-row scan; local db names are path-encodings of absolute dirs (or "codex-..."), so no local scope can collide with it.

View Source 
const ConsolidatedDBName = "consolidated.db"

ConsolidatedDBName is the one store every session and message lands in, regardless of which project, source, or machine it came from. It is the shape Hermes' state.db has: project is a COLUMN, not a filename.

The name is deliberately not a path encoding, so scope discovery can tell it apart from a per-project db by name alone (see scopes.orphanClaudeScopes, which must skip it — otherwise every consolidated row would also be searched a second time as an "orphaned" project scope).

Variables

View Source 
var (
	IncrementalIngestCount atomic.Int64
	FullReindexCount       atomic.Int64
)

Ingest tracing counters for verification and test assertions.

View Source 
var ErrRebuildWouldLoseHistory = errors.New("rebuild would lose history")

ErrRebuildWouldLoseHistory reports a rebuild refused because the vault holds fewer sessions than the store it was about to replace. Callers match on it to print the override rather than a bare failure.

Functions

func ConsolidatedPath added in v0.9.0 

func ConsolidatedPath() string

ConsolidatedPath returns the consolidated store's path in the cache dir.

func DBPath 

func DBPath(transcriptDir string) string

DBPath returns the cache db path for a transcript dir: ~/.cache/session-search/<encoded-dir>.db (creating the dir).

func EnsureFreshContainer added in v0.9.0 

func EnsureFreshContainer(
	dbp string,
	c source.Container,
	msgs MessagesFunc,
	sourceID string,
) (int, error)

EnsureFreshContainer incrementally refreshes one live container, proves its watermark matches the current file, and strictly folds it into the consolidated store. Unlike ordinary advisory indexing, any uncertainty is an error: tag-prep must not print a known-stale partial transcript.

func EnsureOrphanReconciled added in v0.3.0 

func EnsureOrphanReconciled(dbp string) (int, error)

EnsureOrphanReconciled reconciles an orphaned index db read-MOSTLY: a read-only probe first decides whether a reconcile would change anything — a tombstoned session still present, an own-source row not yet stamped only_copy_since, or (mirror mode) an own-source row awaiting the prune. Only pending work opens the db read-write (ReconcileOrphanDB); the common case — re-discovering an already-reconciled archive on every search — is a pure read that never touches the file. A probe failure (e.g. a pre-durability schema without the provenance columns) falls through to the read-write reconcile, whose EnsureSchema migrates it.

func EnsureSchema 

func EnsureSchema(con *sql.DB, sourceID string) error

EnsureSchema creates the base schema, the FTS table if missing, and rebuilds on any SchemaVersion mismatch or missing marker. sourceID is the scope's source ("claude"/"codex"), used only to backfill source_tool on an in-place durability migration (D6).

func FTS5OK 

func FTS5OK() bool

FTS5OK reports whether FTS5 is available on this build (always true for modernc.org/sqlite v1.45.0; kept for graceful-degrade callers).

func IsBusy added in v0.10.0 

func IsBusy(err error) bool

IsBusy reports whether err represents a SQLite busy/locked condition.

func IsConsolidatedDB added in v0.9.0 

func IsConsolidatedDB(dbFileName string) bool

IsConsolidatedDB reports whether a cache db filename is the consolidated store. Callers that enumerate cache dbs as scopes MUST skip it: it is a superset of them, not a peer.

func OpenConsolidated added in v0.9.0 

func OpenConsolidated() (*sql.DB, int, error)

OpenConsolidated opens the one store read-only for a read verb and reports how many sessions it holds. It returns an error — never a usable connection — when the store is absent, unreadable, or empty. Those three states look identical to "nothing matched" once a query has run against them, and a confident empty answer from a store that was never filled is the one failure a reader must not produce. A caller that gets an error falls back to the per-project databases and says which store answered.

func PerProjectDBs added in v0.9.0 

func PerProjectDBs() ([]string, error)

PerProjectDBs lists the per-project cache dbs that feed the consolidated store: every *.db in the cache dir except the consolidated store itself and the non-index sidecars. Archive replicas ARE included — a foreign machine's sessions carry origin_machine on the row, so they belong in the one store exactly like local ones.

func PrepareFreshContainer added in v0.10.0 

func PrepareFreshContainer(
	dbp string,
	c source.Container,
	msgs MessagesFunc,
	sourceID string,
) (int, error)

PrepareFreshContainer incrementally refreshes one live container and proves its watermark matches the current file in the refresh db without folding into the consolidated store.

func ReconcileOrphanDB added in v0.3.0 

func ReconcileOrphanDB(dbp string) (nSessions int, err error)

ReconcileOrphanDB reconciles an existing index db whose source dir has vanished entirely — the 30-day-purge case where AllProjectDirs no longer yields the project, so the normal source→index pass never runs for it (D8). It reconciles against an EMPTY live scan: every own-source session is stamped only_copy_since and RETAINED, an explicit tombstone deletes, a foreign row is untouched — the same rules as an in-place UpdateIndex, minus the reindex (there is no source to read). Returns the surviving top-level session count so the caller can drop a db that reads as fully deleted. A busy/locked db is a soft no-op that degrades to the current read count rather than erroring the whole discovery pass.

func RefreshDBPath added in v0.9.0 

func RefreshDBPath(sourceID, sessionID, sourcePath string) string

RefreshDBPath returns the private per-container cache used by targeted live refreshes. It lives below the cache root so normal scope/orphan discovery never mistakes it for another searchable project database.

func ReindexFile 

func ReindexFile(con *sql.DB, path, transcriptDir string) bool

ReindexFile parses the whole file into memory FIRST, then atomically replaces ReindexFile parses the whole file into memory FIRST, then atomically replaces this session's rows under a single transaction (messages, session row, watermark). Returns true on success. Rows are stamped with this machine's identity; a replicated tree goes through reindexFileWithOrigin instead.

func ResetIngestCountersForTesting added in v0.10.0 

func ResetIngestCountersForTesting()

ResetIngestCountersForTesting resets the incremental and full reindex counters.

func SetVectorTopupHook added in v0.9.0 

func SetVectorTopupHook(fn func(string))

SetVectorTopupHook installs the post-index vector top-up. Called once from package semantic's init; a nil fn restores the no-op.

func StampIngestWatermark added in v0.10.0 

func StampIngestWatermark(con *sql.DB) error

StampIngestWatermark records the current epoch and catalog directory mtime in the consolidated store's meta table so subsequent read verbs can verify freshness in O(1).

func SyncConsolidatedFrom added in v0.9.0 

func SyncConsolidatedFrom(srcPath string) error

SyncConsolidatedFrom folds ONE per-project db into the consolidated store. This is the write-through half: an indexing run updates its own project db, then hands that db here, so the consolidated store tracks without a separate pass over the transcripts.

func UnconsolidatedDBs added in v0.9.0 

func UnconsolidatedDBs(con *sql.DB) ([]string, error)

UnconsolidatedDBs returns the per-project databases the one store has never folded in, by comparing the cache directory against the fold-in watermarks the store stamps. This is what keeps a one-store read honest: a project whose database exists but was never merged is missing from every answer, and a reader has to be able to name it rather than let the corpus quietly shrink.

It compares presence, not freshness — a source that changed after its fold-in is not detected here, because proving that would mean opening every source database, which is the per-project fan-out this work exists to remove.

func UpdateIndex 

func UpdateIndex(con *sql.DB, transcriptDir string) error

UpdateIndex performs the incremental reindex of transcriptDir: fingerprint each contained file, reindex changed ones, prune deleted sessions. Writes commit under database/sql autocommit.

Types

type ConsolidatedFence added in v0.10.0 

type ConsolidatedFence struct {
	// contains filtered or unexported fields
}

func AcquireConsolidatedFence added in v0.10.0 

func AcquireConsolidatedFence(ctx context.Context) (*ConsolidatedFence, error)

func (*ConsolidatedFence) Close added in v0.10.0 

func (f *ConsolidatedFence) Close() error

type IndexFreshness added in v0.10.0 

type IndexFreshness struct {
	Fresh  bool
	Reason string
}

IndexFreshness reports the result of the O(1) index-level freshness check.

func CheckIndexFreshness added in v0.10.0 

func CheckIndexFreshness(con *sql.DB) (IndexFreshness, error)

CheckIndexFreshness evaluates whether the consolidated store is current by comparing the last-ingest watermark in meta against a single stat of the session catalog dir. It is strictly O(1): at most 1 stat of the catalog directory and 1 DB meta query.

func CheckProjectFreshness added in v0.10.0 

func CheckProjectFreshness(con *sql.DB, projectLabel, tdir string, sourceTool ...string) (IndexFreshness, error)

CheckProjectFreshness checks whether a specific project's transcript files have changed since they were last indexed into the consolidated store. Missing signal (e.g. hooks absent, unindexed transcripts, or missing watermarks) reports Fresh: false.

type IndexStatus 

type IndexStatus int

IndexStatus discriminates how EnsureIndexed obtained its result, so callers can honestly report incompleteness (#6) instead of silently treating a stale busy-lock fallback as a fresh index. 

const (
	// IndexStatusUnknown: uninitialised or indeterminate status (e.g. error returned).
	IndexStatusUnknown IndexStatus = iota
	// IndexFresh: the index was built/updated this call (the result is current).
	IndexFresh
	// IndexStale: a busy/lock collision forced a fall-back to the EXISTING
	// (possibly out-of-date) cached index — the result may be incomplete.
	IndexStale
)

func EnsureIndexed 

func EnsureIndexed(tdir string, reindex bool) (dbp string, nSessions int, status IndexStatus, err error)

EnsureIndexed builds/updates one project's FTS index and returns (db_path, n_sessions, status). On busy-lock it falls back to the existing index with CountSessions and reports IndexStale. If reindex is true and the db exists, it is removed first.

func EnsureIndexedContainers 

func EnsureIndexedContainers(dbp string, reindex bool, cs []source.Container, msgs MessagesFunc, sourceID, origin string) (nSessions int, status IndexStatus, err error)

EnsureIndexedContainers builds/updates the db at dbp from cs (one scope's containers), pulling each container's messages via msgs. It mirrors EnsureIndexed's reindex + busy-lock semantics, but is source-agnostic: the containers carry their own id, lineage, and backing path, replacing the Claude-only directory walk of UpdateIndex. sourceID (the source's Registration.ID, e.g. "codex") is stamped as each row's source_tool (D3), injected alongside msgs so the index never imports the concrete adapters. origin is the origin_machine to stamp ("" = this machine) — a replicated tree's containers carry their owner's identity.

CONTRACT — cs MUST be the COMPLETE container set for dbp on every call. The retention pass (updateContainers) reconciles indexed sessions against cs as the full live scan: in a REPLICA scope (origin set) an omitted session is pruned outright; in a local scope it is stamped only_copy_since — either way a partial cs corrupts the outcome for the omitted sessions. Corollary: never point two sources (or two scopes) at the same dbp — give each its own, distinctly-namespaced cache file, so one source's set is never "incomplete" relative to another's rows.

func EnsureIndexedTree added in v0.5.0 

func EnsureIndexedTree(dbp, tdir string, reindex bool, origin string) (nSessions int, status IndexStatus, err error)

EnsureIndexedTree builds/updates the FTS index for one Claude-shaped transcript tree at an EXPLICIT db path, stamping origin as every row's origin_machine ("" = this machine). This is EnsureIndexed with both halves of the identity made injectable: a replicated tree (another machine's transcripts synced onto this disk) indexes into its own namespaced db and carries its owner's identity, while the local path keeps its derived db and local stamp. Reindex + busy-lock semantics are identical to EnsureIndexed.

type MessagesFunc 

type MessagesFunc func(source.Container) ([]model.Message, error)

MessagesFunc yields one container's normalized messages — a source adapter's Messages method, injected so this package never imports the concrete adapters. The index stays source-agnostic; the caller (cli) wires source → index.

type RebuildStats added in v0.9.0 

type RebuildStats struct {
	Sessions   int // sessions written into the store
	Messages   int // messages written into the store
	Missing    int // of those sessions, ones whose original source file is gone
	Tombstoned int // vaulted sessions skipped because the user deleted them
	Unreadable int // vaulted transcripts that could not be read (reported, never silent)
}

RebuildStats reports what one rebuild-from-transcripts pass produced.

func RebuildFromTranscripts added in v0.9.0 

func RebuildFromTranscripts(dbp string) (RebuildStats, error)

RebuildFromTranscripts rebuilds the index db at dbp from the durable transcript vault alone — the guarantee that makes the store a cache: delete it, run this, and every session comes back, INCLUDING the ones whose original source file no longer exists anywhere on disk.

There is deliberately NO retention pass here. Retention reconciles indexed sessions against a live source scan; this pass has no scan — its input is the vault, whose own sidecars already carry the verdict a previous scan reached. Running one here would see every vaulted session's source path and re-derive the flags from scratch, which is both redundant and wrong for a session that was already retained-and-flagged.

type RetainedSession added in v0.3.0 

type RetainedSession struct {
	DBPath       string  // the index db holding the row
	SessionID    string  // .jsonl stem == the claude --resume id
	Label        string  // friendly project label (source_path's dir, or the db name)
	LastTS       float64 // last message timestamp, epoch seconds (0 if never recorded)
	MessageCount int
}

RetainedSession is one RETAINED top-level session matched by RetainedMatches — its backing .jsonl is already gone, so unlike lifecycle.PlanItem it carries no file size (there is no file left to size).

func RetainedMatches added in v0.3.0 

func RetainedMatches(cacheDir string, project string, before time.Time, maxMessages int, sessionID string) ([]RetainedSession, error)

RetainedMatches enumerates every RETAINED top-level session (only_copy_since IS NOT NULL, is_subagent=0) across every index db under cacheDir, applying the same filter semantics lifecycle.DeleteOpts uses so a delete plan can union live matches with retained ones. cacheDir defaults to store.CacheDir() when empty, mirroring lifecycle.TombstonePath's own-default convention.

project, when non-empty, is a case-sensitive substring match against EITHER the row's source_path or the db's own filename — the same "path contains" semantic Delete's Project filter uses on the live transcript-dir path, extended to the db filename so an orphaned db still matches after its source_path predates a rename (or was never backfilled by migrateDurabilityColumns). before, when non-zero, keeps only sessions whose last_ts (epoch seconds) is strictly before it — a row with no recorded last_ts is excluded rather than guessed at. maxMessages, when > 0, keeps only sessions with at most that many messages. sessionID, when non-empty, keeps only the session lifecycle.MatchesSessionID addresses (exact id, or a >=8-char prefix) — the positional-delete form. A zero-value filter is unset and does not constrain the match, same rule as DeleteOpts.

A db that fails to open or query (busy, corrupt, mid-write) is skipped rather than failing the whole scan — the same best-effort tolerance scopes.orphanClaudeScopes applies to its own db enumeration.

type SessionFreshness added in v0.10.0 

type SessionFreshness struct {
	Status     SessionFreshnessStatus
	SessionID  string
	SourcePath string
	Note       string
}

SessionFreshness contains the detailed outcome of an O(1) session freshness check.

func CheckSessionFreshness added in v0.10.0 

func CheckSessionFreshness(con *sql.DB, sessionID string) (SessionFreshness, error)

CheckSessionFreshness checks a specific session's freshness in O(1) by comparing its stored file_index watermark against a single stat of its backing transcript.

type SessionFreshnessStatus added in v0.10.0 

type SessionFreshnessStatus int

SessionFreshnessStatus discriminates the freshness of an individual session. 

const (
	SessionFresh SessionFreshnessStatus = iota
	SessionStale
	SessionMissingBacking
	SessionNotFound
)

type SyncStats added in v0.9.0 

type SyncStats struct {
	Sources        int // per-project dbs read
	Skipped        int // of those, dbs too old to read (reported, never silent)
	SessionsSeen   int // session rows offered by those dbs (sum, duplicates included)
	Sessions       int // distinct sessions in the consolidated store afterwards
	Messages       int // distinct messages in the consolidated store afterwards
	CarriedForward int // rebuild only: sessions kept although no source still offers them
}

SyncStats reports what one consolidation pass moved.

func ConsolidateFrom added in v0.9.0 

func ConsolidateFrom(srcPaths []string, rebuild bool) (st SyncStats, err error)

ConsolidateFrom fills the consolidated store from the given per-project db paths. Passing rebuild drops the store first, so a full re-run costs one pass over the existing dbs rather than a re-read of every transcript on disk.

Sources are read through SQLite's ATTACH, so nothing is parsed twice: the transcripts were already turned into rows once, and this moves those rows.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.