sqlite

package
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 4, 2026 License: MIT Imports: 18 Imported by: 0

Documentation

Overview

Package sqlite is the colony.db substrate: one modernc SQLite file per project, opened in WAL mode, with every write funneled through a single goroutine and reads served from their own pool so recall never waits behind a fold (doc 03 slice 1, D10). It is pure Go, no cgo, so ari stays a single static binary on every platform the release ships to.

The shape mirrors the journal writer (journal.Journal): Open prepares the file, Start brings the writer goroutine up, Close drains and closes it, so the colony controls the goroutine's lifetime exactly as it does the journal's.

Index

Constants

View Source 
const (
	TTLPinned  = "pinned"
	TTLNormal  = "normal"
	TTLFast    = "fast"
	TTLSession = "session"
)

TTL class names the decay policy a row lives under (doc 07): pinned never decays, normal decays per day, fast per hour, session evaporates when its originating session ends.

Variables

View Source 
var ErrClosed = errors.New("memory store is closed")

ErrClosed is returned to a caller whose write did not run because the store was closing. It is not a write failure; the write was never attempted, so a caller may treat it as a shutdown signal.

View Source 
var ErrNotStarted = errors.New("memory store is not started")

ErrNotStarted is returned by Write before Start brings the writer up, so a caller that writes too early gets a clear reason rather than a hang.

Functions

func HeadVersion 

func HeadVersion() (int, error)

HeadVersion is the version the embedded migrations bring a colony to, the number SchemaVersion is measured against. It reads the compiled-in migration set, so it needs no open database.

func NewID 

func NewID(now time.Time) string

NewID returns a time-sortable memory id: a microsecond timestamp big-endian, then random entropy, hex-encoded. It sorts by creation like a ULID so the recall tie-break stays meaningful, without pulling a ULID dependency in.

Types

type Anchor 

type Anchor struct {
	Kind     string // file | symbol | command
	Ref      string
	FileHash string
}

Anchor ties a memory to a file, symbol, or command, with the content hash at write time so the staleness pass can tell which anchors a commit invalidated.

type Candidate 

type Candidate struct {
	Namespace  string
	Kind       Kind // observation | reflection
	Body       string
	Importance int      // 1..10; by kind for the harvester, by the model for remember
	Anchors    []Anchor // file | symbol | command, with file hashes
	Evidence   []string // ids of the observations a reflection rests on
	Source     Source
}

Candidate is a proposed memory a worker ant emits during or after a task, from a deliberate remember call or from the loop harvesting an observation. It is never written to live memory directly; InsertCandidate appends it to the pending table and the consolidator folds it at idle or session end, so nothing an ant proposes is recallable until a fold has weighed it (D12).

type ExportRow 

type ExportRow struct {
	ID         string
	Kind       string
	Namespace  string
	Label      string
	Body       string
	Importance int
	SourceAnt  string
	SourceTask string
	ReadOnly   bool
	Verified   bool
	Anchors    []Anchor
}

ExportRow is one live memory as the export renders it: the text a human reads and edits, the anchors and provenance that round-trip, and the read_only and verified flags the render marks so a developer sees which rows a human already pinned and which the machine has confirmed. It is a projection, not the full Memory, because export needs the durable fields, not the recall bookkeeping.

type FileAnchor 

type FileAnchor struct {
	Ref  string
	Hash string
}

FileAnchor is one file anchor of a row, the path and the content hash stored when the memory was written, the pair the invalidation pass compares against the file's current state.

type Folded 

type Folded struct {
	Memory      Memory
	Anchors     []Anchor
	Evidence    []int    // indices into the merged slice; for a fold-synthesized reflection
	EvidenceIDs []string // existing memory ids; for a reflection carried from a candidate
}

Folded is one live row the consolidator writes at a fold: the memory itself, its anchors, and its evidence. A reflection synthesized in this fold cites the merged rows it rests on by their index in the batch (Evidence), and a reflection carried over from a candidate cites existing memory rows by id (EvidenceIDs). CommitFold assigns the merged ids and wires both kinds of edge, so the consolidator never authors an id or invents an evidence pointer by hand.

type Kind 

type Kind = string

Kind classifies a memory row. The storage layer owns its column vocabulary; the domain types the loop and the consolidator speak map onto these strings. 

const (
	KindObservation Kind = "observation"
	KindReflection  Kind = "reflection"
	KindSkill       Kind = "skill"
	KindPin         Kind = "pin"
)

type Memory 

type Memory struct {
	ID           string
	Namespace    string
	Kind         Kind
	Label        string
	Body         string
	Embedding    []float32 // nil when FTS-only
	EmbedModel   string
	Importance   int
	CreatedAt    int64
	AccessedAt   int64
	AccessCount  int
	SourceAnt    string
	SourceTask   string
	AnchorCommit string
	TTLClass     string
	ReadOnly     bool
	Pinned       bool
	Stale        bool // demoted by the fold's invalidation pass, downranked in recall
}

Memory is one live memory row as the store reads and writes it. It mirrors the memories table column for column; the higher layers translate their domain types to and from this shape.

type PinnedRow 

type PinnedRow struct {
	ID         string
	Label      string
	Importance int
	Anchors    []string // "kind:ref", in a stable order
}

PinnedRow is one pinned memory as the index renders it: its handle, its write-time importance for ordering, and its anchor references. It is a narrow projection, not a full Memory, because the pinned index needs only what fits on one line.

type Source 

type Source struct {
	Ant    string
	Task   string
	Commit string
}

Source is the provenance stamped on a candidate: which ant proposed it, in which task, and the commit the working tree sat at when it did. It travels with the candidate through the pending stage so a folded row can cite where it came from without the consolidator reconstructing it.

type StaleRow 

type StaleRow struct {
	ID           string
	AnchorCommit string
	Stale        bool
	Files        []FileAnchor
}

StaleRow is a live memory the invalidation pass weighs: its id, the commit it was true at, whether it is already demoted, and its file anchors. Rows with no file anchor are not returned, because staleness is about files changing under a memory, and an unanchored row evaporates on its clock instead.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store owns the write channel and the read pool. It is the only thing in the tree that holds the write connection to colony.db: every mutation runs on the one writer goroutine, and reads run on the WAL read pool that never blocks behind a write (D10).

func Open 

func Open(path string) (*Store, error)

Open prepares the store at path, creating the file and its parent if missing and setting the WAL pragmas on every connection. It starts no goroutine until Start, so a caller subscribes to the colony before the writer can produce anything, the same contract Colony.Open keeps.

func (*Store) ArchiveMemory 

func (s *Store) ArchiveMemory(ctx context.Context, ns, id string) (string, bool, error)

ArchiveMemory retires one row by stamping archived_at, forget's archival: the row drops out of recall and the pinned index but stays in the file, because nothing in ari is ever deleted (D11, D13). It returns the archived row's label and whether a row was affected, so an id that names nothing live reports it rather than silently succeeding. Archiving an already-archived row is a no-op that reports archived=false.

func (*Store) CandidateDetails 

func (s *Store) CandidateDetails(ctx context.Context, ids []string) (map[string][]Anchor, map[string][]string, error)

CandidateDetails loads the anchors and evidence for a set of candidate ids, the edges PendingCandidates leaves in their own tables until a fold needs them. It returns two maps keyed by candidate id so the consolidator can union a group's anchors and carry a reflection candidate's evidence forward without a query per candidate.

func (*Store) Close 

func (s *Store) Close() error

Close drains the writer, then closes both connection pools. Idempotent and safe from a signal handler, like Colony.Close.

func (*Store) CommitFold 

func (s *Store) CommitFold(ctx context.Context, merged []Folded, candidateIDs []string, now time.Time) error

CommitFold writes the fold's output and retires its candidates in one transaction, so a fold either lands whole or not at all and a crash mid-fold never leaves half a consolidation behind. It assigns each merged row a fresh sortable id, wires each reflection's evidence to the ids of the merged rows it rests on, and stamps folded_at on the consumed candidates so the next fold does not see them again. A reflection with no evidence is refused here too, the third and last enforcement of the no-evidence rule (D11).

func (*Store) ExportRows 

func (s *Store) ExportRows(ctx context.Context, ns string) ([]ExportRow, error)

ExportRows returns a namespace's live rows with their anchors, ordered by creation then id so the export is stable across runs. Archived rows are left out, because export is the working set a human curates, and a forgotten row stays in the file without cluttering the page a developer edits.

func (*Store) InsertCandidate 

func (s *Store) InsertCandidate(ctx context.Context, id string, c Candidate) error

InsertCandidate appends one candidate to the pending table with its anchors and evidence in a single transaction, stamping created_at from the clock so the consolidator can order intake. A reflection with no evidence is refused before any write, the D11 no-evidence-no-reflection rule at its middle enforcement point: the tool boundary refuses it first, this refuses it second, the consolidator refuses it again at fold time.

func (*Store) InsertMemory 

func (s *Store) InsertMemory(ctx context.Context, m Memory, anchors []Anchor, evidence []string) error

InsertMemory writes one live memory row with its anchors and evidence edges in a single transaction. A reflection with no evidence edge is refused before any write, the earliest of the three enforcement points of the no-evidence-no-reflection rule (D11): the loop refuses it at the tool boundary, the store refuses it here, and the consolidator refuses it again at fold time.

func (*Store) MemoryLabel 

func (s *Store) MemoryLabel(ctx context.Context, ns, id string) (string, bool, error)

MemoryLabel returns the label of one live row in a namespace, so the forget tool can render the row it would archive to the permission pipeline before it acts. found is false when no live row in the namespace has that id.

func (*Store) Migrate 

func (s *Store) Migrate(ctx context.Context) error

Migrate brings colony.db up to the head schema version, applying every embedded migration newer than the recorded version in order, each in its own transaction on the writer so a partial migration never half-lands. It is idempotent: a database already at head is a no-op, so every colony startup can call it (doc 03 slice 2, D10).

func (*Store) PendingCandidates 

func (s *Store) PendingCandidates(ctx context.Context, ns string, limit int) ([]Candidate, []string, error)

PendingCandidates returns the unfolded candidates in one namespace, oldest first, the work queue the consolidator drains at a fold. A limit of zero or less returns every pending row. The anchors and evidence stay in their own tables until the fold needs them, so this read is a cheap scan of the index-backed pending set.

func (*Store) PendingNamespaces 

func (s *Store) PendingNamespaces(ctx context.Context) ([]string, error)

PendingNamespaces lists the namespaces that have at least one unfolded candidate, the set a fold cycle iterates. It is a cheap index-backed scan so a quiet colony with nothing pending pays almost nothing to learn it has nothing to do.

func (*Store) PinnedRows 

func (s *Store) PinnedRows(ctx context.Context, ns string) ([]PinnedRow, error)

PinnedRows returns a namespace's pinned, non-archived rows with their anchors, ordered by importance then id so the render is deterministic across folds. A pinned row that was archived by forget drops out, so the index never shows a retired pin.

func (*Store) Read 

func (s *Store) Read(ctx context.Context, fn func(db *sql.DB) error) error

Read runs fn against the read pool, which serves the last committed WAL snapshot without waiting on the writer. The pool is query-only, so a fn that tries to mutate fails rather than racing the single writer.

func (*Store) Recall 

func (s *Store) Recall(ctx context.Context, ns, query string, queryVec []float32, budget int) ([]Memory, error)

Recall runs hybrid recall for one query in one namespace and returns the rows worth surfacing, most relevant first, within budget. It runs the cheap FTS5 BM25 query first, scopes cosine to the FTS matches plus a bounded same-namespace window, fuses the two ranked lists by reciprocal rank fusion, ranks the survivors by Park's triple, and bumps the access stats of what it returns so a recalled memory stays fresh (the pheromone deposit of research section 4). A nil queryVec skips the cosine stage, so the same code path serves the FTS-only world without a branch the caller must know about. Archived rows and rows outside ns never enter scoring, so a forgotten memory does not resurface and one ant's recall never leaks another's.

func (*Store) SchemaVersion 

func (s *Store) SchemaVersion(ctx context.Context) (int, error)

SchemaVersion reports the highest migration version applied to this database, the read-only view the doctor uses to tell a colony at head from one a newer binary already migrated past. A database that predates the schema_migrations table reads as version zero rather than an error, so an empty or foreign SQLite file is reported as unmigrated, not as a failure.

func (*Store) SetEmbedding 

func (s *Store) SetEmbedding(ctx context.Context, id string, vec []float32, model string) error

SetEmbedding stamps a fresh vector and model tag on one row, the write the re-embed pass makes for each stale row. It runs on the single writer like every other mutation, so a re-embed during a live turn never contends with the reader pool serving recall.

func (*Store) SetStale 

func (s *Store) SetStale(ctx context.Context, id string, stale bool, verifiedAt string) error

SetStale sets or clears a row's demotion flag. Clearing it advances verified_at to the commit the anchors were confirmed against, the field M4's verifier reads and this pass writes when a stale row's files match again.

func (*Store) StaleEmbeddings 

func (s *Store) StaleEmbeddings(ctx context.Context, model string, limit int) ([]Memory, error)

StaleEmbeddings returns live memories whose embed_model tag does not match model, the rows a re-embed pass on a configured endpoint refreshes. A NULL tag never matches, so a row written while the endpoint was down is included; a row already at model is skipped so a fold does no needless work. A limit of zero or less returns every stale row.

It reads label and body so the caller can rebuild the embedding text without a second query, and leaves the vector columns to SetEmbedding.

func (*Store) StaleRows 

func (s *Store) StaleRows(ctx context.Context, ns string) ([]StaleRow, error)

StaleRows returns the anchored, non-archived, non-read-only rows in a namespace with their file anchors attached, the input to the fold's invalidation pass. Human-edited rows are excluded because the consolidator must not touch them. Anchors come back grouped per row in id order.

func (*Store) Start 

func (s *Store) Start(ctx context.Context) error

Start brings the writer goroutine up. Separate from Open so the colony controls goroutine lifetimes (doc 01 section 4.1).

func (*Store) UpdateMemoryText 

func (s *Store) UpdateMemoryText(ctx context.Context, ns, id, label, body string) (bool, error)

UpdateMemoryText rewrites a live row's label and body and marks it read_only, the effect of a human editing an exported memory: from here the consolidator leaves the row exactly as written, because a human edit outranks the machine (D11). It returns whether a live, non-archived row in the namespace matched the id, so import can tell an applied edit from a stale one.

func (*Store) Write 

func (s *Store) Write(ctx context.Context, fn func(tx *sql.Tx) error) error

Write runs fn inside a transaction on the single writer goroutine and blocks for the result, so callers get errors synchronously while every write stays serialized (D10). A write submitted during shutdown returns ErrClosed without running.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.