sqlite

Documentation

Overview

Package sqlite provides an embedded memory.Store backed by SQLite.

Lexical retrieval uses an FTS5 virtual table with the built-in BM25 ranker; vector retrieval computes cosine similarity in process. The two modes are picked per query: when memory.Query.Embedding is set, vector search runs; otherwise FTS5 BM25 runs on memory.Query.Text. Hybrid scoring is left as a future optimization.

A metadata equality filter (memory.Query.Filter) is pushed into SQL via json_extract on the vector path, so a scoped query — e.g. a single topic — only reads and scores the matching subset instead of the whole table.

The backing driver is the pure-Go modernc.org/sqlite — no CGO and therefore no platform-specific build hurdles. The cost is that very large, unfiltered corpora hit the brute-force cosine path; pgvector / qdrant adapters land in their own modules for that case.

Index

• type Store
  • func Open(path string) (*Store, error)
  • func (s *Store) Add(ctx context.Context, chunks []memory.Chunk) error
  • func (s *Store) Close() error
  • func (s *Store) Delete(ctx context.Context, documentID string) error
  • func (s *Store) Len(ctx context.Context) (int, error)
  • func (s *Store) Retrieve(ctx context.Context, q memory.Query) ([]memory.Result, error)

Types

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store is a SQLite-backed memory.Store. Open returns a usable instance; Close releases the underlying *sql.DB.

The store is goroutine-safe (delegated to database/sql's connection pool). Concurrent Add and Retrieve calls are supported; FTS5 updates are wrapped in a transaction per Add.

func Open

func Open(path string) (*Store, error)

Open opens (or creates) a SQLite database at the given path. Use ":memory:" for an ephemeral in-process database, useful for tests. The schema is created on first open and is idempotent across re-opens.

func (*Store) Add

func (s *Store) Add(ctx context.Context, chunks []memory.Chunk) error

Add ingests chunks. Chunks with an empty ID are rejected (callers are expected to assign IDs; see uuid.NewString or any stable hash of the chunk content). Re-adding a chunk with an existing ID overwrites the previous entry, making re-ingestion idempotent.

func (*Store) Close

func (s *Store) Close() error

Close closes the underlying database handle.

func (*Store) Delete

func (s *Store) Delete(ctx context.Context, documentID string) error

Delete removes every chunk whose DocumentID matches the argument. Returns nil even when no chunks were removed (idempotent).

func (*Store) Len

func (s *Store) Len(ctx context.Context) (int, error)

Len reports the total number of chunks currently stored. Not part of the Store interface; useful for tests.

func (*Store) Retrieve

func (s *Store) Retrieve(ctx context.Context, q memory.Query) ([]memory.Result, error)

Retrieve runs q against the store. When q.Embedding is set, a brute-force cosine pass over the chunks table is used; otherwise FTS5 BM25 is queried against q.Text. Results are returned in descending Score order (higher = more relevant).