veclite

package module
v0.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: MIT Imports: 19 Imported by: 0

README

VecLite

Embeddable vector database for Go.

Store vectors with metadata in a single file. Search with cosine similarity, dot product, or Euclidean distance. Add HNSW for fast approximate nearest neighbors. Use BM25 for full-text search. Combine both with hybrid search.

Table of Contents

Features

  • Embeddable Go library -- Core vector storage and search are small and local-first; optional integrations add provider-specific modules
  • Single-file storage -- Database persists to one .veclite file
  • Private by default -- New database, lock, and storage-directory artifacts are owner-only on POSIX systems
  • HNSW indexing -- Fast approximate nearest neighbor search
  • BM25 text search -- Full-text search over record content and payload fields
  • Hybrid search -- Combine vector and text search with Reciprocal Rank Fusion
  • Named vector spaces -- Multiple independent embeddings per record (e.g. text + image) with per-space fusion
  • Embedding profiles -- First-class EmbeddingProfile with dimension validation and compatibility checks
  • Document storage -- Store original text content alongside vectors
  • Auto-embedding -- Pluggable Embedder interface for text-to-vector conversion
  • Local ONNX embedder -- Optional all-MiniLM-L6-v2 embedder for local inference (build with -tags onnx)
  • Metadata filtering -- Rich filter expressions (equality, range, glob, prefix, logical operators)
  • Streaming results -- Process results via callback without materializing all at once
  • Pagination -- Offset/limit on search results and record iteration
  • Observability -- Structured logger interface and atomic metrics counters
  • Thread-safe -- Safe for concurrent read/write access
  • In-memory mode -- Use :memory: for testing
  • CLI and HTTP server -- Manage databases from the command line or over REST
  • MCP server -- Expose VecLite as tools for AI agents via Model Context Protocol
Agent Memory Features
  • TTL and expiration -- Records can expire automatically after a duration
  • Background TTL cleanup -- Automatic periodic cleanup of expired records
  • Importance scoring -- Assign importance (0.0-1.0) to records for prioritized retrieval
  • Temporal decay -- Search scores decay based on record age (exponential, linear, gaussian)
  • Access tracking -- Track when records are accessed and how often
  • Memory pressure handling -- Automatic eviction with FIFO, LRU, or importance policies
  • Conversation memory -- Session-based conversation turns with threading support
  • Real-time subscriptions -- Get notified when new records match a query
  • Memory consolidation -- Cluster and consolidate similar memories
  • Episodic memory -- Group related memories into coherent episodes
  • Knowledge graph -- Entity-relationship graph with traversal and vector search

Quick Start

package main

import (
    "fmt"
    "github.com/abdul-hamid-achik/veclite"
)

func main() {
    db, _ := veclite.Open("vectors.veclite")
    defer db.Close()

    coll, _ := db.CreateCollection("embeddings",
        veclite.WithDimension(384),
        veclite.WithHNSW(16, 200),
    )

    coll.Insert([]float32{0.1, 0.2 /* ... */}, map[string]any{"file": "main.go"})

    results, _ := coll.Search(queryVector, veclite.TopK(10))
    for _, r := range results {
        fmt.Printf("ID: %d, Score: %.4f\n", r.Record.ID, r.Score)
    }
}

Installation

Requires Go 1.25 or later. VecLite v0.23.1 raised the minimum from Go 1.23 because the official MCP Go SDK now requires Go 1.25. VecLite's core storage and search APIs are local-first; optional integrations such as MCP, YAML config, and ONNX embedding use external Go modules.

# Library
go get github.com/abdul-hamid-achik/veclite

# CLI
go install github.com/abdul-hamid-achik/veclite/cmd/veclite@latest

Pre-built binaries are available on the Releases page.

Documentation Site

The documentation site uses VitePress and Bun. Build it locally with:

task site

Use task site-dev while editing documentation and task site-preview to preview the production build.

Use as a Go Library

VecLite is primarily an importable Go library:

import "github.com/abdul-hamid-achik/veclite"

Applications can bring their own embedding pipeline and use VecLite for durable local storage, HNSW vector search, BM25 text search, metadata filtering, and hybrid ranking. See docs/embeddings.md for the app/library boundary and embedding-profile guidance.

Use from any language

VecLite is usable as a Go library and as a standalone engine driven from any language through its CLI and HTTP server, both of which speak JSON. Language drivers (Python, TypeScript, …) are planned, not yet written — but the CLI and HTTP JSON shapes are treated as a stable cross-language contract and are pinned by the behavior specs under specs/glyphrun/. If you are integrating from another language today, drive veclite serve over HTTP or shell out to the veclite CLI with --json.

Embedding Strategy

A collection has one default vector space plus optional named vector spaces, so one logical record can hold several embeddings (e.g. text, image, audio), each with its own dimension and index. Use one collection with named spaces for multimodal records; use separate collections only when records are genuinely unrelated. See the Named Vector Spaces guide and ADR-0001 for the design.

Project Status

See docs/project-status.md for the current implementation boundary, related-project usage notes, and missing work.

Library API

Opening a Database
// File-based (persistent)
db, err := veclite.Open("vectors.veclite")

// In-memory (testing)
db, err := veclite.Open(":memory:")

// With options
db, err := veclite.Open("vectors.veclite",
    veclite.WithWAL(true),          // Write-ahead log: crash-safe writes
    veclite.WithSyncOnWrite(true),  // Sync after each write
    veclite.WithReadOnly(true),     // Read-only mode
    veclite.WithSharedRead(true),   // Shared lock for multi-process reads
    veclite.WithLogger(myLogger),   // Structured logging
)

defer db.Close()
DB Option Description
WithWAL(bool) Write-ahead log. Each write appends the affected records to a *.wal sidecar with one fsync, so writes survive a crash between snapshot saves at a fraction of WithSyncOnWrite's cost. See Durability and the WAL.
WithWALCheckpoint(bytes) Log size at which the WAL is automatically folded into a snapshot and truncated (default 64 MiB). 0 disables auto-checkpointing.
WithSyncOnWrite(bool) Full snapshot save after each write. Slowest, maximally conservative durability.
WithReadOnly(bool) Open in read-only mode. Write operations return errors.
WithSharedRead(bool) Open read-only lock-free (no flock). Requires WithReadOnly(true). A long-lived reader never blocks a writer and is never blocked by one; readers see a point-in-time snapshot, so call db.Reload() to pick up concurrent writes. Consistency is guaranteed by the writer's atomic-replace save.
WithLogger(Logger) Set structured logger. Default is NopLogger (zero overhead).
Durability and the Write-Ahead Log

VecLite persists the whole database as a single snapshot file on Sync() and Close(). By default, writes made between those points live only in memory. Three durability modes cover the spectrum:

Mode Cost per write Crash loses
Default none everything since the last Sync/Close
WithWAL(true) one small append + fsync nothing (completed writes are replayed)
WithSyncOnWrite(true) full snapshot rewrite + fsync nothing 
db, err := veclite.Open("vectors.veclite", veclite.WithWAL(true))

With the WAL enabled, every completed mutation (inserts, updates, deletes, multi-space records, text documents, metadata, collection lifecycle, knowledge-graph and episode-store changes) is appended to vectors.veclite.wal before the call returns. On the next open — after a crash or kill -9 — the log is replayed on top of the last snapshot, indexes (HNSW and BM25) are restored to match, and the recovered state is folded into a fresh snapshot. A clean Sync() or Close() truncates the log, and long-running writers that never call Sync are covered by automatic checkpointing: once the log exceeds WithWALCheckpoint bytes (default 64 MiB), it is folded into a snapshot and truncated on the spot.

Notes:

  • Opening a database without WithWAL still recovers and folds a log left behind by a crashed WAL-enabled writer; nothing is lost by mixing modes.
  • Read-only opens (including WithSharedRead) replay the log in memory without touching it, and Reload() re-applies it — so shared readers can observe a live writer's not-yet-snapshotted writes.
  • Entries are CRC-checked; a torn append from a crash mid-write is discarded.
  • Read-path bookkeeping (access counts) is not logged; it persists on the next full save.
Collections
// Get or create with defaults (auto-detects dimension on first insert)
coll := db.Collection("embeddings")

// Create with explicit options
coll, err := db.CreateCollection("embeddings",
    veclite.WithDimension(384),
    veclite.WithDistanceType(veclite.DistanceCosine),
    veclite.WithHNSW(16, 200),
    veclite.WithTextIndex("title", "body"),
    veclite.WithEmbedder(myEmbedder),
)

// Get existing (returns error if not found)
coll, err := db.GetCollection("embeddings")

// List, check, drop
names := db.Collections()
exists := db.HasCollection("embeddings")
err := db.DropCollection("embeddings")
Collection Option Description
WithDimension(int) Fixed vector dimension. 0 (default) auto-detects on first insert.
WithDistanceType(DistanceType) Distance metric. Default: DistanceCosine.
WithHNSW(m, efConstruction) Enable HNSW index with given parameters.
WithHNSWConfig(HNSWConfig) Enable HNSW with full configuration struct.
WithTextIndex(fields...) Enable BM25 text indexing on named payload fields. Content is always indexed.
WithEmbedder(Embedder) Set auto-embedding plugin for InsertText/SearchText.
Database and Collection Metadata

Store application-level or collection-level metadata alongside the database file:

err := db.SetMetadataValue("app", "vecgrep")
dbMeta := db.Metadata()

err = coll.SetMetadataValue("embedding_profile", map[string]any{
    "provider":   "ollama",
    "model":      "nomic-embed-text",
    "dimensions": 768,
    "distance":   "cosine",
})
collMeta := coll.Metadata()

err = coll.DeleteMetadataValue("deprecated_key")

Metadata returns a deep copy. Use database and collection metadata for schema/profile information. Use record payloads for per-record fields that need filtering or retrieval.

Distance metrics:

Metric Constant Interpretation
Cosine Similarity DistanceCosine Higher = more similar
Dot Product DistanceDot Higher = more similar
Euclidean DistanceEuclidean Lower = more similar
Inserting Vectors
// Single insert with metadata
id, err := coll.Insert(vector, map[string]any{
    "file": "main.go",
    "type": "code",
})

// Batch insert
ids, err := coll.InsertBatch(
    [][]float32{v1, v2, v3},
    []map[string]any{p1, p2, p3},
)
Document Storage

Store original text content alongside vectors for text search and retrieval:

id, err := coll.InsertDocument(
    vector,
    "Go is a statically typed language designed at Google",
    map[string]any{"title": "Go Language", "category": "programming"},
)

The Content field is stored on the Record and automatically indexed by BM25 when text indexing is enabled.

For keyword-first workflows, store text without a vector:

id, err := coll.InsertTextDocument(
    "00:12 OCR and transcript evidence",
    map[string]any{"frame": "frames/frame_0012.png"},
)

id, err = coll.InsertTextDocumentWithOptions(
    "temporary transcript evidence",
    map[string]any{"frame": "frames/frame_0013.png"},
    veclite.WithTTL(24 * time.Hour),
    veclite.WithImportance(0.8),
)

Text-only records are returned by TextSearch, filters, iteration, and direct lookup. Vector search skips them.

Searching
// Basic search
results, err := coll.Search(queryVector, veclite.TopK(10))

// With minimum similarity threshold
results, err := coll.Search(queryVector,
    veclite.TopK(10),
    veclite.Threshold(0.8),
)

// With HNSW tuning (higher ef = better recall, slower)
results, err := coll.Search(queryVector,
    veclite.TopK(10),
    veclite.WithEfSearch(200),
)

// Access results
for _, r := range results {
    fmt.Printf("ID: %d, Score: %.4f, Payload: %v\n",
        r.Record.ID, r.Score, r.Record.Payload)
}
Search Option Description
TopK(k) Maximum results to return. Default: 10.
Threshold(t) Minimum similarity score.
WithFilter(f) Add a filter. Multiple filters use AND logic.
WithFilters(f...) Add multiple filters (AND logic).
WithEfSearch(ef) HNSW ef parameter. Higher = better recall, slower.
WithOffset(n) Skip first n results (pagination).
WithLimit(n) Alias for TopK in pagination contexts.
WithContent(bool) Include/exclude Content field in results.
WithVectorWeight(w) Vector search weight in hybrid search. Default: 1.0.
WithTextWeight(w) Text search weight in hybrid search. Default: 1.0.
Text Search (BM25)

Full-text search using BM25 ranking. Requires WithTextIndex on the collection.

coll, _ := db.CreateCollection("docs",
    veclite.WithTextIndex("title", "body"),
)

// Insert a keyword-searchable document without an embedding vector
coll.InsertTextDocument("Go programming language", map[string]any{
    "title": "Go Language",
    "body":  "Fast and efficient",
})

// Search by text
results, err := coll.TextSearch("Go programming", veclite.TopK(10))

BM25 indexes the Content field automatically, plus any payload fields specified in WithTextIndex. Uses standard BM25 parameters (k1=1.2, b=0.75).

Combine vector similarity and BM25 text search using Reciprocal Rank Fusion (RRF):

results, err := coll.HybridSearch(
    queryVector,
    "Go programming",
    veclite.TopK(10),
    veclite.WithVectorWeight(1.0),
    veclite.WithTextWeight(0.5),
)

RRF merges ranked lists from both searches with configurable weights. This produces better results than either search alone when queries have both semantic and keyword components.

Named Vector Spaces

A collection has one implicit default space (backed by Record.Vector) and may declare additional named spaces — each with its own dimension, distance metric, and HNSW index. One logical record can then carry several embeddings at once (e.g. text and image). This is fully additive: the entire single-vector API above keeps working on the default space, and databases written before this feature load unchanged.

coll, _ := db.CreateCollection("items",
    veclite.WithDimension(1536), // default space (text)
    veclite.WithVectorSpace(veclite.VectorSpaceConfig{
        Name: "image", Dimension: 512, Distance: veclite.DistanceCosine, Modality: "image",
        HNSW: &veclite.HNSWConfig{M: 16, EfConstruction: 200, EfSearch: 100, UseHeuristic: true},
    }),
)

// One logical record with vectors in two spaces.
id, _ := coll.InsertRecord(veclite.RecordInput{
    Content: "a red apple",
    Payload: map[string]any{"label": "apple"},
    Vectors: map[string][]float32{
        veclite.DefaultVectorSpace: textVector,
        "image":                    imageVector,
    },
})

// Search one space, or fuse several with Reciprocal Rank Fusion.
byImage, _ := coll.SearchSpace("image", imageQuery, veclite.TopK(10))
fused, _   := coll.MultiSpaceSearch(map[string][]float32{
    veclite.DefaultVectorSpace: textQuery,
    "image":                    imageQuery,
}, veclite.TopK(10))

FuseRRF is exposed publicly for custom weighted fusion across any result sets (vector spaces, BM25, or externally produced rankings). Attach a first-class EmbeddingProfile to a collection (WithEmbeddingProfile) or a space (VectorSpaceConfig.Profile) to validate inserts and detect index-invalidating provider/model changes via EmbeddingProfile.Compatible.

Named spaces also have the upsert-by-key and hybrid-search analogs of the single-space API:

// Upsert by a payload key, carrying vectors in several spaces atomically.
// Returns (id, inserted, err) — inserted=false means an existing record was replaced
// (its CreatedAt and AccessCount are preserved).
id, inserted, err := coll.UpsertRecordByKey("evidence_id", "doc-1", veclite.RecordInput{
    Content: "checkout fails",
    Payload: map[string]any{"evidence_id": "doc-1"},
    Vectors: map[string][]float32{"text": textVec},
})

// Hybrid search over a named space: fuses vector results from the space with
// BM25 text results via Reciprocal Rank Fusion. Passing "" or DefaultVectorSpace
// is equivalent to HybridSearch.
results, err := coll.HybridSearchSpace("text", queryVec, "checkout", veclite.TopK(10))

See the full guide: Named Vector Spaces.

Streaming Results

Process results one at a time via callback. Return false to stop early:

err := coll.SearchStream(queryVector, func(r veclite.Result) bool {
    fmt.Printf("ID: %d, Score: %.4f\n", r.Record.ID, r.Score)
    return r.Score > 0.5 // stop when score drops below threshold
}, veclite.TopK(100))
Filtering

Filter search results and find operations by metadata:

// Equality
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilter(veclite.Equal("type", "code")),
)

// Multiple filters (AND)
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilters(
        veclite.Equal("language", "go"),
        veclite.Prefix("file", "src/"),
    ),
)

// Logical operators
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilter(veclite.Or(
        veclite.Equal("lang", "go"),
        veclite.Equal("lang", "rust"),
    )),
)

// Range filters
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilters(
        veclite.GT("score", 0.5),
        veclite.Between("line", 100, 500),
    ),
)

// Find records without vector search
records, _ := coll.Find(veclite.Equal("type", "code"))
record, _ := coll.FindOne(veclite.Equal("file", "main.go"))

All filter functions:

Filter Description
Equal(key, value) Exact match
NotEqual(key, value) Not equal
In(key, values...) Value in list
NotIn(key, values...) Value not in list
Glob(key, pattern) Glob pattern (e.g., *.go)
Prefix(key, prefix) String prefix
Suffix(key, suffix) String suffix
Contains(key, substr) String contains
Exists(key) Key exists in payload
GT(key, value) Greater than (numeric)
GTE(key, value) Greater than or equal
LT(key, value) Less than (numeric)
LTE(key, value) Less than or equal
Between(key, min, max) Value in range (inclusive)
And(filters...) All must match
Or(filters...) Any can match
Not(filter) Negate
Pagination

Use WithOffset and TopK (or WithLimit) to paginate search results:

// Page 1
page1, _ := coll.Search(query, veclite.TopK(10))

// Page 2
page2, _ := coll.Search(query, veclite.TopK(10), veclite.WithOffset(10))

// Page 3
page3, _ := coll.Search(query, veclite.TopK(10), veclite.WithOffset(20))
Iteration

Browse records without vector search:

// Iterator with offset and limit
it := coll.Iterate(veclite.IterOffset(0), veclite.IterLimit(100))
for {
    record, ok := it.Next()
    if !ok {
        break
    }
    fmt.Println(record.ID, record.Payload)
}
it.Close()

// ForEach (return false to stop early)
coll.ForEach(func(r *veclite.Record) bool {
    fmt.Println(r.ID)
    return true // continue
})

// Get all records
all := coll.All()
Upsert

Insert or update records:

// Upsert by ID (0 = generate new ID)
id, err := coll.Upsert(0, vector, payload)      // insert new
id, err := coll.Upsert(42, vector, payload)     // update if exists, insert otherwise

// Upsert by key field (useful for incremental indexing)
id, wasInsert, err := coll.UpsertByKey("file", "main.go", vector, map[string]any{
    "file": "main.go",
    "line": 100,
})
Migrating a Collection Layout

When your application changes its collection layout across versions (e.g. merging two collections into one with a named vector space), use the read-transform-insert-drop pattern with the existing API. There is no built-in RenameCollection because the transformation is almost always application-specific.

// Example: merge a BM25-only collection and a vector collection into one
// collection that uses a named "text" space for vectors.
oldText, _ := db.GetCollection("evidence_text")   // vectors + BM25
oldKeyword, _ := db.GetCollection("evidence_keyword") // BM25 only

merged, _ := db.CreateCollection("evidence",
    veclite.WithTextIndex("evidence_id"),
    veclite.WithVectorSpace(veclite.VectorSpaceConfig{Name: "text", Dimension: dim}),
)

// 1. Copy keyword records (Content + Payload, no vector yet).
for _, r := range oldKeyword.All() {
    merged.InsertRecord(veclite.RecordInput{
        Content: r.Content, Payload: r.Payload,
    })
}

// 2. Attach the matching vector from the text collection by natural key.
for _, r := range oldText.All() {
    key := r.Payload["evidence_id"]
    merged.UpsertRecordByKey("evidence_id", key, veclite.RecordInput{
        Content: r.Content, Payload: r.Payload,
        Vectors: map[string][]float32{"text": r.Vector},
    })
}

// 3. Drop the old collections.
db.DropCollection("evidence_text")
db.DropCollection("evidence_keyword")

This keeps the migration logic in the consumer (where the schema lives) and uses only stable public APIs (GetCollection, All, InsertRecord, UpsertRecordByKey, DropCollection).

Updating Records
// Update payload only (keep vector)
err := coll.Update(id, newPayload)

// Update vector only (keep payload)
err := coll.UpdateVector(id, newVector)
Deleting Records
// Delete by ID
err := coll.Delete(42)

// Delete by filter
count, err := coll.DeleteWhere(veclite.Equal("type", "temp"))

// Clear all records
err := coll.Clear()
Auto-Embedding

Implement the Embedder interface to enable automatic text-to-vector conversion:

type Embedder interface {
    Embed(text string) ([]float32, error)
    EmbedBatch(texts []string) ([][]float32, error)
    Dimension() int
}

Usage:

coll, _ := db.CreateCollection("docs",
    veclite.WithEmbedder(myEmbedder),
)

// Insert text (auto-embeds to vector)
id, err := coll.InsertText("Go is a programming language", payload)

// Search by text (auto-embeds query)
results, err := coll.SearchText("programming languages", veclite.TopK(10))

The Embedder interface lives in the core library. VecLite provides built-in implementations for OpenAI, Ollama, and ONNX.

Embedding Providers

VecLite supports multiple embedding providers out of the box:

Provider Package Dimensions Notes
OpenAI embed/openai 1536/3072 API key required, best quality
Ollama embed/ollama 768/1024/384 Local, no API key needed
ONNX embed/onnx 384 Local, offline capable, build with -tags onnx
OpenAI Embedder

Use OpenAI's embedding API for high-quality embeddings:

import "github.com/abdul-hamid-achik/veclite/embed/openai"

// Create embedder (uses OPENAI_API_KEY env var by default)
embedder, err := openai.NewEmbedder()

// Or with explicit options
embedder, err := openai.NewEmbedder(
    openai.WithAPIKey("sk-..."),
    openai.WithModel("text-embedding-3-small"),  // or text-embedding-3-large
    openai.WithDimension(512),                    // Reduced dimensions (optional)
)
if err != nil {
    log.Fatal(err)
}
defer embedder.Close()

// Use with veclite
db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(embedder.Dimension()),
    veclite.WithEmbedder(embedder),
)

coll.InsertText("Hello world", nil)
results, _ := coll.SearchText("greeting", veclite.TopK(5))

OpenAI Models:

Model Default Dimensions Notes
text-embedding-3-small 1536 Cheapest, good quality (default)
text-embedding-3-large 3072 Best quality
text-embedding-ada-002 1536 Legacy

Options:

Option Description
WithAPIKey(key) Set API key (default: OPENAI_API_KEY env)
WithModel(model) Set model (default: text-embedding-3-small)
WithBaseURL(url) Custom API endpoint (Azure, proxies)
WithDimension(dim) Reduced dimensions for v3 models
WithTimeout(dur) Request timeout (default: 30s)
Ollama Embedder

Use Ollama for local embeddings with no API key required:

import "github.com/abdul-hamid-achik/veclite/embed/ollama"

// Create embedder with defaults (localhost:11434, nomic-embed-text)
embedder, err := ollama.NewEmbedder()

// Or with options
embedder, err := ollama.NewEmbedder(
    ollama.WithBaseURL("http://localhost:11434"),
    ollama.WithModel("nomic-embed-text"),
)
if err != nil {
    log.Fatal(err)
}
defer embedder.Close()

// Use with veclite
db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(embedder.Dimension()),
    veclite.WithEmbedder(embedder),
)

Prerequisites: Pull the model first:

ollama pull nomic-embed-text

Popular Ollama Models:

Model Dimensions Notes
nomic-embed-text 768 Good general purpose (default)
mxbai-embed-large 1024 High quality
all-minilm 384 Fast, lightweight

Options:

Option Description
WithBaseURL(url) Ollama API endpoint (default: http://localhost:11434)
WithModel(model) Embedding model (default: nomic-embed-text)
WithTimeout(dur) Request timeout (default: 30s)
Local ONNX Embedder

VecLite provides an optional local embedding system using ONNX Runtime with the all-MiniLM-L6-v2 model. This enables:

  • Zero external API dependencies -- No Ollama or OpenAI required
  • Offline capability -- Text never leaves your machine
  • Low latency -- ~10ms per embedding vs ~50ms with external services
  • 384-dimensional vectors -- Compatible with most vector search use cases
Quick Start (macOS)

Complete setup in 4 steps:

# 1. Install ONNX Runtime
brew install onnxruntime

# 2. Download libtokenizers
mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

# 3. Download model files (~90MB)
mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx"

# 4. Build with ONNX support
CGO_LDFLAGS="-L./lib" go build -tags onnx ./...
Installation Details

The ONNX embedder requires native libraries and model files.

Step 1: Install ONNX Runtime

Platform Command
macOS (Homebrew) brew install onnxruntime
Linux Download from onnxruntime releases
Windows Download from onnxruntime releases

Step 2: Install libtokenizers

The github.com/daulet/tokenizers package requires the libtokenizers native library.

macOS ARM64 (Apple Silicon):

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

macOS Intel:

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-amd64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

Linux:

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.linux-amd64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

See github.com/daulet/tokenizers for other platforms.

Step 3: Download Model Files

Option A - Using curl (~90MB):

mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx"

Option B - Quantized model (~25MB, slightly lower quality):

mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model_quantized.onnx"

Option C - Using Go (requires native libs installed first):

import "github.com/abdul-hamid-achik/veclite/embed/onnx"

// Full model
err := onnx.DownloadMiniLM("./models")

// Or quantized
err := onnx.DownloadMiniLMQuantized("./models")

Step 4: Build

# Set library path and build
CGO_LDFLAGS="-L./lib" go build -tags onnx ./...
Running Tests
# Run all ONNX tests
CGO_LDFLAGS="-L./lib" VECLITE_ONNX_MODEL_DIR=./models \
  go test -tags onnx -v ./embed/onnx/...

# Run benchmarks
CGO_LDFLAGS="-L./lib" VECLITE_ONNX_MODEL_DIR=./models \
  go test -tags onnx -bench=. -benchmem ./embed/onnx/...
Usage
import (
    "github.com/abdul-hamid-achik/veclite"
    "github.com/abdul-hamid-achik/veclite/embed/onnx"
)

// Create ONNX embedder
embedder, err := onnx.NewMiniLM("./models")
if err != nil {
    log.Fatal(err)
}
defer embedder.Close()

// Use with veclite
db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(384),  // MiniLM outputs 384 dimensions
    veclite.WithEmbedder(embedder),
)

// Insert and search by text
id, _ := coll.InsertText("Go is a statically typed language", nil)
results, _ := coll.SearchText("programming languages", veclite.TopK(5))
Batch Embedding

For better performance with multiple texts, use batch embedding:

texts := []string{
    "First document",
    "Second document",
    "Third document",
}

vectors, err := embedder.EmbedBatch(texts)
// Then insert with InsertBatch
Performance

Benchmarks on Apple M5:

Operation Time Throughput
Single embed ~12ms ~83 texts/sec
Batch 10 ~100ms ~100 texts/sec
Batch 100 ~875ms ~114 texts/sec

Batching improves throughput by ~37% compared to single embedding.

Custom Models

You can use other ONNX-exported sentence transformers:

embedder, err := onnx.NewEmbedder(
    "/path/to/model.onnx",
    "/path/to/tokenizer.json",
    onnx.WithDimension(768),    // For larger models
    onnx.WithMaxLength(512),    // Max token sequence length
)
Troubleshooting

"library 'tokenizers' not found"

# Ensure CGO_LDFLAGS points to the lib directory
CGO_LDFLAGS="-L/full/path/to/lib" go build -tags onnx ./...

"onnxruntime.so not found"

# The library auto-detects common paths. If not found, set manually:
export ONNXRUNTIME_LIB=/opt/homebrew/lib/libonnxruntime.dylib

"model.onnx not found"

# Ensure VECLITE_ONNX_MODEL_DIR points to the models directory
VECLITE_ONNX_MODEL_DIR=/full/path/to/models go test -tags onnx ./embed/onnx/...
Config-Based Embedding

Configure embedding providers via veclite.yaml for easy switching between providers:

# veclite.yaml
embedder:
  provider: openai  # openai | ollama | onnx

  openai:
    api_key: ${OPENAI_API_KEY}  # Supports env var expansion
    model: text-embedding-3-small
    dimension: 1536
    timeout: 30s

  ollama:
    base_url: http://localhost:11434
    model: nomic-embed-text
    timeout: 30s

  onnx:
    model_dir: ~/.veclite/models

Usage:

import "github.com/abdul-hamid-achik/veclite"

// Load config (searches ./veclite.yaml, ~/.veclite/config.yaml)
cfg, err := veclite.LoadConfig("")

// Or from explicit path
cfg, err := veclite.LoadConfig("/path/to/veclite.yaml")

// Create embedder from config
embedder, err := veclite.NewEmbedderFromConfig(cfg.Embedder)
if err != nil {
    log.Fatal(err)
}
defer embedder.(interface{ Close() error }).Close()

// Use with veclite
db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(embedder.Dimension()),
    veclite.WithEmbedder(embedder),
)

Environment Variable Expansion:

The config supports ${VAR} and ${VAR:-default} syntax:

embedder:
  provider: openai
  openai:
    api_key: ${OPENAI_API_KEY}           # Required env var
    base_url: ${OPENAI_BASE_URL:-https://api.openai.com/v1}  # With default

Config Search Order:

  1. Explicit path provided to LoadConfig()
  2. ./veclite.yaml (current directory)
  3. ~/.veclite/config.yaml (home directory)
  4. Returns default configuration if no file found
HNSW Configuration

HNSW (Hierarchical Navigable Small World) provides approximate nearest neighbor search:

// Basic
coll, _ := db.CreateCollection("vectors",
    veclite.WithHNSW(16, 200),
)

// Full configuration
coll, _ := db.CreateCollection("vectors",
    veclite.WithHNSWConfig(veclite.HNSWConfig{
        M:              32,
        EfConstruction: 400,
        EfSearch:       100,
    }),
)
Parameter Default Range Trade-off
M 16 12-48 Higher = better recall, more memory
EfConstruction 200 100-500 Higher = better index quality, slower build
EfSearch 100 50-500 Higher = better recall, slower search

Without HNSW, search uses brute-force linear scan (exact results, slower on large datasets).

Observability
Logger

Implement the Logger interface to integrate with your logging library:

type Logger interface {
    Debug(msg string, keysAndValues ...any)
    Info(msg string, keysAndValues ...any)
    Error(msg string, keysAndValues ...any)
}

NopLogger is the default (zero overhead). Set a logger when opening the database:

db, _ := veclite.Open("data.veclite", veclite.WithLogger(myLogger))
Metrics

VecLite tracks operation counts and latency using atomic counters:

snapshot := db.Metrics()
fmt.Printf("Searches: %d, Inserts: %d, Deletes: %d, Avg Search: %v\n",
    snapshot.SearchCount,
    snapshot.InsertCount,
    snapshot.DeleteCount,
    snapshot.AvgSearchTime,
)

MetricsSnapshot fields:

Field Type Description
SearchCount int64 Total search operations
InsertCount int64 Total insert operations
DeleteCount int64 Total delete operations
AvgSearchTime time.Duration Average search latency
Statistics
// Database stats
dbStats := db.Stats()
fmt.Printf("Collections: %d, Total Records: %d\n",
    dbStats.Collections, dbStats.TotalRecords)

// Collection stats
collStats := coll.Stats()
fmt.Printf("Count: %d, Dimension: %d, Index: %s\n",
    collStats.Count, collStats.Dimension, collStats.IndexType)

Agent Memory Features

VecLite includes features designed for AI agent memory systems, enabling intelligent storage, retrieval, and management of memories with temporal awareness, importance scoring, and relationship tracking.

TTL and Expiration

Records can have a time-to-live (TTL) and expire automatically:

// Insert with TTL
id, err := coll.InsertWithOptions(vector, payload,
    veclite.WithTTL(24 * time.Hour),  // Expires in 24 hours
)

// Insert with explicit expiration time
id, err := coll.InsertWithOptions(vector, payload,
    veclite.WithExpiresAt(time.Now().Add(7 * 24 * time.Hour)),
)

// Check if record is expired
record, _ := coll.Get(id)
if record.IsExpired() {
    fmt.Println("Record has expired")
}
if record.HasTTL() {
    fmt.Printf("TTL remaining: %v\n", record.TTL())
}

// Clean up expired records (manual)
count, err := coll.CleanupExpired()
fmt.Printf("Removed %d expired records\n", count)

// Count expired without removing
expiredCount := coll.CountExpired()

// Automatic background cleanup
stop := db.StartTTLCleaner(5*time.Minute, func(collection string, deleted int) {
    fmt.Printf("Cleaned %d expired records from %s\n", deleted, collection)
})
defer stop()  // Stop the cleaner when done
Importance and Decay Scoring

Assign importance scores to records and apply temporal decay to search results:

// Insert with importance score (0.0 to 1.0)
id, err := coll.InsertWithOptions(vector, payload,
    veclite.WithImportance(0.9),  // High importance
)

// Search with importance boost
results, err := coll.Search(query,
    veclite.TopK(10),
    veclite.WithImportanceBoost(1.5),  // Multiply score by importance * factor
)

// Apply temporal decay to search results
results, err := coll.Search(query,
    veclite.TopK(10),
    veclite.WithDecay(veclite.DecayExponential, 24*time.Hour),  // Half-life of 24 hours
)

// Enable access tracking (updates LastAccessedAt and AccessCount)
results, err := coll.Search(query,
    veclite.TopK(10),
    veclite.WithAccessTracking(true),
)

Decay types:

Type Behavior
DecayNone No decay applied
DecayExponential Score halves every half-life period
DecayLinear Score decreases linearly over time
DecayGaussian Bell curve decay centered at creation time
Timestamp Filters

Filter records by creation time, update time, access time, and expiration:

// Time-based filters
results, _ := coll.Search(query,
    veclite.WithFilter(veclite.CreatedAfter(time.Now().Add(-24*time.Hour))),
)

records, _ := coll.Find(
    veclite.AgeNewerThan(1 * time.Hour),      // Created within last hour
    veclite.UpdatedAfter(yesterday),           // Modified since yesterday
)

// TTL and expiration filters
activeRecords, _ := coll.Find(veclite.NotExpired())
expiringRecords, _ := coll.Find(veclite.ExpiredBefore(time.Now().Add(time.Hour)))
hasExpiration, _ := coll.Find(veclite.HasTTLFilter())

// Importance filters
important, _ := coll.Find(veclite.ImportanceAbove(0.7))
lowPriority, _ := coll.Find(veclite.ImportanceBelow(0.3))
midRange, _ := coll.Find(veclite.ImportanceBetween(0.4, 0.6))

// Access tracking filters
recentlyAccessed, _ := coll.Find(veclite.AccessedAfter(time.Now().Add(-time.Hour)))
neverAccessed, _ := coll.Find(veclite.NeverAccessed())
frequentlyAccessed, _ := coll.Find(veclite.AccessCountAbove(10))
Conversation Memory

Track conversation turns with session and thread support:

// Insert a conversation turn
id, err := coll.InsertTurn(veclite.ConversationTurn{
    SessionID:  "session-123",
    TurnNumber: 1,
    Role:       "user",
    Content:    "Hello, how are you?",
    Vector:     vector,  // Or use embedder
    Importance: 0.5,
    TTL:        24 * time.Hour,
})

// Insert a reply (threaded)
replyID, err := coll.InsertTurn(veclite.ConversationTurn{
    SessionID:     "session-123",
    TurnNumber:    2,
    Role:          "assistant",
    Content:       "I'm doing well, thank you!",
    Vector:        vector,
    ParentChunkID: id,  // Links to parent
})

// Get all turns in a session (ordered by turn number)
turns, err := coll.GetSession("session-123")

// Get a conversation thread (follows parent-child links)
thread, err := coll.GetThread(id)

// Search within a specific session
results, err := coll.SearchInSession("session-123", query, veclite.TopK(5))

// List all sessions and get stats
sessions := coll.ListSessions()
stats, err := coll.GetSessionStats("session-123")
fmt.Printf("Turns: %d, Roles: %v\n", stats.TurnCount, stats.Roles)
Subscriptions (Real-time Notifications)

Subscribe to be notified when new records match a query:

// Subscribe to matching records
sub, err := coll.Subscribe(
    queryVector,
    veclite.WithSubscriptionThreshold(0.8),    // Minimum similarity
    veclite.WithSubscriptionFilter(veclite.Equal("type", "important")),
    veclite.WithSubscriptionBufferSize(100),   // Event buffer size
)
defer sub.Close()

// Listen for matching records (non-blocking)
go func() {
    for event := range sub.Events() {
        fmt.Printf("New match: ID=%d, Score=%.4f\n",
            event.Record.ID, event.Score)
    }
}()

// Insert triggers notifications automatically
coll.Insert(similarVector, map[string]any{"type": "important"})

// Unsubscribe when done
coll.Unsubscribe(sub.ID)
Memory Consolidation

Cluster similar memories and consolidate them into summaries:

// Find clusters of similar memories
clusters, err := coll.FindSimilarClusters(veclite.ConsolidationConfig{
    SimilarityThreshold: 0.9,   // How similar records must be
    MinGroupSize:        3,     // Minimum cluster size
    MaxGroupSize:        10,    // Maximum cluster size
    Filters:             []veclite.Filter{veclite.NotExpired()},
})

for _, cluster := range clusters {
    fmt.Printf("Cluster %s: %d records, avg importance: %.2f\n",
        cluster.ID, len(cluster.Records), cluster.AverageImportance)
}

// Consolidate clusters into summary records
result, err := coll.Consolidate(veclite.ConsolidationConfig{
    SimilarityThreshold: 0.9,
    MinGroupSize:        3,
    ArchiveOriginals:    true,  // Archive source records
    Embedder:            embedder,
    SummaryGenerator: func(records []*veclite.Record) (string, map[string]any, error) {
        // Generate summary from records (e.g., using LLM)
        summary := "Summary of " + strconv.Itoa(len(records)) + " memories"
        return summary, map[string]any{"source": "consolidation"}, nil
    },
})

fmt.Printf("Consolidated %d records into %d summaries\n",
    result.RecordsConsolidated, len(result.ConsolidatedRecordIDs))

// Archive/unarchive records manually
coll.ArchiveRecord(id)
coll.UnarchiveRecord(id)
archived, _ := coll.GetArchived()

// Get all consolidation records
consolidations, _ := coll.GetConsolidations()

// Expand a consolidation to see original records
originals, _ := coll.ExpandConsolidation(consolidationID)
Episodic Memory

Group related memories into coherent episodes:

// Create an episode store for a collection
episodeStore, err := db.CreateEpisodeStore("memories")

// Manually create an episode from record IDs
episode, err := episodeStore.CreateEpisode(
    []uint64{id1, id2, id3},
    "Morning standup meeting",
)

// Automatically detect episodes based on time gaps
episodes, err := episodeStore.DetectEpisodes(veclite.EpisodeConfig{
    TimeGapThreshold: 30 * time.Minute,  // Gap between episodes
    MinRecords:       2,                  // Minimum records per episode
    MaxRecords:       100,                // Maximum records per episode
})

// Get episode details
episode, _ := episodeStore.GetEpisode(episodeID)
fmt.Printf("Episode: %s, Duration: %v, Records: %d\n",
    episode.Title, episode.Duration(), episode.RecordCount())

// Expand an episode to get all its records
records, _ := episodeStore.ExpandEpisode(episodeID)

// Search with episode expansion (includes context from same episode)
results, err := episodeStore.SearchWithEpisodeExpansion(query, veclite.TopK(10))
for _, r := range results {
    if r.Episode != nil {
        fmt.Printf("Match in episode: %s (%d related records)\n",
            r.Episode.Title, len(r.EpisodeRecords))
    }
}

// Search for similar episodes
episodes, _ := episodeStore.SearchEpisodes(query, 5)

// Find which episode contains a record
episode, _ := episodeStore.FindRecordEpisode(recordID)

// List and delete episodes
allEpisodes := episodeStore.ListEpisodes()
episodeStore.DeleteEpisode(episodeID)
Memory Pressure Handling

Manage collection size with automatic eviction policies:

// Configure memory limits when creating a collection
coll, _ := db.CreateCollection("memories",
    veclite.WithDimension(384),
    veclite.WithMemoryLimits(veclite.MemoryConfig{
        MaxRecords:        10000,           // Maximum records allowed
        EvictionPolicy:    "importance",    // "fifo", "lru", or "importance"
        EvictionBatchSize: 100,             // Records to evict per cycle
        CleanupInterval:   5 * time.Minute, // Background check interval
    }),
)

// Or enforce limits manually after inserts
evicted := coll.EnforceMemoryLimit(veclite.MemoryConfig{
    MaxRecords:     5000,
    EvictionPolicy: "fifo",  // Remove oldest records first
})
fmt.Printf("Evicted %d records\n", evicted)

// Start a background memory limiter
stop := coll.StartMemoryLimiter(veclite.MemoryConfig{
    MaxRecords:        10000,
    EvictionPolicy:    "lru",        // Remove least recently accessed
    CleanupInterval:   time.Minute,
    EvictionBatchSize: 50,
})
defer stop()

Eviction Policies:

Policy Behavior
fifo Remove oldest records first (by creation time)
lru Remove least recently accessed records first
importance Remove lowest importance records first

Archived records (via ArchiveRecord) are never evicted regardless of policy.

Knowledge Graph

Build entity-relationship graphs with vector search:

// Create a knowledge graph
kg, err := db.CreateKnowledgeGraph("knowledge")

// Add entities with vectors
kg.AddEntity(veclite.Entity{
    ID:     "alice",
    Type:   "person",
    Name:   "Alice Smith",
    Vector: aliceVector,
    Properties: map[string]any{
        "role": "engineer",
        "team": "backend",
    },
})

kg.AddEntity(veclite.Entity{
    ID:     "acme",
    Type:   "company",
    Name:   "Acme Corp",
    Vector: acmeVector,
})

// Add relationships
kg.AddRelationship(veclite.Relationship{
    ID:       "rel-1",
    SourceID: "alice",
    TargetID: "acme",
    Type:     "works_at",
    Weight:   0.9,
})

kg.AddRelationship(veclite.Relationship{
    ID:            "rel-2",
    SourceID:      "alice",
    TargetID:      "bob",
    Type:          "knows",
    Weight:        0.8,
    Bidirectional: true,  // Creates edges in both directions
})

// Traverse the graph
result, err := kg.Traverse([]string{"alice"}, veclite.TraversalConfig{
    MaxDepth:          2,                       // How far to traverse
    MaxNodes:          100,                     // Maximum nodes to visit
    MinWeight:         0.5,                     // Minimum relationship weight
    RelationshipTypes: []string{"knows"},       // Filter by relationship type
    EntityTypes:       []string{"person"},      // Filter by entity type
    Direction:         "both",                  // "outgoing", "incoming", or "both"
})

for _, entity := range result.Entities {
    fmt.Printf("Found: %s (depth %d)\n", entity.Name, result.Depths[entity.ID])
}

// Search with graph expansion
results, err := kg.SearchWithExpansion(query,
    veclite.TraversalConfig{MaxDepth: 1},
    veclite.TopK(10),
)

for _, r := range results {
    fmt.Printf("Entity: %s (score: %.4f)\n", r.Entity.Name, r.Score)
    fmt.Printf("  Related: %d entities via %d relationships\n",
        len(r.RelatedEntities), len(r.Relationships))
}

// Get graph statistics
stats := kg.Stats()
fmt.Printf("Entities: %d, Relationships: %d\n",
    stats.EntityCount, stats.RelationshipCount)
fmt.Printf("Entity types: %v\n", stats.EntityTypes)
fmt.Printf("Relationship types: %v\n", stats.RelationshipTypes)

// Entity and relationship management
entity, _ := kg.GetEntity("alice")
kg.UpdateEntity(updatedEntity)
kg.DeleteEntity("alice")  // Also removes relationships

rel, _ := kg.GetRelationship("rel-1")
kg.DeleteRelationship("rel-1")

// Get relationships for an entity
outgoing := kg.GetRelationships("alice", "outgoing")
incoming := kg.GetRelationships("alice", "incoming")
all := kg.GetRelationships("alice", "both")

CLI Usage

veclite <command> [arguments]
Commands
Read Commands
Command Description
version Show version information
info <file> Show database summary, including WAL sidecar status
collections <file> List all collections
stats <file> Show detailed statistics
dump <file> Export database as JSON
get <file> <collection> Get a vector by ID
Write Commands
Command Description
create-collection <file> <name> Create a new collection
drop-collection <file> <name> Drop a collection
insert <file> <collection> Insert a vector
batch-insert <file> <collection> Insert vectors from JSON file
delete <file> <collection> Delete a vector by ID
upsert <file> <collection> Insert or update a vector
update <file> <collection> Update payload for a record
delete-where <file> <collection> Delete records matching a filter
find <file> <collection> Find records by filter
search <file> <collection> Search for similar vectors
Named Vector Spaces
Command Description
spaces <file> <collection> List a collection's vector spaces
space-add <file> <collection> Declare a named vector space (--name, --dim, --distance, --modality, --hnsw)
record-insert <file> <collection> Insert a record with vectors in several spaces (--vectors, --input)
record-upsert-by-key <file> <collection> Insert or replace a multi-space record by a payload key (--key-field, --key-value, --vectors)
search-space <file> <collection> <space> Search a single named vector space
hybrid-search-space <file> <collection> <space> Hybrid vector+BM25 search over a named space (--query, --text)
fuse-search <file> <collection> Search several spaces and fuse with RRF (--queries)
Server Mode
Command Description
serve <file> Start HTTP server
mcp <file> Start MCP tool server over stdio
Maintenance
Command Description
compact <file> Compact database and reclaim space
validate <file> Validate database integrity
benchmark <file> Run search performance benchmark

All commands support --json for JSON output.

CLI Examples
# Create a collection with HNSW index and text search
veclite create-collection data.veclite embeddings \
    --dimension=384 --distance=cosine --hnsw --text-index=title,body

# Insert a vector
veclite insert data.veclite embeddings \
    --vector='[0.1,0.2,0.3,...]' --payload='{"file":"main.go"}'

# Batch insert from JSON
veclite batch-insert data.veclite embeddings --input=vectors.json

# Search
veclite search data.veclite embeddings \
    --query='[0.1,0.2,0.3,...]' --top-k=10 --filter='type=code'

# Named vector spaces: declare a space, insert a multi-space record, search and fuse
veclite space-add data.veclite items --name=image --dim=512 --modality=image --hnsw
veclite record-insert data.veclite items \
    --vectors='{"default":[0.1,0.2],"image":[0.3,0.4]}' --content='a red apple'
veclite spaces data.veclite items --json
veclite search-space data.veclite items image --query='[0.3,0.4]' --top-k=5
veclite fuse-search data.veclite items \
    --queries='{"default":[0.1,0.2],"image":[0.3,0.4]}' --top-k=10

# Named spaces: upsert by a payload key (idempotent), then hybrid search over a named space
veclite record-upsert-by-key data.veclite evidence \
    --key-field=evidence_id --key-value=doc-1 \
    --vectors='{"text":[0.1,0.2,0.3]}' --content='checkout fails'
veclite hybrid-search-space data.veclite evidence text \
    --query='[0.1,0.2,0.3]' --text='checkout' --top-k=5

# Upsert
veclite upsert data.veclite embeddings \
    --id=42 --vector='[0.1,0.2,0.3,...]' --payload='{"file":"main.go"}'

# Find by filter
veclite find data.veclite embeddings --filter='type=code'

# Delete by filter
veclite delete-where data.veclite embeddings --filter='type=temp'

# Start HTTP server
veclite serve data.veclite --port=8080 --cors

# Start MCP server
veclite mcp data.veclite
Batch Insert File Format

JSON Array:

[
  {"vector": [0.1, 0.2, 0.3], "payload": {"file": "a.go"}},
  {"vector": [0.4, 0.5, 0.6], "payload": {"file": "b.go"}}
]

JSONL (one object per line):

{"vector": [0.1, 0.2, 0.3], "payload": {"file": "a.go"}}
{"vector": [0.4, 0.5, 0.6], "payload": {"file": "b.go"}}

HTTP Server

veclite serve data.veclite --port=8080 --host=127.0.0.1 --cors

# Long-running servers should enable the write-ahead log so accepted writes
# survive a crash between syncs (see "Durability and the Write-Ahead Log").
veclite serve data.veclite --wal
Endpoints
Method Endpoint Description
GET /health Health check
GET /info Database info
GET /metrics Operation metrics
GET /collections List collections
POST /collections Create collection
GET /collections/{name} Collection info
DELETE /collections/{name} Drop collection
GET /collections/{name}/vectors List all vectors (paginated)
POST /collections/{name}/vectors Insert vector(s)
GET /collections/{name}/vectors/{id} Get vector by ID
PUT /collections/{name}/vectors/{id} Update vector and/or payload
DELETE /collections/{name}/vectors/{id} Delete vector
DELETE /collections/{name}/vectors Delete by filter (with body)
POST /collections/{name}/search Search vectors
GET /collections/{name}/spaces List vector spaces
POST /collections/{name}/spaces Add a named vector space
POST /collections/{name}/records Insert a multi-space record
POST /collections/{name}/records-upsert-by-key Upsert a multi-space record by a payload key
POST /collections/{name}/search-space Search one named vector space
POST /collections/{name}/hybrid-search-space Hybrid vector+BM25 search over a named space
POST /collections/{name}/fuse-search Fuse search across vector spaces
POST /collections/{name}/upsert Upsert vector
POST /collections/{name}/find Find records by filter
POST /collections/{name}/compact Compact collection
POST /collections/{name}/validate Validate integrity
POST /sync Force sync to disk
POST /reload Reload database from disk (pick up external writes)
Streaming Results

Set Accept: application/x-ndjson on search requests to receive results as newline-delimited JSON:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Accept: application/x-ndjson" \
  -H "Content-Type: application/json" \
  -d '{"query": [0.1, 0.2, 0.3], "top_k": 100}'

Each line is a JSON object:

{"id":1,"score":0.9821,"payload":{"file":"main.go"}}
{"id":5,"score":0.9654,"payload":{"file":"util.go"}}
API Examples

Create Collection:

curl -X POST http://localhost:8080/collections \
  -H "Content-Type: application/json" \
  -d '{"name": "embeddings", "dimension": 384, "distance": "cosine", "hnsw": true}'

Insert Vector:

curl -X POST http://localhost:8080/collections/embeddings/vectors \
  -H "Content-Type: application/json" \
  -d '{"vector": [0.1, 0.2, 0.3], "payload": {"file": "main.go"}}'

Batch Insert:

curl -X POST http://localhost:8080/collections/embeddings/vectors \
  -H "Content-Type: application/json" \
  -d '{"vectors": [[0.1,0.2,0.3], [0.4,0.5,0.6]], "payloads": [{"file":"a.go"}, {"file":"b.go"}]}'

Search:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Content-Type: application/json" \
  -d '{"query": [0.1, 0.2, 0.3], "top_k": 10}'

Search with Filters:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "top_k": 10,
    "filters": [{"key": "type", "op": "eq", "value": "code"}]
  }'

Upsert:

curl -X POST http://localhost:8080/collections/embeddings/upsert \
  -H "Content-Type: application/json" \
  -d '{"id": 42, "vector": [0.1, 0.2, 0.3], "payload": {"file": "main.go"}}'

Find by Filter:

curl -X POST http://localhost:8080/collections/embeddings/find \
  -H "Content-Type: application/json" \
  -d '{"filters": [{"key": "type", "op": "eq", "value": "code"}]}'

Update Vector:

curl -X PUT http://localhost:8080/collections/embeddings/vectors/42 \
  -H "Content-Type: application/json" \
  -d '{"vector": [0.4, 0.5, 0.6], "payload": {"file": "updated.go"}}'
Filter Operators
Operator Aliases Description
eq = Equal
neq != Not equal
gt > Greater than (numeric)
gte >= Greater than or equal
lt < Less than (numeric)
lte <= Less than or equal
glob Glob pattern match
prefix String prefix
suffix String suffix
contains String contains
exists Key exists
Python Client Example
import requests

base = "http://localhost:8080"

# Create collection
requests.post(f"{base}/collections", json={
    "name": "embeddings", "dimension": 384, "hnsw": True
})

# Insert
r = requests.post(f"{base}/collections/embeddings/vectors", json={
    "vector": [0.1] * 384,
    "payload": {"file": "main.py"}
})
print(r.json())  # {"status": "inserted", "id": 1}

# Search
r = requests.post(f"{base}/collections/embeddings/search", json={
    "query": [0.1] * 384, "top_k": 5
})
print(r.json())  # {"results": [...], "count": 5}
Go Client

For Go consumers that need multi-process access, use the client package to talk to a running veclite serve instance. The API mirrors the embedded library so you can swap between veclite.Open(path) (single-process) and client.Open(url) (multi-process) with minimal code change:

import "github.com/abdul-hamid-achik/veclite/client"

// Instead of:  db, _ := veclite.Open("data.veclite")
// Use:          db, _ := client.Open("http://localhost:8080")

db, err := client.Open("http://localhost:8080")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

// Create a collection
coll, err := db.CreateCollection("docs",
    client.WithDimension(384),
    client.WithHNSW(16, 200),
)

// Insert
id, err := coll.Insert([]float32{0.1, 0.2, 0.3}, map[string]any{"source": "wiki"})

// Search
results, err := coll.Search([]float32{0.1, 0.2, 0.3}, client.TopK(10))

// Find by filter
records, err := coll.Find(client.Equal("source", "wiki"))

// Upsert by key
id, inserted, err := coll.UpsertByKey("file", "main.go", vec, payload)

// Sync and reload
db.Sync()   // force write to disk
db.Reload() // pick up external writes

The client supports the same filter operators (Equal, GT, Glob, Prefix, ...), search options (TopK, Threshold, WithFilter), and collection options (WithDimension, WithHNSW, WithTextIndex) as the embedded library.

See the Go client API reference for the full surface.

MCP Tool Server

VecLite can run as an MCP tool server, making it available to AI agents like Claude Code, Cursor, and other MCP-compatible clients.

veclite mcp data.veclite

The server communicates over stdio using JSON-RPC 2.0. It exposes 56 tools across several categories:

Core Vector Operations
Tool Description
veclite_collections List all collections with stats
veclite_stats Get database statistics
veclite_search Vector similarity search
veclite_text_search BM25 full-text search
veclite_hybrid_search Combined vector + text search with RRF fusion
veclite_find Find records by filter
veclite_insert Insert a vector with optional payload and content
veclite_get Retrieve a record by ID
veclite_delete Delete a record by ID
veclite_update Update a record's payload
veclite_upsert Insert or update by ID
veclite_delete_where Delete records matching filter conditions
veclite_clear Clear all records from a collection (requires confirm: true)
veclite_insert_batch Bulk insert multiple vectors
veclite_upsert_by_key Insert or update by payload key field
veclite_embed Convert text to vector using configured embedder
Collection Management
Tool Description
veclite_create_collection Create a collection with options (dimension, distance, index)
veclite_drop_collection Delete a collection (requires confirm: true)
veclite_collection_schema Discover a collection's schema: payload fields, types, vector dimension, index type, content availability
veclite_sync Force persist all changes to disk
veclite_metrics Get performance metrics (search/insert/delete counts)
Agent Memory Tools
Tool Description
memory_remember Store a memory with importance, tags, and TTL
memory_recall Semantic search for memories with filters
memory_forget Remove memories by criteria
memory_enforce_limit Enforce memory limit with eviction policy (fifo/lru/importance)
memory_consolidate Find similar memory clusters
memory_expand_consolidation Get original records from a consolidation
Knowledge Graph Tools
Tool Description
graph_add_entity Add an entity node with optional vector
graph_add_relationship Add a relationship edge between entities
graph_get_entity Get an entity by ID
graph_update_entity Update an existing entity
graph_delete_entity Delete an entity and its relationships
graph_get_relationships Get relationships for an entity
graph_delete_relationship Delete a relationship by ID
graph_list_entities List entities, optionally filtered by type
graph_traverse BFS traversal from starting entities
graph_expanded_search Vector search with graph context expansion
Conversation Memory Tools
Tool Description
conversation_add_turn Add a conversation turn with session tracking
conversation_get_session Get all turns in a session
conversation_search_session Search within a specific session
conversation_list_sessions List all session IDs
conversation_get_thread Get a conversation thread by chunk ID
conversation_delete_session Delete all turns in a session (requires confirm: true)
conversation_get_stats Get session statistics (turn count, roles, duration)
Episodic Memory Tools
Tool Description
episode_detect Auto-detect episodes using time gaps and similarity
episode_create Create an episode from record IDs
episode_get Get episode details including records
episode_list List all episodes in a collection
episode_search Search episodes by vector similarity
episode_search_expanded Search with episode context expansion
Memory Consolidation Tools
Tool Description
memory_find_clusters Find clusters of similar memories
memory_archive Archive a memory (excludes from searches, protects from eviction)
memory_unarchive Restore an archived memory
memory_get_archived List archived memories
TTL/Cleanup Tools
Tool Description
veclite_cleanup_expired Remove all expired records from a collection
veclite_count_expired Count expired records without removing them
MCP Configuration

Add to your MCP client configuration (e.g., .claude/settings.json):

{
  "mcpServers": {
    "veclite": {
      "command": "veclite",
      "args": ["mcp", "/path/to/data.veclite"]
    }
  }
}

Examples

Runnable examples are in the examples/ directory:

go run ./examples/basic        # Open, insert, search, close
go run ./examples/hnsw         # HNSW index configuration and benchmarking
go run ./examples/filtering    # Rich filter expressions
go run ./examples/batch        # Batch operations, upsert, iteration, pagination
go run ./examples/http-client  # HTTP API client (start server first)

All examples use :memory: for zero-setup running (except http-client, which connects to a running server).

Performance

Benchmark results on 10,000 384-dimensional vectors:

Method Time Speedup
Brute Force ~2.5ms 1x
HNSW ~0.4ms 6x

HNSW provides >95% recall at 6-7x speedup over brute force.

Thread Safety

VecLite is safe for concurrent access:

  • Multiple goroutines can read simultaneously
  • Writes are serialized with sync.RWMutex
  • Metrics use atomic counters for lock-free reads
  • Use WithSyncOnWrite(true) for durability after each write
Multi-Process Access

By default, VecLite uses an exclusive file lock for writers — only one process at a time can open the database read-write. This prevents data corruption from concurrent writes (VecLite is an in-memory DB that persists by rewriting the entire snapshot file via an atomic replace).

For multi-process read access, use WithSharedRead(true) together with WithReadOnly(true). Read-only opens are lock-free: they take no flock at all, so a long-lived reader (an MCP server, a daemon's query client, a CLI search) never blocks a writer, and a writer's exclusive lock never blocks a read-only open. Consistency is guaranteed by the writer's atomic-replace save (.tmp → rename): a reader's os.Open always resolves to a complete old or new snapshot, never a torn write.

// Writer process — exclusive lock (only one writer at a time)
db, err := veclite.Open("data.veclite",
    veclite.WithSyncOnWrite(true),
)

// Reader process(es) — lock-free, no conflict with the writer
reader, err := veclite.Open("data.veclite",
    veclite.WithReadOnly(true),
    veclite.WithSharedRead(true),
)

defer reader.Close()

// Readers see a point-in-time snapshot taken at Open.
// Call Reload() to pick up changes written by other processes:
if err := reader.Reload(); err != nil {
    log.Fatal(err)
}

Key points:

  • Writer holds an exclusive lock (LOCK_EX) — only one writer at a time; a second writer gets ErrFileLocked
  • Readers with WithSharedRead(true) take no lock — they never block a writer and are never blocked by one
  • Readers see a point-in-time snapshot from when they opened; they do not auto-detect changes
  • Call db.Reload() to discard the in-memory state and re-load the latest snapshot from disk
  • Reload() rebuilds all collections, HNSW indexes, BM25 indexes, and knowledge graphs
  • All read-only CLI commands (veclite search, veclite stats, etc.) already use WithSharedRead(true)
  • The veclite serve HTTP server remains the canonical multi-client surface for write-heavy workloads
  • For multi-process write access, run veclite serve and use the Go client from other processes

Persistence

Data is stored using Go's gob encoding in a single .veclite file:

  • Call db.Sync() to persist changes manually
  • Use WithSyncOnWrite(true) for automatic persistence after each write
  • db.Close() syncs before closing
  • HNSW indexes, BM25 inverted indexes, and record content are all persisted
  • File locking prevents concurrent writer access; WithSharedRead(true) enables multi-process read-only access
  • CRC32 checksums validate data integrity on load
  • New snapshots and lock files use owner-only permissions; a save also tightens legacy broad database modes while preserving an already stricter owner-only mode

Contributing

Contributions welcome. Please:

  1. Run go test -race ./... before submitting
  2. Add tests for new features
  3. Follow existing code style

License

MIT License - see LICENSE

Documentation

Overview

Package veclite provides an embeddable vector database for Go applications.

VecLite stores vectors, text documents, and metadata in a single file, supports HNSW indexing for fast approximate nearest-neighbor search, and keeps the core storage and search APIs local-first. Optional integrations provide embedders, config, and MCP tooling.

Quick Start

Open a database, create a collection, insert vectors, and search: 

db, err := veclite.Open("data.veclite")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

coll := db.Collection("embeddings")
id, err := coll.Insert(vector, map[string]any{"text": "hello world"})
results, err := coll.Search(queryVector, veclite.TopK(10))

Collections

Collections are namespaced containers for vectors. Create them with options: 

coll, err := db.CreateCollection("docs",
    veclite.WithDimension(384),
    veclite.WithHNSW(16, 200),
    veclite.WithDistanceType(veclite.DistanceCosine),
)

Vector search supports filtering, pagination, and hybrid vector+text search: 

results, err := coll.Search(query,
    veclite.TopK(20),
    veclite.WithFilter(veclite.Equal("category", "science")),
    veclite.Threshold(0.7),
)

Text-only documents can be stored for BM25-first workflows: 

id, err := coll.InsertTextDocument("searchable text", map[string]any{"source": "timeline"})

Persistence

Data is persisted to a single file using gob encoding with atomic writes. Use ":memory:" for an in-memory database: 

db, err := veclite.Open(":memory:")

Thread Safety

All operations on DB and Collection are safe for concurrent use. Multiple goroutines can read and write simultaneously.

Package veclite provides an embeddable vector database for Go. It stores vectors with metadata in a single file using gob encoding.

Basic usage: 

db, err := veclite.Open("data.veclite")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

coll := db.Collection("embeddings")
id, err := coll.Insert(vector, map[string]any{"file": "main.go"})

results, err := coll.Search(queryVector, veclite.TopK(10))

Index

Constants

View Source 
const (
	// PayloadKeyArchived indicates if a record has been archived.
	PayloadKeyArchived = "_archived"
	// PayloadKeyConsolidationGroup is the ID of the consolidation group.
	PayloadKeyConsolidationGroup = "_consolidation_group"
	// PayloadKeyConsolidatedFrom contains IDs of records this was consolidated from.
	PayloadKeyConsolidatedFrom = "_consolidated_from"
	// PayloadKeyIsConsolidation indicates this record is a consolidation of others.
	PayloadKeyIsConsolidation = "_is_consolidation"
)

Reserved payload keys for memory consolidation.

View Source 
const (
	// PayloadKeySessionID identifies which session/conversation a record belongs to.
	PayloadKeySessionID = "_session_id"
	// PayloadKeyTurnNumber is the sequential turn number within a session.
	PayloadKeyTurnNumber = "_turn_number"
	// PayloadKeyRole indicates the role (e.g., "user", "assistant", "system").
	PayloadKeyRole = "_role"
	// PayloadKeyParentChunk links to the parent chunk ID for threaded conversations.
	PayloadKeyParentChunk = "_parent_chunk"
	// PayloadKeyChildChunks contains IDs of child chunks.
	PayloadKeyChildChunks = "_child_chunks"
	// PayloadKeyThreadRoot is the ID of the root chunk in a thread.
	PayloadKeyThreadRoot = "_thread_root"
)

Reserved payload keys for conversation tracking.

View Source 
const (
	// DistanceCosine uses cosine similarity (higher = more similar).
	DistanceCosine = floats.DistanceCosine
	// DistanceDot uses dot product (higher = more similar).
	DistanceDot = floats.DistanceDot
	// DistanceEuclidean uses Euclidean distance (lower = more similar).
	DistanceEuclidean = floats.DistanceEuclidean
	// DistanceEuclideanSquared uses squared Euclidean distance (lower = more similar).
	// Faster than Euclidean since it avoids sqrt.
	DistanceEuclideanSquared = floats.DistanceEuclideanSquared
)
View Source 
const DefaultVectorSpace = "default"

DefaultVectorSpace is the reserved name of the implicit vector space backed by Record.Vector and the collection's primary dimension/distance/index. Every collection has this space, including those created before named vector spaces existed. It cannot be removed or redeclared with AddVectorSpace.

View Source 
const DefaultWALCheckpointBytes int64 = 64 << 20 // 64 MiB

DefaultWALCheckpointBytes is the WAL size at which a WAL-enabled database automatically folds the log into a fresh snapshot (see WithWALCheckpoint).

View Source 
const Version = "0.24.0"

Version is the library version.

Variables

View Source 
var (
	// ErrNotFound is returned when a record or collection is not found.
	ErrNotFound = errors.New("veclite: not found")

	// ErrDimensionMismatch is returned when vector dimensions don't match.
	ErrDimensionMismatch = errors.New("veclite: dimension mismatch")

	// ErrEmptyVector is returned when an empty vector is provided.
	ErrEmptyVector = errors.New("veclite: empty vector")

	// ErrCollectionExists is returned when trying to create a collection that already exists.
	ErrCollectionExists = errors.New("veclite: collection already exists")

	// ErrDatabaseClosed is returned when operations are attempted on a closed database.
	ErrDatabaseClosed = errors.New("veclite: database closed")

	// ErrInvalidPath is returned when an invalid file path is provided.
	ErrInvalidPath = errors.New("veclite: invalid path")

	// ErrBatchSizeMismatch is returned when batch operation input sizes don't match.
	ErrBatchSizeMismatch = errors.New("veclite: batch size mismatch")

	// ErrReadOnly is returned when a write operation is attempted on a read-only database.
	ErrReadOnly = errors.New("veclite: database is read-only")

	// ErrSharedReadRequiresReadOnly is returned when WithSharedRead is enabled
	// without WithReadOnly. A shared file lock is only safe for read-only access.
	ErrSharedReadRequiresReadOnly = errors.New("veclite: shared read requires read-only mode")

	// ErrVectorSpaceExists is returned when declaring a vector space whose name
	// is already in use (including the reserved "default" space).
	ErrVectorSpaceExists = errors.New("veclite: vector space already exists")

	// ErrVectorSpaceNotFound is returned when referencing an undeclared vector space.
	ErrVectorSpaceNotFound = errors.New("veclite: vector space not found")

	// ErrInvalidVectorSpace is returned when a vector-space declaration is invalid,
	// e.g. an empty name or an attempt to redeclare the reserved "default" space.
	ErrInvalidVectorSpace = errors.New("veclite: invalid vector space")

	// ErrProfileMismatch is returned when two embedding profiles are incompatible,
	// or when a vector does not match a declared embedding profile.
	ErrProfileMismatch = errors.New("veclite: embedding profile mismatch")
)

Sentinel errors for common conditions.

View Source 
var (
	// ErrFileLocked is returned when the database file is locked by another process.
	ErrFileLocked = storage.ErrFileLocked

	// ErrChecksumMismatch is returned when the file checksum does not match.
	ErrChecksumMismatch = storage.ErrChecksumMismatch

	// ErrCorruptedFile is returned when the database file is corrupted.
	ErrCorruptedFile = storage.ErrCorruptedFile

	// ErrInvalidVersion is returned when the file version is not supported.
	ErrInvalidVersion = storage.ErrInvalidVersion
)

Storage-level sentinel errors re-exported for consumer use.

View Source 
var ErrNoEmbedder = errors.New("veclite: no embedder configured")

ErrNoEmbedder is returned when an embedding operation is attempted without an embedder configured on the collection.

Functions

func ExpandEnvVars added in v0.18.0 

func ExpandEnvVars(s string) string

ExpandEnvVars expands ${VAR} and ${VAR:-default} patterns in a string.

func ExpandPath added in v0.11.0 

func ExpandPath(path string) string

ExpandPath expands ~ and environment variables in a path.

func ParseDuration added in v0.18.0 

func ParseDuration(s string, defaultDur time.Duration) time.Duration

ParseDuration parses a duration string with a default fallback.

Types

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

Collection represents a collection of vectors with the same dimension.

func (*Collection) AddVectorSpace added in v0.16.0 

func (c *Collection) AddVectorSpace(config VectorSpaceConfig) error

AddVectorSpace declares an additional named vector space on the collection.

The space is independent of the default space and of other spaces: it has its own dimension, distance metric, and optional HNSW index. After declaring it, insert vectors into it with InsertRecord (or SetRecordVector) and query it with SearchSpace. The reserved name DefaultVectorSpace cannot be used.

func (*Collection) All 

func (c *Collection) All() []*Record

All returns all records in the collection.

func (*Collection) ArchiveRecord added in v0.8.0 

func (c *Collection) ArchiveRecord(id uint64) error

ArchiveRecord marks a record as archived. Archived records are excluded from normal searches but can be retrieved with GetArchived.

func (*Collection) CleanupExpired added in v0.8.0 

func (c *Collection) CleanupExpired() (int, error)

CleanupExpired removes all expired records from the collection. Returns the number of records removed.

func (*Collection) Clear 

func (c *Collection) Clear() error

Clear removes all records from the collection. It preserves the nextID counter to avoid ID reuse after reinsertion. Use Reset if you want to also reset the ID counter. Returns an error if the database is read-only.

func (*Collection) Consolidate added in v0.8.0 

func (c *Collection) Consolidate(config ConsolidationConfig) (*ConsolidationResult, error)

Consolidate finds similar memory clusters and optionally creates consolidated records.

func (*Collection) Count 

func (c *Collection) Count() int

Count returns the number of records in the collection.

func (*Collection) CountExpired added in v0.8.0 

func (c *Collection) CountExpired() int

CountExpired returns the number of expired records in the collection.

func (*Collection) Delete 

func (c *Collection) Delete(id uint64) error

Delete removes a record by ID.

func (*Collection) DeleteMetadataValue added in v0.15.0 

func (c *Collection) DeleteMetadataValue(key string) error

DeleteMetadataValue removes one collection metadata value.

func (*Collection) DeleteWhere 

func (c *Collection) DeleteWhere(filters ...Filter) (int, error)

DeleteWhere removes all records matching the filters. Returns the number of deleted records.

func (*Collection) Dimension 

func (c *Collection) Dimension() int

Dimension returns the vector dimension. Returns 0 if no vectors have been inserted yet.

func (*Collection) DistanceType 

func (c *Collection) DistanceType() floats.DistanceType

DistanceType returns the distance metric type.

func (*Collection) EmbeddingProfile added in v0.16.0 

func (c *Collection) EmbeddingProfile() (EmbeddingProfile, bool)

EmbeddingProfile returns the collection's default-space embedding profile and whether one is set.

func (*Collection) EnforceMemoryLimit added in v0.9.0 

func (c *Collection) EnforceMemoryLimit(config MemoryConfig) int

EnforceMemoryLimit checks and enforces the memory limit for a collection. Call this after inserts if you want immediate enforcement without background monitoring.

func (*Collection) ExpandConsolidation added in v0.8.0 

func (c *Collection) ExpandConsolidation(consolidationID uint64) ([]*Record, error)

ExpandConsolidation retrieves the original records that were consolidated into a consolidation record.

func (*Collection) Find 

func (c *Collection) Find(filters ...Filter) ([]*Record, error)

Find retrieves all records matching the filters.

func (*Collection) FindOne 

func (c *Collection) FindOne(filters ...Filter) (*Record, error)

FindOne retrieves the first record matching the filters.

func (*Collection) FindSimilarClusters added in v0.8.0 

func (c *Collection) FindSimilarClusters(config ConsolidationConfig) ([]MemoryCluster, error)

FindSimilarClusters identifies clusters of similar memories using single-linkage clustering.

func (*Collection) ForEach added in v0.6.0 

func (c *Collection) ForEach(fn func(*Record) bool)

ForEach iterates over all records in the collection, calling fn for each. If fn returns false, iteration stops early. Records are cloned before being passed to fn.

func (*Collection) Get 

func (c *Collection) Get(id uint64) (*Record, error)

Get retrieves a record by ID.

func (*Collection) GetArchived added in v0.8.0 

func (c *Collection) GetArchived() ([]*Record, error)

GetArchived retrieves all archived records.

func (*Collection) GetConsolidations added in v0.8.0 

func (c *Collection) GetConsolidations() ([]*Record, error)

GetConsolidations retrieves all consolidation records.

func (*Collection) GetSession added in v0.8.0 

func (c *Collection) GetSession(sessionID string) ([]*Record, error)

GetSession retrieves all records belonging to a session. Returns records in turn number order.

func (*Collection) GetSessionStats added in v0.8.0 

func (c *Collection) GetSessionStats(sessionID string) (SessionStats, error)

GetSessionStats returns statistics about a session.

func (*Collection) GetThread added in v0.8.0 

func (c *Collection) GetThread(chunkID uint64) ([]*Record, error)

GetThread retrieves all records in a thread starting from the given chunk ID. Returns records in chronological order.

func (*Collection) GetVector 

func (c *Collection) GetVector(id uint64) ([]float32, error)

GetVector retrieves just the vector for a record.

func (*Collection) HasIndex added in v0.2.0 

func (c *Collection) HasIndex() bool

HasIndex returns true if this collection has an index.

func (*Collection) HasVectorSpace added in v0.16.0 

func (c *Collection) HasVectorSpace(name string) bool

HasVectorSpace reports whether the collection has the named space. The default space always exists.

func (*Collection) HybridSearch added in v0.6.0 

func (c *Collection) HybridSearch(query []float32, text string, opts ...SearchOption) ([]Result, error)

HybridSearch performs both vector search and BM25 text search, then fuses results using Reciprocal Rank Fusion (RRF) with k=60. Requires text indexing to be enabled via WithTextIndex. Use WithVectorWeight and WithTextWeight to control the balance.

func (*Collection) HybridSearchSpace added in v0.17.0 

func (c *Collection) HybridSearchSpace(space string, query []float32, text string, opts ...SearchOption) ([]Result, error)

HybridSearchSpace performs vector search over a named vector space and BM25 text search over the collection, then fuses the two result sets with Reciprocal Rank Fusion (k=60). It is the named-space analog of HybridSearch: use it when the query vector lives in a named space declared via AddVectorSpace rather than the default space.

Passing DefaultVectorSpace (or "") is equivalent to HybridSearch. Requires text indexing to be enabled via WithTextIndex. Use WithVectorWeight and WithTextWeight to control the fusion balance.

func (*Collection) IndexStats added in v0.2.0 

func (c *Collection) IndexStats() *HNSWStats

IndexStats returns statistics about the collection's HNSW index. Returns nil if no HNSW index is configured.

func (*Collection) IndexType added in v0.2.0 

func (c *Collection) IndexType() IndexType

IndexType returns the index type for this collection.

func (*Collection) Insert 

func (c *Collection) Insert(vector []float32, payload map[string]any) (uint64, error)

Insert adds a vector with optional payload to the collection. Returns the assigned record ID.

func (*Collection) InsertBatch 

func (c *Collection) InsertBatch(vectors [][]float32, payloads []map[string]any) ([]uint64, error)

InsertBatch adds multiple vectors with payloads to the collection. Returns the assigned record IDs. If payloads is nil or shorter than vectors, missing payloads are treated as nil.

func (*Collection) InsertDocument added in v0.6.0 

func (c *Collection) InsertDocument(vector []float32, content string, payload map[string]any) (uint64, error)

InsertDocument inserts a vector with content text and payload. Content is automatically indexed for BM25 text search when text indexing is enabled.

func (*Collection) InsertRecord added in v0.16.0 

func (c *Collection) InsertRecord(in RecordInput) (uint64, error)

InsertRecord inserts (or, when in.ID names an existing record, replaces) one logical record that may carry vectors in several named vector spaces at once, plus optional content and payload. Returns the record ID.

Each key of in.Vectors must be DefaultVectorSpace (or "") or a space declared via AddVectorSpace; unknown spaces return ErrVectorSpaceNotFound. Vectors are validated against each space's dimension and embedding profile before any state changes.

func (*Collection) InsertText added in v0.6.0 

func (c *Collection) InsertText(text string, payload map[string]any) (uint64, error)

InsertText embeds the text using the configured embedder and inserts the result. Requires an embedder to be set via WithEmbedder.

func (*Collection) InsertTextDocument added in v0.15.0 

func (c *Collection) InsertTextDocument(content string, payload map[string]any) (uint64, error)

InsertTextDocument inserts content and payload without a vector. Text-only records are indexed by BM25 when text indexing is enabled and are skipped by vector search, hybrid vector search, and vector subscriptions.

func (*Collection) InsertTextDocumentWithOptions added in v0.15.0 

func (c *Collection) InsertTextDocumentWithOptions(content string, payload map[string]any, opts ...InsertOption) (uint64, error)

InsertTextDocumentWithOptions inserts content and payload without a vector, applying insert options such as TTL and importance.

func (*Collection) InsertTurn added in v0.8.0 

func (c *Collection) InsertTurn(turn ConversationTurn) (uint64, error)

InsertTurn inserts a conversation turn with conversation metadata. Returns the record ID.

func (*Collection) InsertWithOptions added in v0.8.0 

func (c *Collection) InsertWithOptions(vector []float32, payload map[string]any, opts ...InsertOption) (uint64, error)

InsertWithOptions adds a vector with optional payload and insert options. Use this method to set TTL, importance, and other options.

func (*Collection) Iterate added in v0.6.0 

func (c *Collection) Iterate(opts ...IterOption) *Iterator

Iterate returns an iterator over collection records. Options can control offset and limit for pagination.

func (*Collection) ListSessions added in v0.8.0 

func (c *Collection) ListSessions() []string

ListSessions returns all unique session IDs in the collection.

func (*Collection) Metadata added in v0.15.0 

func (c *Collection) Metadata() map[string]any

Metadata returns a deep copy of the collection metadata.

func (*Collection) MultiSpaceSearch added in v0.16.0 

func (c *Collection) MultiSpaceSearch(queries map[string][]float32, opts ...SearchOption) ([]Result, error)

MultiSpaceSearch runs one query per named vector space and fuses the result sets into a single ranking with Reciprocal Rank Fusion. Keys of queries are vector-space names (DefaultVectorSpace or "" targets the default space).

This is the multimodal entry point: e.g. fuse a "text" query and an "image" query for the same item. For weighted fusion or to also fold in BM25 text results, call SearchSpace / TextSearch and combine the sets with FuseRRF.

func (*Collection) Name 

func (c *Collection) Name() string

Name returns the collection name.

func (*Collection) Reset added in v0.14.0 

func (c *Collection) Reset() error

Reset removes all records from the collection and resets the ID counter to 1. Unlike Clear, this allows ID reuse which may be desirable for testing or when the collection is being fully repopulated. Returns an error if the database is read-only.

func (*Collection) Search 

func (c *Collection) Search(query []float32, opts ...SearchOption) ([]Result, error)

Search finds the most similar vectors to the query vector.

func (*Collection) SearchExplain added in v0.2.0 

func (c *Collection) SearchExplain(query []float32, opts ...SearchOption) (*SearchExplanation, error)

SearchExplain performs a search and returns detailed statistics.

func (*Collection) SearchInSession added in v0.8.0 

func (c *Collection) SearchInSession(sessionID string, query []float32, opts ...SearchOption) ([]Result, error)

SearchInSession searches for similar vectors within a specific session.

func (*Collection) SearchSpace added in v0.16.0 

func (c *Collection) SearchSpace(space string, query []float32, opts ...SearchOption) ([]Result, error)

SearchSpace searches a single named vector space. Passing DefaultVectorSpace (or "") is equivalent to Search. All standard SearchOptions apply (TopK, filters, threshold, pagination, efSearch).

func (*Collection) SearchStream added in v0.6.0 

func (c *Collection) SearchStream(query []float32, fn SearchFunc, opts ...SearchOption) error

SearchStream performs a search and streams results to the callback function. For brute-force searches, results are yielded as they are found, enabling early termination without computing all scores. For HNSW searches, the full result set is fetched first since the index returns ordered results.

func (*Collection) SearchText added in v0.6.0 

func (c *Collection) SearchText(text string, opts ...SearchOption) ([]Result, error)

SearchText embeds the text query using the configured embedder and searches. Requires an embedder to be set via WithEmbedder.

func (*Collection) SetEmbeddingProfile added in v0.16.0 

func (c *Collection) SetEmbeddingProfile(profile EmbeddingProfile) error

SetEmbeddingProfile attaches (or replaces) the collection's first-class default-space embedding profile. Subsequent default-space inserts validate against the profile's dimension.

func (*Collection) SetMetadata added in v0.15.0 

func (c *Collection) SetMetadata(metadata map[string]any) error

SetMetadata replaces the collection metadata.

func (*Collection) SetMetadataValue added in v0.15.0 

func (c *Collection) SetMetadataValue(key string, value any) error

SetMetadataValue sets one collection metadata value.

func (*Collection) SetRecordVector added in v0.16.0 

func (c *Collection) SetRecordVector(id uint64, space string, vector []float32) error

SetRecordVector sets or replaces a single space's vector on an existing record, leaving the record's other spaces, content, and payload untouched. Use DefaultVectorSpace (or "") to target the default space.

func (*Collection) StartMemoryLimiter added in v0.9.0 

func (c *Collection) StartMemoryLimiter(config MemoryConfig) func()

StartMemoryLimiter starts background memory pressure monitoring for a collection. Returns a function to stop the limiter.

func (*Collection) Stats 

func (c *Collection) Stats() CollectionStats

Stats returns statistics about the collection.

func (*Collection) Subscribe added in v0.8.0 

func (c *Collection) Subscribe(query []float32, opts ...SubscriptionOption) (*Subscription, error)

Subscribe creates a subscription that receives events when new records match the query. Returns the subscription which can be used to receive events and close the subscription.

func (*Collection) TextSearch added in v0.6.0 

func (c *Collection) TextSearch(query string, opts ...SearchOption) ([]Result, error)

TextSearch performs BM25 full-text search over indexed fields. Requires text indexing to be enabled via WithTextIndex.

func (*Collection) UnarchiveRecord added in v0.8.0 

func (c *Collection) UnarchiveRecord(id uint64) error

UnarchiveRecord removes the archived flag from a record.

func (*Collection) Unsubscribe added in v0.8.0 

func (c *Collection) Unsubscribe(subscriptionID string) error

Unsubscribe removes a subscription by ID.

func (*Collection) Update 

func (c *Collection) Update(id uint64, payload map[string]any) error

Update updates the payload for a record.

func (*Collection) UpdateDocument added in v0.15.0 

func (c *Collection) UpdateDocument(id uint64, content string, payload map[string]any) error

UpdateDocument updates the content and payload for a record without changing its vector.

func (*Collection) UpdateVector added in v0.4.0 

func (c *Collection) UpdateVector(id uint64, vector []float32) error

UpdateVector updates the vector for a record.

func (*Collection) Upsert added in v0.4.0 

func (c *Collection) Upsert(id uint64, vector []float32, payload map[string]any) (uint64, error)

Upsert inserts a new record or updates an existing one by ID. If the ID is 0, a new record is created with an auto-generated ID. If the ID exists, the vector and payload are updated. If the ID doesn't exist, a new record is created with that ID. Returns the record ID (either the provided one or newly generated).

func (*Collection) UpsertByKey added in v0.4.0 

func (c *Collection) UpsertByKey(keyField string, keyValue any, vector []float32, payload map[string]any) (uint64, bool, error)

UpsertByKey inserts a new record or updates an existing one based on a key field. If a record with payload[keyField] == keyValue exists, it is updated. Otherwise, a new record is inserted. Returns the record ID and whether it was an insert (true) or update (false).

func (*Collection) UpsertRecordByKey added in v0.17.0 

func (c *Collection) UpsertRecordByKey(keyField string, keyValue any, in RecordInput) (uint64, bool, error)

UpsertRecordByKey inserts or replaces the record identified by a payload key, carrying vectors across several named vector spaces atomically. It is the named-vector-space analog of UpsertTextDocumentByKey: it scans for an existing record whose payload[keyField] equals keyValue and, if found, replaces that record's content, payload, and vectors with in's; otherwise it inserts a new record.

On replace, the existing record's CreatedAt, ExpiresAt, Importance, AccessCount, and LastAccessedAt are preserved (matching UpsertTextDocumentByKey semantics). Vectors for every space the record previously participated in are removed from their indexes before the new vectors are inserted, so a space that the replacement no longer carries is correctly dropped.

Returns (id, inserted, error) where inserted is true when a new record was created and false when an existing record was replaced.

keyField must be a non-empty payload key. keyValue is compared with the same type-aware rules as payload filters (see Equal). Each key of in.Vectors must be DefaultVectorSpace (or "") or a space declared via AddVectorSpace.

func (*Collection) UpsertTextDocument added in v0.15.0 

func (c *Collection) UpsertTextDocument(id uint64, content string, payload map[string]any) (uint64, error)

UpsertTextDocument inserts or updates a text-only document by ID. If id is 0, a new record is created with an auto-generated ID.

func (*Collection) UpsertTextDocumentByKey added in v0.15.0 

func (c *Collection) UpsertTextDocumentByKey(keyField string, keyValue any, content string, payload map[string]any) (uint64, bool, error)

UpsertTextDocumentByKey inserts or updates a text-only document by payload key.

func (*Collection) VectorSpace added in v0.16.0 

func (c *Collection) VectorSpace(name string) (VectorSpaceInfo, error)

VectorSpace returns information about a single vector space, or ErrVectorSpaceNotFound.

func (*Collection) VectorSpaces added in v0.16.0 

func (c *Collection) VectorSpaces() []VectorSpaceInfo

VectorSpaces returns information about every vector space on the collection, starting with the default space, followed by named spaces in name order.

type CollectionOption 

type CollectionOption interface {
	// contains filtered or unexported methods
}

CollectionOption configures a collection.

func WithDimension 

func WithDimension(dim int) CollectionOption

WithDimension sets the vector dimension for the collection. If set, all vectors must match this dimension. If not set (0), the dimension is determined by the first insert.

func WithDistanceType 

func WithDistanceType(t floats.DistanceType) CollectionOption

WithDistanceType sets the distance metric for the collection. Default is cosine similarity.

func WithEmbedder added in v0.6.0 

func WithEmbedder(e Embedder) CollectionOption

WithEmbedder sets an auto-embedding plugin for the collection. When set, InsertText and SearchText methods become available.

func WithEmbeddingProfile added in v0.16.0 

func WithEmbeddingProfile(profile EmbeddingProfile) CollectionOption

WithEmbeddingProfile attaches a first-class embedding profile to the collection's default vector space. Vectors inserted into the default space are then validated against the profile's declared dimension.

func WithHNSW added in v0.2.0 

func WithHNSW(m, efConstruction int) CollectionOption

WithHNSW enables HNSW indexing for the collection. m is the maximum number of connections per node (default: 16, recommended: 12-48). efConstruction is the candidate list size during construction (default: 200).

func WithHNSWConfig added in v0.2.0 

func WithHNSWConfig(config HNSWConfig) CollectionOption

WithHNSWConfig enables HNSW indexing with custom configuration.

func WithMemoryLimits added in v0.9.0 

func WithMemoryLimits(config MemoryConfig) CollectionOption

WithMemoryLimits returns a collection option that enables memory pressure handling. When the collection exceeds MaxRecords, older/less important records are evicted.

func WithTextIndex added in v0.6.0 

func WithTextIndex(fields ...string) CollectionOption

WithTextIndex enables BM25 full-text indexing on the specified payload fields. When enabled, string values in these fields are tokenized and indexed for text search. The Content field of records is always indexed when text indexing is enabled.

func WithVectorSpace added in v0.16.0 

func WithVectorSpace(config VectorSpaceConfig) CollectionOption

WithVectorSpace declares an additional named vector space at collection creation time. May be passed multiple times to declare several spaces. The reserved DefaultVectorSpace name is rejected (the default space always exists). Equivalent to calling Collection.AddVectorSpace after creation.

type CollectionSnapshot 

type CollectionSnapshot = storage.CollectionSnapshot

CollectionSnapshot is the serializable state of a collection.

func NewCollectionSnapshot 

func NewCollectionSnapshot(name string, dimension int, distanceType floats.DistanceType) *CollectionSnapshot

NewCollectionSnapshot creates a new empty collection snapshot.

type CollectionStats 

type CollectionStats struct {
	// Name is the collection name.
	Name string

	// Count is the number of records in the collection.
	Count int

	// VectorCount is the number of records with a vector.
	VectorCount int

	// TextOnlyCount is the number of records without a vector.
	TextOnlyCount int

	// Dimension is the vector dimension (0 if not yet set).
	Dimension int

	// DistanceType is the distance metric used.
	DistanceType string

	// IndexType is the index type (none, hnsw).
	IndexType string
}

CollectionStats contains statistics about a collection.

type Config added in v0.11.0 

type Config struct {
	Embedder EmbedderConfig `yaml:"embedder"`
}

Config represents the full veclite configuration.

func DefaultConfig added in v0.11.0 

func DefaultConfig() *Config

DefaultConfig returns a configuration with sensible defaults.

func LoadConfig added in v0.11.0 

func LoadConfig(path string) (*Config, error)

LoadConfig loads configuration from a YAML file. It searches for config files in the following order: 1. The explicit path provided (if not empty) 2. ./veclite.yaml 3. ~/.veclite/config.yaml If no config file is found, returns default configuration.

type ConsolidationConfig added in v0.8.0 

type ConsolidationConfig struct {
	// SimilarityThreshold is the minimum similarity for records to be grouped (0.0-1.0).
	// Higher values mean stricter grouping.
	SimilarityThreshold float32
	// MinGroupSize is the minimum number of records to form a cluster.
	MinGroupSize int
	// MaxGroupSize is the maximum number of records in a cluster.
	MaxGroupSize int
	// SummaryGenerator is a function that creates a summary from a group of records.
	// It returns the summary text, additional payload, and any error.
	// If nil, no consolidation records are created (only clustering is performed).
	SummaryGenerator func([]*Record) (string, map[string]any, error)
	// Embedder is used to generate embeddings for consolidated summaries.
	// Required if SummaryGenerator is provided.
	Embedder Embedder
	// ArchiveOriginals determines whether to archive original records after consolidation.
	ArchiveOriginals bool
	// Filters can be used to limit which records are considered for consolidation.
	Filters []Filter
}

ConsolidationConfig configures memory consolidation.

type ConsolidationResult added in v0.8.0 

type ConsolidationResult struct {
	// ClustersFound is the number of clusters identified.
	ClustersFound int
	// RecordsConsolidated is the total number of records that were consolidated.
	RecordsConsolidated int
	// ConsolidatedRecordIDs are the IDs of newly created consolidation records.
	ConsolidatedRecordIDs []uint64
	// ArchivedRecordIDs are the IDs of records that were archived.
	ArchivedRecordIDs []uint64
	// Clusters contains details about each cluster found.
	Clusters []MemoryCluster
}

ConsolidationResult contains the results of a consolidation operation.

type ConversationTurn added in v0.8.0 

type ConversationTurn struct {
	// SessionID identifies the conversation session.
	SessionID string

	// TurnNumber is the sequential turn number (1-indexed).
	TurnNumber int

	// Role is the speaker role (e.g., "user", "assistant").
	Role string

	// Content is the text content of the turn.
	Content string

	// Vector is the embedding vector. If nil, the collection's embedder is used.
	Vector []float32

	// ParentChunkID links to a parent chunk for threaded conversations.
	ParentChunkID uint64

	// Payload contains additional metadata.
	Payload map[string]any

	// Importance is the importance score (0.0-1.0).
	Importance float32

	// TTL is the time-to-live for this turn.
	TTL time.Duration
}

ConversationTurn represents a single turn in a conversation.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a VecLite database.

func Open 

func Open(path string, opts ...Option) (*DB, error)

Open opens or creates a VecLite database at the given path. Use ":memory:" for an in-memory database that won't be persisted.

func (*DB) Close 

func (db *DB) Close() error

Close closes the database, syncing any pending changes.

func (*DB) Collection 

func (db *DB) Collection(name string) *Collection

Collection returns a collection by name, creating it if it doesn't exist. This is the preferred way to get collections for most use cases. In read-only mode, returns nil if the collection doesn't exist (cannot create).

func (*DB) Collections 

func (db *DB) Collections() []string

Collections returns the names of all collections.

func (*DB) CreateCollection 

func (db *DB) CreateCollection(name string, opts ...CollectionOption) (*Collection, error)

CreateCollection creates a new collection with the given options. Returns an error if the collection already exists.

func (*DB) CreateEpisodeStore added in v0.8.0 

func (db *DB) CreateEpisodeStore(memoriesCollectionName string) (*EpisodeStore, error)

CreateEpisodeStore creates a new episode store for a collection.

func (*DB) CreateKnowledgeGraph added in v0.8.0 

func (db *DB) CreateKnowledgeGraph(name string) (*KnowledgeGraph, error)

CreateKnowledgeGraph creates a new knowledge graph.

func (*DB) DeleteMetadataValue added in v0.15.0 

func (db *DB) DeleteMetadataValue(key string) error

DeleteMetadataValue removes one database metadata value.

func (*DB) DropCollection 

func (db *DB) DropCollection(name string) error

DropCollection removes a collection and all its data.

func (*DB) GetCollection 

func (db *DB) GetCollection(name string) (*Collection, error)

GetCollection returns an existing collection or ErrNotFound.

func (*DB) GetEpisodeStore added in v0.14.0 

func (db *DB) GetEpisodeStore(name string) (*EpisodeStore, error)

func (*DB) GetKnowledgeGraph added in v0.14.0 

func (db *DB) GetKnowledgeGraph(name string) (*KnowledgeGraph, error)

func (*DB) HasCollection 

func (db *DB) HasCollection(name string) bool

HasCollection returns true if a collection exists.

func (*DB) IsClosed 

func (db *DB) IsClosed() bool

IsClosed returns true if the database is closed.

func (*DB) Metadata added in v0.15.0 

func (db *DB) Metadata() map[string]any

Metadata returns a deep copy of the database metadata.

func (*DB) Metrics added in v0.6.0 

func (db *DB) Metrics() MetricsSnapshot

Metrics returns the current metrics snapshot.

func (*DB) Path 

func (db *DB) Path() string

Path returns the database file path.

func (*DB) Reload added in v0.20.0 

func (db *DB) Reload() error

Reload re-reads the database from storage, rebuilding all in-memory state (collections, HNSW indexes, BM25 inverted indexes, knowledge graphs, episode stores). It is intended for read-only databases opened with WithSharedRead so they can pick up writes performed by another process without closing and reopening.

Reload acquires the DB write lock for the duration of the rebuild. It is not safe to call concurrently with reads or writes on the same DB instance.

Background workers (TTL cleaner, memory limiter) are stopped before reload because they reference the old collection structs that are replaced during reload; callers that need those workers must restart them after Reload returns.

Reload returns an error if the database is closed, if the storage backend does not support reloading (e.g. in-memory), or if the reloaded snapshot is corrupt. On error, the in-memory state is left unchanged.

func (*DB) SetMetadata added in v0.15.0 

func (db *DB) SetMetadata(metadata map[string]any) error

SetMetadata replaces the database metadata.

func (*DB) SetMetadataValue added in v0.15.0 

func (db *DB) SetMetadataValue(key string, value any) error

SetMetadataValue sets one database metadata value.

func (*DB) StartTTLCleaner added in v0.9.0 

func (db *DB) StartTTLCleaner(interval time.Duration, callback TTLCleanerCallback) func()

StartTTLCleaner starts a background goroutine that periodically cleans up expired records from all collections in the database. Returns a function to stop the cleaner.

func (*DB) Stats 

func (db *DB) Stats() DatabaseStats

Stats returns statistics about the database.

func (*DB) Sync 

func (db *DB) Sync() error

Sync writes all pending changes to storage.

type DatabaseSnapshot 

type DatabaseSnapshot = storage.DatabaseSnapshot

DatabaseSnapshot is the serializable state of the database.

func NewDatabaseSnapshot 

func NewDatabaseSnapshot() *DatabaseSnapshot

NewDatabaseSnapshot creates a new empty database snapshot.

type DatabaseStats 

type DatabaseStats struct {
	// Path is the database file path (":memory:" for in-memory).
	Path string

	// Collections is the number of collections.
	Collections int

	// TotalRecords is the total number of records across all collections.
	TotalRecords int

	// CollectionStats contains stats for each collection.
	CollectionStats []CollectionStats
}

DatabaseStats contains statistics about the database.

type DecayConfig added in v0.8.0 

type DecayConfig struct {
	// Type is the decay function type.
	Type DecayType
	// HalfLife is the time after which the score is halved (for exponential decay).
	// For linear decay, this is the maximum age after which the score is zero.
	// For gaussian decay, this is the sigma parameter.
	HalfLife time.Duration
}

DecayConfig holds the configuration for temporal decay.

type DecayType added in v0.8.0 

type DecayType string

DecayType specifies the type of temporal decay function to apply. 

const (
	// DecayNone applies no temporal decay.
	DecayNone DecayType = "none"
	// DecayExponential applies exponential decay: score * 2^(-age/halfLife)
	DecayExponential DecayType = "exponential"
	// DecayLinear applies linear decay: score * max(0, 1 - age/maxAge)
	DecayLinear DecayType = "linear"
	// DecayGaussian applies Gaussian decay: score * exp(-0.5 * (age/sigma)^2)
	DecayGaussian DecayType = "gaussian"
)

type DimensionError 

type DimensionError struct {
	Expected int
	Got      int
}

DimensionError provides details about dimension mismatches.

func (*DimensionError) Error 

func (e *DimensionError) Error() string

func (*DimensionError) Unwrap 

func (e *DimensionError) Unwrap() error

type DistanceType added in v0.2.0 

type DistanceType = floats.DistanceType

Re-export distance types for external use.

type Embedder added in v0.6.0 

type Embedder interface {
	// Embed converts a single text into a vector embedding.
	Embed(text string) ([]float32, error)

	// EmbedBatch converts multiple texts into vector embeddings.
	EmbedBatch(texts []string) ([][]float32, error)

	// Dimension returns the output vector dimension.
	Dimension() int
}

Embedder is the interface for auto-embedding text to vectors. Implementations live in separate modules to maintain zero-dependency core.

func NewEmbedderFromConfig added in v0.11.0 

func NewEmbedderFromConfig(cfg EmbedderConfig) (Embedder, error)

NewEmbedderFromConfig creates an embedder based on the provided configuration. It returns an Embedder that can be used with veclite collections.

Supported providers:

  • "openai": OpenAI embedding API (requires API key)
  • "ollama": Ollama local embedding (requires Ollama running)
  • "onnx": Local ONNX inference (requires onnx build tag)

Example: 

cfg, _ := veclite.LoadConfig("veclite.yaml")
embedder, _ := veclite.NewEmbedderFromConfig(cfg.Embedder)
defer embedder.Close()

db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(embedder.Dimension()),
    veclite.WithEmbedder(embedder),
)

type EmbedderConfig added in v0.11.0 

type EmbedderConfig struct {
	Provider string       `yaml:"provider"`
	OpenAI   OpenAIConfig `yaml:"openai"`
	Ollama   OllamaConfig `yaml:"ollama"`
	ONNX     ONNXConfig   `yaml:"onnx"`
}

EmbedderConfig specifies which embedder provider to use and its settings.

type EmbeddingProfile added in v0.16.0 

type EmbeddingProfile struct {
	// Provider is the embedding provider, e.g. "openai" or "ollama".
	Provider string

	// Model is the embedding model identifier.
	Model string

	// Dimension is the expected vector dimension. 0 means "unspecified".
	Dimension int

	// Distance is the distance metric the embeddings are intended for.
	Distance DistanceType

	// Normalize records whether vectors are L2-normalized.
	Normalize bool

	// Version is an optional app-defined revision for the embedding pipeline.
	Version string
}

EmbeddingProfile is a first-class description of how an embedding was produced.

It is the typed replacement for storing provider/model details loosely in metadata. Attaching a profile to a collection or vector space lets VecLite reject vectors that do not match the declared dimension (and, between two profiles, detect when a model/provider/distance change invalidates an index).

func EmbedderProfile added in v0.23.0 

func EmbedderProfile(e Embedder) (EmbeddingProfile, bool)

EmbedderProfile extracts an EmbeddingProfile from an embedder when it self-describes. It supports both ProfiledEmbedder implementations and the built-in providers under embed/ (which return common.ProfileData). The second return value reports whether the embedder provided a profile.

func (EmbeddingProfile) Compatible added in v0.16.0 

func (p EmbeddingProfile) Compatible(other EmbeddingProfile) error

Compatible reports whether two embedding profiles describe interchangeable vectors. It returns nil when compatible, or an error describing the first mismatch. Provider, Model, Dimension (when both set), Distance (when both set), and Normalize must agree. Version is advisory and never causes an error.

func (EmbeddingProfile) IsZero added in v0.16.0 

func (p EmbeddingProfile) IsZero() bool

IsZero reports whether the profile carries no information.

type Entity added in v0.8.0 

type Entity struct {
	// ID is the unique identifier for this entity.
	ID string
	// Type categorizes the entity (e.g., "person", "company", "concept").
	Type string
	// Name is the human-readable name of the entity.
	Name string
	// Vector is the embedding that represents this entity.
	Vector []float32
	// Properties contains additional entity attributes.
	Properties map[string]any
}

Entity represents a node in the knowledge graph.

func (*Entity) Clone added in v0.8.0 

func (e *Entity) Clone() *Entity

Clone creates a copy of the entity.

type EntitySnapshot added in v0.14.0 

type EntitySnapshot = storage.EntitySnapshot

EntitySnapshot is the serializable state of a knowledge graph entity.

type Episode added in v0.8.0 

type Episode struct {
	// ID is the unique identifier for this episode.
	ID string
	// Title is a human-readable summary of the episode.
	Title string
	// Vector is the embedding that represents the episode (centroid of contained records).
	Vector []float32
	// TimeRange is the span from the first to last record in the episode.
	TimeRange TimeRange
	// RecordIDs are the IDs of records that belong to this episode.
	RecordIDs []uint64
	// CreatedAt is when the episode was created.
	CreatedAt time.Time
	// Metadata contains additional episode information.
	Metadata map[string]any
}

Episode represents a coherent group of related memories forming a discrete experience.

func (*Episode) Duration added in v0.8.0 

func (e *Episode) Duration() time.Duration

Duration returns the duration of the episode.

func (*Episode) RecordCount added in v0.8.0 

func (e *Episode) RecordCount() int

RecordCount returns the number of records in the episode.

type EpisodeConfig added in v0.8.0 

type EpisodeConfig struct {
	// TimeGapThreshold is the maximum time gap between records in the same episode.
	// Records separated by more than this are considered part of different episodes.
	TimeGapThreshold time.Duration
	// MinRecords is the minimum number of records to form an episode.
	MinRecords int
	// MaxRecords is the maximum number of records in an episode.
	MaxRecords int
	// SimilarityThreshold is the minimum similarity for records to be grouped.
	// Used when temporal clustering alone isn't sufficient.
	SimilarityThreshold float32
	// Filters can be used to limit which records are considered for episodes.
	Filters []Filter
}

EpisodeConfig configures episode detection.

type EpisodeResult added in v0.8.0 

type EpisodeResult struct {
	// Result is the underlying search result.
	Result Result
	// Episode is the episode containing this result (if any).
	Episode *Episode
	// EpisodeRecords are other records in the same episode.
	EpisodeRecords []*Record
}

EpisodeResult represents a search result that includes episode context.

type EpisodeSnapshot added in v0.14.0 

type EpisodeSnapshot = storage.EpisodeSnapshot

EpisodeSnapshot is the serializable state of a single episode.

type EpisodeStore added in v0.8.0 

type EpisodeStore struct {
	// contains filtered or unexported fields
}

EpisodeStore manages episodes for a collection.

func (*EpisodeStore) CreateEpisode added in v0.8.0 

func (es *EpisodeStore) CreateEpisode(recordIDs []uint64, title string) (*Episode, error)

CreateEpisode manually creates an episode from a set of record IDs.

func (*EpisodeStore) DeleteEpisode added in v0.8.0 

func (es *EpisodeStore) DeleteEpisode(episodeID string) error

DeleteEpisode removes an episode (does not delete the underlying records).

func (*EpisodeStore) DetectEpisodes added in v0.8.0 

func (es *EpisodeStore) DetectEpisodes(config EpisodeConfig) ([]*Episode, error)

DetectEpisodes automatically detects episodes using temporal and similarity clustering.

func (*EpisodeStore) ExpandEpisode added in v0.8.0 

func (es *EpisodeStore) ExpandEpisode(episodeID string) ([]*Record, error)

ExpandEpisode retrieves all records belonging to an episode.

func (*EpisodeStore) FindRecordEpisode added in v0.8.0 

func (es *EpisodeStore) FindRecordEpisode(recordID uint64) (*Episode, error)

FindRecordEpisode finds the episode containing a specific record (if any).

func (*EpisodeStore) GetEpisode added in v0.8.0 

func (es *EpisodeStore) GetEpisode(episodeID string) (*Episode, error)

GetEpisode retrieves an episode by ID.

func (*EpisodeStore) ListEpisodes added in v0.8.0 

func (es *EpisodeStore) ListEpisodes() []*Episode

ListEpisodes returns all episodes.

func (*EpisodeStore) SearchEpisodes added in v0.8.0 

func (es *EpisodeStore) SearchEpisodes(query []float32, limit int) ([]*Episode, error)

SearchEpisodes searches for episodes by their vector representation.

func (*EpisodeStore) SearchWithEpisodeExpansion added in v0.8.0 

func (es *EpisodeStore) SearchWithEpisodeExpansion(query []float32, opts ...SearchOption) ([]EpisodeResult, error)

SearchWithEpisodeExpansion performs a search and includes episode context for results.

type EpisodeStoreSnapshot added in v0.14.0 

type EpisodeStoreSnapshot = storage.EpisodeStoreSnapshot

EpisodeStoreSnapshot is the serializable state of an episode store.

type ExpandedSearchResult added in v0.8.0 

type ExpandedSearchResult struct {
	// Entity is the primary search result.
	Entity *Entity
	// Score is the similarity score.
	Score float32
	// RelatedEntities are entities connected to the primary result.
	RelatedEntities []*Entity
	// Relationships are the connections to related entities.
	Relationships []*Relationship
}

ExpandedSearchResult contains search results with graph context.

type Filter 

type Filter interface {
	// Match returns true if the record matches the filter criteria.
	Match(r *Record) bool
}

Filter is an interface for filtering records based on payload values.

func AccessCountAbove added in v0.8.0 

func AccessCountAbove(n uint64) Filter

AccessCountAbove creates a filter that matches records accessed more than n times.

func AccessCountBelow added in v0.8.0 

func AccessCountBelow(n uint64) Filter

AccessCountBelow creates a filter that matches records accessed fewer than n times.

func AccessedAfter added in v0.8.0 

func AccessedAfter(t time.Time) Filter

AccessedAfter creates a filter that matches records last accessed after the given time.

func AccessedBefore added in v0.8.0 

func AccessedBefore(t time.Time) Filter

AccessedBefore creates a filter that matches records last accessed before the given time.

func AgeNewerThan added in v0.8.0 

func AgeNewerThan(d time.Duration) Filter

AgeNewerThan creates a filter that matches records created less than d ago.

func AgeOlderThan added in v0.8.0 

func AgeOlderThan(d time.Duration) Filter

AgeOlderThan creates a filter that matches records created more than d ago.

func And 

func And(filters ...Filter) Filter

And creates a filter that matches records matching all given filters.

func Between added in v0.4.0 

func Between(key string, min, max float64) Filter

Between creates a filter that matches records where min <= payload[key] <= max.

func Contains 

func Contains(key, substr string) Filter

Contains creates a filter that matches records where payload[key] contains the substring.

func CreatedAfter added in v0.8.0 

func CreatedAfter(t time.Time) Filter

CreatedAfter creates a filter that matches records created after the given time.

func CreatedBefore added in v0.8.0 

func CreatedBefore(t time.Time) Filter

CreatedBefore creates a filter that matches records created before the given time.

func Equal 

func Equal(key string, value any) Filter

Equal creates a filter that matches records where payload[key] equals value.

func Exists 

func Exists(key string) Filter

Exists creates a filter that matches records where payload[key] exists.

func ExpiredBefore added in v0.8.0 

func ExpiredBefore(t time.Time) Filter

ExpiredBefore creates a filter that matches records that expired before the given time.

func GT added in v0.4.0 

func GT(key string, value float64) Filter

GT is an alias for GreaterThan.

func GTE added in v0.4.0 

func GTE(key string, value float64) Filter

GTE is an alias for GreaterThanOrEqual.

func Glob 

func Glob(key, pattern string) Filter

Glob creates a filter that matches records where payload[key] matches the glob pattern.

func GreaterThan added in v0.4.0 

func GreaterThan(key string, value float64) Filter

GreaterThan creates a filter that matches records where payload[key] > value.

func GreaterThanOrEqual added in v0.4.0 

func GreaterThanOrEqual(key string, value float64) Filter

GreaterThanOrEqual creates a filter that matches records where payload[key] >= value.

func HasTTLFilter added in v0.8.0 

func HasTTLFilter() Filter

HasTTLFilter creates a filter that matches records with a TTL set.

func ImportanceAbove added in v0.8.0 

func ImportanceAbove(threshold float32) Filter

ImportanceAbove creates a filter that matches records with importance > threshold.

func ImportanceBelow added in v0.8.0 

func ImportanceBelow(threshold float32) Filter

ImportanceBelow creates a filter that matches records with importance < threshold.

func ImportanceBetween added in v0.8.0 

func ImportanceBetween(min, max float32) Filter

ImportanceBetween creates a filter that matches records with min <= importance <= max.

func In 

func In(key string, values ...any) Filter

In creates a filter that matches records where payload[key] is in the given values.

func LT added in v0.4.0 

func LT(key string, value float64) Filter

LT is an alias for LessThan.

func LTE added in v0.4.0 

func LTE(key string, value float64) Filter

LTE is an alias for LessThanOrEqual.

func LessThan added in v0.4.0 

func LessThan(key string, value float64) Filter

LessThan creates a filter that matches records where payload[key] < value.

func LessThanOrEqual added in v0.4.0 

func LessThanOrEqual(key string, value float64) Filter

LessThanOrEqual creates a filter that matches records where payload[key] <= value.

func NeverAccessed added in v0.8.0 

func NeverAccessed() Filter

NeverAccessed creates a filter that matches records that have never been accessed via search.

func Not 

func Not(filter Filter) Filter

Not creates a filter that negates the given filter.

func NotEqual 

func NotEqual(key string, value any) Filter

NotEqual creates a filter that matches records where payload[key] does not equal value.

func NotExpired added in v0.8.0 

func NotExpired() Filter

NotExpired creates a filter that matches records that have not expired. This includes records with no TTL set.

func NotIn 

func NotIn(key string, values ...any) Filter

NotIn creates a filter that matches records where payload[key] is not in the given values.

func Or 

func Or(filters ...Filter) Filter

Or creates a filter that matches records matching any given filter.

func Prefix 

func Prefix(key, prefix string) Filter

Prefix creates a filter that matches records where payload[key] has the given prefix.

func Suffix 

func Suffix(key, suffix string) Filter

Suffix creates a filter that matches records where payload[key] has the given suffix.

func UpdatedAfter added in v0.8.0 

func UpdatedAfter(t time.Time) Filter

UpdatedAfter creates a filter that matches records updated after the given time.

func UpdatedBefore added in v0.8.0 

func UpdatedBefore(t time.Time) Filter

UpdatedBefore creates a filter that matches records updated before the given time.

type FilterFunc 

type FilterFunc func(r *Record) bool

FilterFunc is a function adapter for the Filter interface.

func (FilterFunc) Match 

func (f FilterFunc) Match(r *Record) bool

Match implements Filter interface.

type FuseOption added in v0.16.0 

type FuseOption interface {
	// contains filtered or unexported methods
}

FuseOption configures result fusion (see FuseRRF).

func WithFusionTopK added in v0.16.0 

func WithFusionTopK(n int) FuseOption

WithFusionTopK truncates the fused output to at most n results (0 = no limit).

func WithFusionWeights added in v0.16.0 

func WithFusionWeights(weights ...float64) FuseOption

WithFusionWeights sets a per-result-set weight. The i-th weight scales the contribution of the i-th result set. Missing weights default to 1.0.

func WithRRFK added in v0.16.0 

func WithRRFK(k int) FuseOption

WithRRFK sets the Reciprocal Rank Fusion constant k (default 60). Higher values flatten the influence of top ranks.

type GraphSnapshot added in v0.14.0 

type GraphSnapshot = storage.GraphSnapshot

GraphSnapshot is the serializable state of a knowledge graph.

type HNSWConfig added in v0.2.0 

type HNSWConfig = storage.HNSWConfig

HNSWConfig holds HNSW index configuration.

type HNSWStats added in v0.12.0 

type HNSWStats struct {
	// NodeCount is the number of vectors in the index.
	NodeCount int `json:"node_count"`
	// MaxLevel is the highest level in the HNSW graph.
	MaxLevel int `json:"max_level"`
	// EntryPointID is the ID of the entry point node.
	EntryPointID uint64 `json:"entry_point_id"`
}

HNSWStats contains statistics about an HNSW index. This is the public wrapper for internal index statistics.

type Index added in v0.2.0 

type Index interface {
	// Insert adds a vector with the given ID to the index.
	Insert(id uint64, vector []float32) error

	// Delete removes a vector from the index.
	Delete(id uint64) error

	// Search finds the k nearest neighbors to the query vector.
	// Returns IDs and distances/similarities.
	Search(query []float32, k int) ([]IndexResult, error)

	// SearchWithEf searches with a custom ef parameter (for HNSW).
	// For indexes that don't support ef, this should behave like Search.
	SearchWithEf(query []float32, k int, ef int) ([]IndexResult, error)

	// Count returns the number of vectors in the index.
	Count() int

	// Type returns the index type name.
	Type() string

	// Clear removes all vectors from the index.
	Clear()
}

Index is the interface for vector search indexes. Implementations can provide different algorithms (brute-force, HNSW, etc.).

type IndexResult added in v0.2.0 

type IndexResult struct {
	ID       uint64
	Distance float32
}

IndexResult represents a search result from an index.

type IndexType added in v0.2.0 

type IndexType string

IndexType represents the type of index. 

const (
	// IndexTypeNone means no index (brute force search).
	IndexTypeNone IndexType = "none"

	// IndexTypeHNSW means HNSW approximate nearest neighbor index.
	IndexTypeHNSW IndexType = "hnsw"
)

type InsertOption added in v0.8.0 

type InsertOption interface {
	// contains filtered or unexported methods
}

InsertOption configures record insertion behavior.

func WithContentOption added in v0.8.0 

func WithContentOption(content string) InsertOption

WithContentOption sets the content field for the record. This is an alternative to using InsertDocument.

func WithExpiresAt added in v0.8.0 

func WithExpiresAt(t time.Time) InsertOption

WithExpiresAt sets an explicit expiration time for the record. This takes precedence over WithTTL if both are specified.

func WithImportance added in v0.8.0 

func WithImportance(score float32) InsertOption

WithImportance sets the importance score for the record. Value should be between 0.0 and 1.0. Values outside this range are clamped.

func WithTTL added in v0.8.0 

func WithTTL(d time.Duration) InsertOption

WithTTL sets a time-to-live duration for the record. The record will expire after this duration from the time of insertion.

type InvertedIndexSnapshot added in v0.6.0 

type InvertedIndexSnapshot = storage.InvertedIndexSnapshot

InvertedIndexSnapshot is the serializable state of the BM25 inverted index.

type IterOption added in v0.6.0 

type IterOption interface {
	// contains filtered or unexported methods
}

IterOption configures the iterator.

func IterLimit added in v0.6.0 

func IterLimit(n int) IterOption

IterLimit sets the maximum number of records to return.

func IterOffset added in v0.6.0 

func IterOffset(n int) IterOption

IterOffset sets the number of records to skip.

type Iterator added in v0.6.0 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator allows iterating over collection records one at a time.

func (*Iterator) Close added in v0.6.0 

func (it *Iterator) Close()

Close releases resources held by the iterator.

func (*Iterator) Next added in v0.6.0 

func (it *Iterator) Next() (*Record, bool)

Next returns the next record and true, or nil and false if done.

type KnowledgeGraph added in v0.8.0 

type KnowledgeGraph struct {
	// contains filtered or unexported fields
}

KnowledgeGraph provides a graph-based knowledge base with vector search.

func (*KnowledgeGraph) AddEntity added in v0.8.0 

func (kg *KnowledgeGraph) AddEntity(entity Entity) error

AddEntity adds an entity to the knowledge graph.

func (*KnowledgeGraph) AddRelationship added in v0.8.0 

func (kg *KnowledgeGraph) AddRelationship(rel Relationship) error

AddRelationship adds a relationship between two entities.

func (*KnowledgeGraph) DeleteEntity added in v0.8.0 

func (kg *KnowledgeGraph) DeleteEntity(entityID string) error

DeleteEntity removes an entity and all its relationships.

func (*KnowledgeGraph) DeleteRelationship added in v0.8.0 

func (kg *KnowledgeGraph) DeleteRelationship(relID string) error

DeleteRelationship removes a relationship.

func (*KnowledgeGraph) GetEntity added in v0.8.0 

func (kg *KnowledgeGraph) GetEntity(entityID string) (*Entity, error)

GetEntity retrieves an entity by ID.

func (*KnowledgeGraph) GetRelationship added in v0.8.0 

func (kg *KnowledgeGraph) GetRelationship(relID string) (*Relationship, error)

GetRelationship retrieves a relationship by ID.

func (*KnowledgeGraph) GetRelationships added in v0.8.0 

func (kg *KnowledgeGraph) GetRelationships(entityID string, direction string) []*Relationship

GetRelationships returns all relationships for an entity.

func (*KnowledgeGraph) ListEntities added in v0.8.0 

func (kg *KnowledgeGraph) ListEntities(entityType string) []*Entity

ListEntities returns all entities, optionally filtered by type.

func (*KnowledgeGraph) Name added in v0.8.0 

func (kg *KnowledgeGraph) Name() string

Name returns the name of the knowledge graph.

func (*KnowledgeGraph) SearchWithExpansion added in v0.8.0 

func (kg *KnowledgeGraph) SearchWithExpansion(query []float32, traversalConfig TraversalConfig, opts ...SearchOption) ([]ExpandedSearchResult, error)

SearchWithExpansion searches for similar entities and expands results with graph context.

func (*KnowledgeGraph) Stats added in v0.8.0 

func (kg *KnowledgeGraph) Stats() KnowledgeGraphStats

Stats returns statistics about the knowledge graph.

func (*KnowledgeGraph) Traverse added in v0.8.0 

func (kg *KnowledgeGraph) Traverse(startIDs []string, config TraversalConfig) (*TraversalResult, error)

Traverse performs a graph traversal starting from the given entity IDs.

func (*KnowledgeGraph) UpdateEntity added in v0.8.0 

func (kg *KnowledgeGraph) UpdateEntity(entity Entity) error

UpdateEntity updates an existing entity.

type KnowledgeGraphStats added in v0.8.0 

type KnowledgeGraphStats struct {
	// EntityCount is the total number of entities.
	EntityCount int
	// RelationshipCount is the total number of relationships.
	RelationshipCount int
	// EntityTypes maps entity types to counts.
	EntityTypes map[string]int
	// RelationshipTypes maps relationship types to counts.
	RelationshipTypes map[string]int
}

KnowledgeGraphStats contains statistics about a knowledge graph.

type Logger added in v0.6.0 

type Logger interface {
	// Debug logs a debug message with key-value pairs.
	Debug(msg string, keysAndValues ...any)

	// Info logs an informational message with key-value pairs.
	Info(msg string, keysAndValues ...any)

	// Error logs an error message with key-value pairs.
	Error(msg string, keysAndValues ...any)
}

Logger is the interface for structured logging in VecLite. Implementations can bridge to any logging library (slog, zap, zerolog, etc.).

type MatchEvent added in v0.8.0 

type MatchEvent struct {
	// Record is the matching record.
	Record *Record
	// Score is the similarity score to the subscription query.
	Score float32
	// Timestamp is when the match was detected.
	Timestamp time.Time
	// SubscriptionID is the ID of the subscription that matched.
	SubscriptionID string
}

MatchEvent represents an event when a new record matches a subscription.

type MemoryCluster added in v0.8.0 

type MemoryCluster struct {
	// ID is a unique identifier for this cluster.
	ID string
	// Records are the records in this cluster.
	Records []*Record
	// Centroid is the average vector of all records in the cluster.
	Centroid []float32
	// AverageImportance is the average importance score of records.
	AverageImportance float32
	// TimeRange is the span from oldest to newest record.
	TimeRange TimeRange
}

MemoryCluster represents a group of similar memories that could be consolidated.

type MemoryConfig added in v0.9.0 

type MemoryConfig struct {
	// MaxRecords is the maximum number of records allowed in the collection.
	// When exceeded, eviction is triggered.
	MaxRecords int

	// EvictionPolicy determines how records are selected for eviction.
	// Supported values: "lru", "fifo", "importance"
	EvictionPolicy string

	// EvictionBatchSize is the number of records to evict at once when pressure is detected.
	// Default is 10% of MaxRecords.
	EvictionBatchSize int

	// CleanupInterval is how often to check for memory pressure.
	// If zero, cleanup only happens on insert.
	CleanupInterval time.Duration
}

MemoryConfig configures memory pressure handling for a collection.

type MemoryLimiter added in v0.9.0 

type MemoryLimiter struct {
	// contains filtered or unexported fields
}

MemoryLimiter manages memory pressure for a collection.

func (*MemoryLimiter) Stop added in v0.9.0 

func (ml *MemoryLimiter) Stop()

Stop stops the memory limiter and waits for it to finish.

type Metrics added in v0.6.0 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics provides observable counters for database operations. All operations are atomic and safe for concurrent access.

func (*Metrics) Snapshot added in v0.6.0 

func (m *Metrics) Snapshot() MetricsSnapshot

Snapshot returns a point-in-time snapshot of the metrics.

type MetricsSnapshot added in v0.6.0 

type MetricsSnapshot struct {
	SearchCount   int64         `json:"search_count"`
	InsertCount   int64         `json:"insert_count"`
	DeleteCount   int64         `json:"delete_count"`
	AvgSearchTime time.Duration `json:"avg_search_time_ns"`
}

MetricsSnapshot is a point-in-time snapshot of database metrics.

type NopLogger added in v0.6.0 

type NopLogger struct{}

NopLogger is a no-op logger that discards all messages. This is the default logger used when none is configured, ensuring zero overhead.

func (NopLogger) Debug added in v0.6.0 

func (NopLogger) Debug(string, ...any)

func (NopLogger) Error added in v0.6.0 

func (NopLogger) Error(string, ...any)

func (NopLogger) Info added in v0.6.0 

func (NopLogger) Info(string, ...any)

type NotFoundError 

type NotFoundError struct {
	Type string // "record", "collection", etc.
	ID   string // Identifier that was not found
}

NotFoundError provides details about what was not found.

func (*NotFoundError) Error 

func (e *NotFoundError) Error() string

func (*NotFoundError) Unwrap 

func (e *NotFoundError) Unwrap() error

type ONNXConfig added in v0.11.0 

type ONNXConfig struct {
	ModelDir string `yaml:"model_dir"`
	Model    string `yaml:"model"`
}

ONNXConfig holds ONNX embedder configuration.

type OllamaConfig added in v0.11.0 

type OllamaConfig struct {
	BaseURL string `yaml:"base_url"`
	Model   string `yaml:"model"`
	Timeout string `yaml:"timeout"`
}

OllamaConfig holds Ollama embedder configuration.

type OpenAIConfig added in v0.11.0 

type OpenAIConfig struct {
	APIKey    string `yaml:"api_key"`
	Model     string `yaml:"model"`
	BaseURL   string `yaml:"base_url"`
	Dimension int    `yaml:"dimension"`
	Timeout   string `yaml:"timeout"`
}

OpenAIConfig holds OpenAI embedder configuration.

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option configures the database.

func WithLogger added in v0.6.0 

func WithLogger(l Logger) Option

WithLogger sets a logger for the database. Pass nil or NopLogger{} to disable logging (default).

func WithReadOnly 

func WithReadOnly(enabled bool) Option

WithReadOnly opens the database in read-only mode. Write operations will return an error.

func WithSharedRead added in v0.20.0 

func WithSharedRead(enabled bool) Option

WithSharedRead enables a shared file lock when opening a read-only database, allowing multiple processes to open the same database file simultaneously for read-only access. This is useful for scenarios where one process writes (e.g. an indexer) while other processes read (e.g. search tools).

SharedRead requires ReadOnly — opening a writable database with a shared lock would risk data loss from concurrent full-snapshot saves. If SharedRead is enabled without ReadOnly, Open returns an error.

Readers opened with SharedRead see a point-in-time snapshot taken at Open. Call Reload() to pick up writes from other processes.

func WithSyncOnWrite 

func WithSyncOnWrite(enabled bool) Option

WithSyncOnWrite enables automatic sync after each write operation. This is slower but ensures durability.

func WithWAL added in v0.24.0 

func WithWAL(enabled bool) Option

WithWAL enables the write-ahead log for a file-backed database. Every completed write appends the affected records to a sidecar log (path + ".wal") with a single fsync, so mutations survive a crash between full-snapshot saves without WithSyncOnWrite's cost of rewriting the whole database on every write. The log is replayed and folded into a fresh snapshot on the next open, and truncated by every successful Sync or Close.

The option is ignored for in-memory databases. Opens without WithWAL still recover a log left behind by a crashed WAL-enabled writer.

Durability caveat: a failed log append (disk full, I/O error) does not fail the write — it is logged, the affected records are retried on the next flush, and the data still persists on the next successful Sync or Close. Between a failed append and that point, those writes would not survive a crash.

func WithWALCheckpoint added in v0.24.0 

func WithWALCheckpoint(bytes int64) Option

WithWALCheckpoint sets the log size (in bytes) at which a WAL-enabled database automatically folds the log into a fresh snapshot and truncates it, bounding log growth for long-running writers that rarely call Sync. The default is DefaultWALCheckpointBytes (64 MiB). Pass 0 to disable automatic checkpointing (the log then grows until an explicit Sync or Close). Has no effect without WithWAL.

type ProfiledEmbedder added in v0.23.0 

type ProfiledEmbedder interface {
	Embedder

	// Profile describes the embedder's provider, model, dimension,
	// intended distance metric, and normalization behavior.
	Profile() EmbeddingProfile
}

ProfiledEmbedder is an optional extension of Embedder for implementations that can describe how they produce vectors. It is not required: an Embedder may or may not support it, and callers discover support with a type assertion (or via EmbedderProfile, which also understands the built-in providers under embed/). 

if pe, ok := e.(veclite.ProfiledEmbedder); ok {
    profile := pe.Profile()
    ...
}

type Record 

type Record struct {
	// ID is the unique identifier for this record.
	ID uint64

	// Vector is the embedding vector for the implicit "default" vector space.
	Vector []float32

	// Vectors holds embeddings for additional named vector spaces, keyed by
	// space name (see Collection.AddVectorSpace). The "default" space is always
	// represented by the Vector field above and never stored here. Nil when the
	// record only uses the default space.
	Vectors map[string][]float32

	// Payload contains arbitrary metadata associated with the vector.
	Payload map[string]any

	// Content is the optional original text content associated with this record.
	// Used for document-oriented storage and automatically indexed by BM25 when text indexing is enabled.
	Content string

	// CreatedAt is when the record was inserted.
	CreatedAt time.Time

	// UpdatedAt is when the record was last updated.
	UpdatedAt time.Time

	// ExpiresAt is when this record expires. Zero value means never expires.
	ExpiresAt time.Time

	// Importance is a score from 0.0 to 1.0 indicating how important this record is.
	// Higher values make the record more likely to be returned in search results.
	Importance float32

	// AccessCount tracks how many times this record has been accessed via search.
	AccessCount uint64

	// LastAccessedAt is when this record was last accessed via search.
	LastAccessedAt time.Time
}

Record represents a stored vector with its metadata.

func (*Record) Clone 

func (r *Record) Clone() *Record

Clone creates a deep copy of the record.

func (*Record) HasTTL added in v0.8.0 

func (r *Record) HasTTL() bool

HasTTL returns true if the record has an expiration time set.

func (*Record) IsExpired added in v0.8.0 

func (r *Record) IsExpired() bool

IsExpired returns true if the record has a TTL set and has expired.

func (*Record) TTL added in v0.8.0 

func (r *Record) TTL() time.Duration

TTL returns the remaining time until expiration. Returns 0 if no TTL is set or if the record has already expired.

func (*Record) VectorFor added in v0.16.0 

func (r *Record) VectorFor(space string) ([]float32, bool)

VectorFor returns the vector this record holds for the named vector space. The reserved name "default" (DefaultVectorSpace) returns the Vector field. The second result is false when the record has no vector in that space.

type RecordInput added in v0.16.0 

type RecordInput struct {
	// ID selects the record to upsert. 0 assigns a fresh auto-incremented ID.
	ID uint64

	// Content is optional text indexed by BM25 when text indexing is enabled.
	Content string

	// Payload is arbitrary metadata stored with the record.
	Payload map[string]any

	// Vectors maps vector-space name to embedding. See the type doc for the
	// reserved "default" key.
	Vectors map[string][]float32
}

RecordInput is one logical record that may carry vectors in several named vector spaces simultaneously, plus optional text content and payload.

Keys of Vectors are vector-space names. The reserved key DefaultVectorSpace (or the empty string) targets the default space (Record.Vector). Spaces other than "default" must already be declared via AddVectorSpace. A record may omit any space; it is then absent from that space's index. A record with no vectors at all behaves like a text-only document.

type RecordSnapshot 

type RecordSnapshot = storage.RecordSnapshot

RecordSnapshot is the serializable state of a record.

type Relationship added in v0.8.0 

type Relationship struct {
	// ID is the unique identifier for this relationship.
	ID string
	// SourceID is the ID of the source entity.
	SourceID string
	// TargetID is the ID of the target entity.
	TargetID string
	// Type describes the relationship (e.g., "works_at", "knows", "related_to").
	Type string
	// Weight is the strength of the relationship (0.0-1.0).
	Weight float32
	// Properties contains additional relationship attributes.
	Properties map[string]any
	// Bidirectional indicates if the relationship goes both ways.
	Bidirectional bool
}

Relationship represents an edge between two entities in the knowledge graph.

func (*Relationship) Clone added in v0.8.0 

func (r *Relationship) Clone() *Relationship

Clone creates a copy of the relationship.

type RelationshipSnapshot added in v0.14.0 

type RelationshipSnapshot = storage.RelationshipSnapshot

RelationshipSnapshot is the serializable state of a knowledge graph relationship.

type Result 

type Result struct {
	// Record is the matched record.
	Record *Record

	// Score is the similarity/distance score.
	// For cosine/dot: higher is more similar.
	// For euclidean: lower is more similar.
	Score float32
}

Result represents a search result with its similarity score.

func FuseRRF added in v0.16.0 

func FuseRRF(resultSets [][]Result, opts ...FuseOption) []Result

FuseRRF merges multiple ranked result sets into a single ranking using Reciprocal Rank Fusion. It is the public, modality-agnostic fusion primitive behind HybridSearch and Collection.MultiSpaceSearch: callers can fuse any mix of vector-space results, BM25 text results, or externally produced rankings.

Records are deduplicated by ID; a record appearing in several sets accumulates the (weighted) reciprocal-rank contribution of each. The returned results are sorted by fused score descending.

type SearchExplanation added in v0.2.0 

type SearchExplanation struct {
	// Results contains the search results.
	Results []Result

	// IndexType is the type of index used (none, hnsw).
	IndexType string

	// NodesVisited is the number of nodes visited during search.
	// Only populated for HNSW searches.
	NodesVisited int

	// LayersVisited is the number of HNSW layers visited.
	// Only populated for HNSW searches.
	LayersVisited int

	// Duration is how long the search took.
	Duration time.Duration

	// BruteForce indicates whether brute-force search was used.
	BruteForce bool
}

SearchExplanation provides details about how a search was performed.

type SearchFunc added in v0.6.0 

type SearchFunc func(result Result) bool

SearchFunc is a callback for streaming search results. Return false to stop receiving results.

type SearchOption 

type SearchOption interface {
	// contains filtered or unexported methods
}

SearchOption configures search behavior.

func Threshold 

func Threshold(t float32) SearchOption

Threshold sets the minimum similarity score for results. For cosine/dot: results with score >= threshold are returned. For euclidean: results with score <= threshold are returned.

func TopK 

func TopK(k int) SearchOption

TopK sets the maximum number of results to return. Default is 10.

func WithAccessTracking added in v0.8.0 

func WithAccessTracking(enabled bool) SearchOption

WithAccessTracking enables access tracking for search results. When enabled, accessing a record via search increments its access count and updates its last accessed timestamp.

func WithContent added in v0.6.0 

func WithContent(include bool) SearchOption

WithContent controls whether the Content field is included in search results. By default, Content is included. Set to false to exclude it for smaller results.

func WithDecay added in v0.8.0 

func WithDecay(decayType DecayType, halfLife time.Duration) SearchOption

WithDecay sets the temporal decay function for search results. The decay is applied based on record age (time since creation).

func WithEfSearch added in v0.2.0 

func WithEfSearch(ef int) SearchOption

WithEfSearch sets the efSearch parameter for HNSW search. Higher values improve recall at the cost of speed. Has no effect on collections without HNSW index.

func WithFilter 

func WithFilter(f Filter) SearchOption

WithFilter adds a filter to the search. Multiple filters are combined with AND logic.

func WithFilters 

func WithFilters(filters ...Filter) SearchOption

WithFilters adds multiple filters to the search. All filters are combined with AND logic.

func WithImportanceBoost added in v0.8.0 

func WithImportanceBoost(factor float32) SearchOption

WithImportanceBoost sets a boost factor for importance scores. The final score is: baseScore * (1 + importanceBoost * importance) A factor of 1.0 means importance can double the score.

func WithLimit added in v0.6.0 

func WithLimit(n int) SearchOption

WithLimit sets the maximum number of results to return. This is an alias for TopK for use in pagination contexts.

func WithOffset added in v0.6.0 

func WithOffset(n int) SearchOption

WithOffset sets the number of results to skip before returning. Use with TopK for pagination: WithOffset(20), TopK(10) returns results 21-30.

func WithTextWeight added in v0.6.0 

func WithTextWeight(w float64) SearchOption

WithTextWeight sets the weight for the text search component in hybrid search. Default is 1.0.

func WithVectorWeight added in v0.6.0 

func WithVectorWeight(w float64) SearchOption

WithVectorWeight sets the weight for the vector search component in hybrid search. Default is 1.0.

type SessionStats added in v0.8.0 

type SessionStats struct {
	// SessionID is the session identifier.
	SessionID string
	// TurnCount is the number of turns in the session.
	TurnCount int
	// FirstTurn is the timestamp of the first turn.
	FirstTurn time.Time
	// LastTurn is the timestamp of the last turn.
	LastTurn time.Time
	// Roles contains the count of turns by role.
	Roles map[string]int
}

SessionStats returns statistics about a session.

type Storage 

type Storage = storage.Backend

Storage is the interface for database persistence. Implement this interface to provide custom storage backends.

type StorageError 

type StorageError = storage.Error

StorageError wraps storage-related errors with context.

type Subscription added in v0.8.0 

type Subscription struct {
	// ID is the unique identifier for this subscription.
	ID string
	// Query is the embedding vector to match against.
	Query []float32
	// Threshold is the minimum similarity score for a match.
	Threshold float32
	// Filters are additional filters to apply.
	Filters []Filter
	// contains filtered or unexported fields
}

Subscription represents an active subscription for matching records.

func (*Subscription) Close added in v0.8.0 

func (s *Subscription) Close() error

Close closes the subscription and its event channel.

func (*Subscription) Events added in v0.8.0 

func (s *Subscription) Events() <-chan MatchEvent

Events returns the channel for receiving match events.

func (*Subscription) IsClosed added in v0.8.0 

func (s *Subscription) IsClosed() bool

IsClosed returns true if the subscription is closed.

type SubscriptionOption added in v0.8.0 

type SubscriptionOption interface {
	// contains filtered or unexported methods
}

SubscriptionOption configures a subscription.

func WithSubscriptionBufferSize added in v0.8.0 

func WithSubscriptionBufferSize(size int) SubscriptionOption

WithSubscriptionBufferSize sets the event channel buffer size.

func WithSubscriptionFilter added in v0.8.0 

func WithSubscriptionFilter(f Filter) SubscriptionOption

WithSubscriptionFilter adds a filter to the subscription.

func WithSubscriptionThreshold added in v0.8.0 

func WithSubscriptionThreshold(threshold float32) SubscriptionOption

WithSubscriptionThreshold sets the minimum similarity threshold for matches.

type TFEntry added in v0.14.0 

type TFEntry = storage.TFEntry

TFEntry stores a document ID and its term frequency for a term.

type TTLCleaner added in v0.9.0 

type TTLCleaner struct {
	// contains filtered or unexported fields
}

TTLCleaner periodically removes expired records from collections.

func (*TTLCleaner) Stop added in v0.9.0 

func (tc *TTLCleaner) Stop()

Stop stops the TTL cleaner and waits for it to finish.

type TTLCleanerCallback added in v0.9.0 

type TTLCleanerCallback func(collection string, deleted int)

TTLCleanerCallback is called after each cleanup cycle with the number of deleted records.

type TimeRange added in v0.8.0 

type TimeRange struct {
	Start time.Time
	End   time.Time
}

TimeRange represents a time span.

func (TimeRange) Duration added in v0.8.0 

func (tr TimeRange) Duration() time.Duration

Duration returns the duration of the time range.

type TimeRangeSnapshot added in v0.14.0 

type TimeRangeSnapshot = storage.TimeRangeSnapshot

TimeRangeSnapshot is the serializable state of a time range.

type TraversalConfig added in v0.8.0 

type TraversalConfig struct {
	// MaxDepth is the maximum number of hops from start nodes.
	MaxDepth int
	// MaxNodes is the maximum number of nodes to visit.
	MaxNodes int
	// MinWeight is the minimum relationship weight to follow.
	MinWeight float32
	// RelationshipTypes limits traversal to specific relationship types.
	// Empty means all types.
	RelationshipTypes []string
	// EntityTypes limits traversal to specific entity types.
	// Empty means all types.
	EntityTypes []string
	// Direction controls traversal direction: "outgoing", "incoming", or "both".
	Direction string
}

TraversalConfig configures graph traversal behavior.

type TraversalResult added in v0.8.0 

type TraversalResult struct {
	// Entities are the entities found during traversal.
	Entities []*Entity
	// Relationships are the relationships traversed.
	Relationships []*Relationship
	// Depths maps entity IDs to their depth from start nodes.
	Depths map[string]int
}

TraversalResult contains the results of a graph traversal.

type VectorSpaceConfig added in v0.16.0 

type VectorSpaceConfig struct {
	// Name uniquely identifies the space within its collection. Required, and
	// must not be the reserved DefaultVectorSpace name.
	Name string

	// Dimension fixes the vector length for this space. 0 auto-detects from the
	// first inserted vector.
	Dimension int

	// Distance is the distance metric for this space. Empty defaults to cosine.
	Distance DistanceType

	// Modality is an optional free-form hint such as "text", "image", or "audio".
	Modality string

	// Provider and Model record the embedding source. They are advisory metadata
	// used by embedding-profile compatibility checks.
	Provider string
	Model    string

	// HNSW, when non-nil, enables an HNSW index for this space. Nil uses
	// brute-force search.
	HNSW *HNSWConfig

	// Profile, when non-nil, attaches a first-class embedding profile to the
	// space and enables compatibility validation on insert.
	Profile *EmbeddingProfile
}

VectorSpaceConfig declares a named vector space on a collection.

A vector space is an independent index over one named embedding per record: its own dimension, distance metric, and optional HNSW index. One logical record (see RecordInput) can carry vectors in several spaces at once, for example a "text" embedding and an "image" embedding for the same item.

Apps still own embedding generation; VecLite only stores and searches the vectors the app provides for each space.

type VectorSpaceInfo added in v0.16.0 

type VectorSpaceInfo struct {
	// Name is the vector-space name ("default" for the implicit space).
	Name string

	// Dimension is the configured/auto-detected dimension (0 if not yet known).
	Dimension int

	// Distance is the distance metric used by this space.
	Distance DistanceType

	// Modality is the optional modality hint.
	Modality string

	// Provider and Model record the embedding source, if declared.
	Provider string
	Model    string

	// IndexType is "none" or "hnsw".
	IndexType string

	// VectorCount is the number of records carrying a vector in this space.
	VectorCount int

	// Profile is the space-level embedding profile, if any.
	Profile *EmbeddingProfile
}

VectorSpaceInfo is a read-only view of a vector space's configuration and current state, returned by Collection.VectorSpaces and VectorSpace.

Source Files

View all Source files

Directories

Path Synopsis
Package client provides a thin Go client for a VecLite HTTP server.
 Package client provides a thin Go client for a VecLite HTTP server.
cmd
veclite command
Command veclite provides a CLI for interacting with VecLite databases.
 Command veclite provides a CLI for interacting with VecLite databases.
embed
common
Package common provides shared utilities for embedder implementations.
 Package common provides shared utilities for embedder implementations.
ollama
Package ollama provides an Ollama-based embedding system for veclite.
 Package ollama provides an Ollama-based embedding system for veclite.
openai
Package openai provides an OpenAI-based embedding system for veclite.
 Package openai provides an OpenAI-based embedding system for veclite.
examples
basic command
Example basic demonstrates the core VecLite operations: open a database, insert vectors, search, and close.
 Example basic demonstrates the core VecLite operations: open a database, insert vectors, search, and close.
batch command
Example batch demonstrates batch operations, upsert, and iteration.
 Example batch demonstrates batch operations, upsert, and iteration.
filtering command
Example filtering demonstrates VecLite's rich filter expressions.
 Example filtering demonstrates VecLite's rich filter expressions.
hnsw command
Example hnsw demonstrates HNSW index configuration and performance.
 Example hnsw demonstrates HNSW index configuration and performance.
http-client command
Example http-client demonstrates using VecLite's HTTP API.
 Example http-client demonstrates using VecLite's HTTP API.
internal
floats
Package floats provides optimized floating-point vector operations.
 Package floats provides optimized floating-point vector operations.
hnsw
Package hnsw implements the Hierarchical Navigable Small World graph algorithm for approximate nearest neighbor search.
 Package hnsw implements the Hierarchical Navigable Small World graph algorithm for approximate nearest neighbor search.
storage
Package storage provides persistence backends for VecLite databases.
 Package storage provides persistence backends for VecLite databases.
Package session provides a lazy dual-handle database session that resolves the multi-process lock contention problem with veclite's file-based storage.
 Package session provides a lazy dual-handle database session that resolves the multi-process lock contention problem with veclite's file-based storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.