veclite

package module
v0.7.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jan 29, 2026 License: MIT Imports: 18 Imported by: 0

README

VecLite

Embeddable vector database for Go with zero external dependencies.

Store vectors with metadata in a single file. Search with cosine similarity, dot product, or Euclidean distance. Add HNSW for fast approximate nearest neighbors. Use BM25 for full-text search. Combine both with hybrid search.

Table of Contents

Features

  • Zero dependencies -- Standard library only, no CGO
  • Single-file storage -- Database persists to one .veclite file
  • HNSW indexing -- Fast approximate nearest neighbor search
  • BM25 text search -- Full-text search over record content and payload fields
  • Hybrid search -- Combine vector and text search with Reciprocal Rank Fusion
  • Document storage -- Store original text content alongside vectors
  • Auto-embedding -- Pluggable Embedder interface for text-to-vector conversion
  • Local ONNX embedder -- Optional all-MiniLM-L6-v2 embedder with zero external dependencies (build with -tags onnx)
  • Metadata filtering -- Rich filter expressions (equality, range, glob, prefix, logical operators)
  • Streaming results -- Process results via callback without materializing all at once
  • Pagination -- Offset/limit on search results and record iteration
  • Observability -- Structured logger interface and atomic metrics counters
  • Thread-safe -- Safe for concurrent read/write access
  • In-memory mode -- Use :memory: for testing
  • CLI and HTTP server -- Manage databases from the command line or over REST
  • MCP server -- Expose VecLite as tools for AI agents via Model Context Protocol

Quick Start

package main

import (
    "fmt"
    "github.com/abdul-hamid-achik/veclite"
)

func main() {
    db, _ := veclite.Open("vectors.veclite")
    defer db.Close()

    coll, _ := db.CreateCollection("embeddings",
        veclite.WithDimension(384),
        veclite.WithHNSW(16, 200),
    )

    coll.Insert([]float32{0.1, 0.2 /* ... */}, map[string]any{"file": "main.go"})

    results, _ := coll.Search(queryVector, veclite.TopK(10))
    for _, r := range results {
        fmt.Printf("ID: %d, Score: %.4f\n", r.Record.ID, r.Score)
    }
}

Installation

Requires Go 1.21 or later. No external dependencies.

# Library
go get github.com/abdul-hamid-achik/veclite

# CLI
go install github.com/abdul-hamid-achik/veclite/cmd/veclite@latest

Pre-built binaries are available on the Releases page.

Library API

Opening a Database
// File-based (persistent)
db, err := veclite.Open("vectors.veclite")

// In-memory (testing)
db, err := veclite.Open(":memory:")

// With options
db, err := veclite.Open("vectors.veclite",
    veclite.WithSyncOnWrite(true),  // Sync after each write
    veclite.WithReadOnly(true),     // Read-only mode
    veclite.WithLogger(myLogger),   // Structured logging
)

defer db.Close()
DB Option Description
WithSyncOnWrite(bool) Sync to disk after each write. Slower but durable.
WithReadOnly(bool) Open in read-only mode. Write operations return errors.
WithLogger(Logger) Set structured logger. Default is NopLogger (zero overhead).
Collections
// Get or create with defaults (auto-detects dimension on first insert)
coll := db.Collection("embeddings")

// Create with explicit options
coll, err := db.CreateCollection("embeddings",
    veclite.WithDimension(384),
    veclite.WithDistanceType(veclite.DistanceCosine),
    veclite.WithHNSW(16, 200),
    veclite.WithTextIndex("title", "body"),
    veclite.WithEmbedder(myEmbedder),
)

// Get existing (returns error if not found)
coll, err := db.GetCollection("embeddings")

// List, check, drop
names := db.Collections()
exists := db.HasCollection("embeddings")
err := db.DropCollection("embeddings")
Collection Option Description
WithDimension(int) Fixed vector dimension. 0 (default) auto-detects on first insert.
WithDistanceType(DistanceType) Distance metric. Default: DistanceCosine.
WithHNSW(m, efConstruction) Enable HNSW index with given parameters.
WithHNSWConfig(HNSWConfig) Enable HNSW with full configuration struct.
WithTextIndex(fields...) Enable BM25 text indexing on named payload fields. Content is always indexed.
WithEmbedder(Embedder) Set auto-embedding plugin for InsertText/SearchText.

Distance metrics:

Metric Constant Interpretation
Cosine Similarity DistanceCosine Higher = more similar
Dot Product DistanceDot Higher = more similar
Euclidean DistanceEuclidean Lower = more similar
Inserting Vectors
// Single insert with metadata
id, err := coll.Insert(vector, map[string]any{
    "file": "main.go",
    "type": "code",
})

// Batch insert
ids, err := coll.InsertBatch(
    [][]float32{v1, v2, v3},
    []map[string]any{p1, p2, p3},
)
Document Storage

Store original text content alongside vectors for text search and retrieval:

id, err := coll.InsertDocument(
    vector,
    "Go is a statically typed language designed at Google",
    map[string]any{"title": "Go Language", "category": "programming"},
)

The Content field is stored on the Record and automatically indexed by BM25 when text indexing is enabled.

Searching
// Basic search
results, err := coll.Search(queryVector, veclite.TopK(10))

// With minimum similarity threshold
results, err := coll.Search(queryVector,
    veclite.TopK(10),
    veclite.Threshold(0.8),
)

// With HNSW tuning (higher ef = better recall, slower)
results, err := coll.Search(queryVector,
    veclite.TopK(10),
    veclite.WithEfSearch(200),
)

// Access results
for _, r := range results {
    fmt.Printf("ID: %d, Score: %.4f, Payload: %v\n",
        r.Record.ID, r.Score, r.Record.Payload)
}
Search Option Description
TopK(k) Maximum results to return. Default: 10.
Threshold(t) Minimum similarity score.
WithFilter(f) Add a filter. Multiple filters use AND logic.
WithFilters(f...) Add multiple filters (AND logic).
WithEfSearch(ef) HNSW ef parameter. Higher = better recall, slower.
WithOffset(n) Skip first n results (pagination).
WithLimit(n) Alias for TopK in pagination contexts.
WithContent(bool) Include/exclude Content field in results.
WithVectorWeight(w) Vector search weight in hybrid search. Default: 1.0.
WithTextWeight(w) Text search weight in hybrid search. Default: 1.0.
Text Search (BM25)

Full-text search using BM25 ranking. Requires WithTextIndex on the collection.

coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(384),
    veclite.WithTextIndex("title", "body"),
)

// Insert documents with content
coll.InsertDocument(vector, "Go programming language", map[string]any{
    "title": "Go Language",
    "body":  "Fast and efficient",
})

// Search by text
results, err := coll.TextSearch("Go programming", veclite.TopK(10))

BM25 indexes the Content field automatically, plus any payload fields specified in WithTextIndex. Uses standard BM25 parameters (k1=1.2, b=0.75).

Combine vector similarity and BM25 text search using Reciprocal Rank Fusion (RRF):

results, err := coll.HybridSearch(
    queryVector,
    "Go programming",
    veclite.TopK(10),
    veclite.WithVectorWeight(1.0),
    veclite.WithTextWeight(0.5),
)

RRF merges ranked lists from both searches with configurable weights. This produces better results than either search alone when queries have both semantic and keyword components.

Streaming Results

Process results one at a time via callback. Return false to stop early:

err := coll.SearchStream(queryVector, func(r veclite.Result) bool {
    fmt.Printf("ID: %d, Score: %.4f\n", r.Record.ID, r.Score)
    return r.Score > 0.5 // stop when score drops below threshold
}, veclite.TopK(100))
Filtering

Filter search results and find operations by metadata:

// Equality
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilter(veclite.Equal("type", "code")),
)

// Multiple filters (AND)
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilters(
        veclite.Equal("language", "go"),
        veclite.Prefix("file", "src/"),
    ),
)

// Logical operators
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilter(veclite.Or(
        veclite.Equal("lang", "go"),
        veclite.Equal("lang", "rust"),
    )),
)

// Range filters
results, _ := coll.Search(query,
    veclite.TopK(10),
    veclite.WithFilters(
        veclite.GT("score", 0.5),
        veclite.Between("line", 100, 500),
    ),
)

// Find records without vector search
records, _ := coll.Find(veclite.Equal("type", "code"))
record, _ := coll.FindOne(veclite.Equal("file", "main.go"))

All filter functions:

Filter Description
Equal(key, value) Exact match
NotEqual(key, value) Not equal
In(key, values...) Value in list
NotIn(key, values...) Value not in list
Glob(key, pattern) Glob pattern (e.g., *.go)
Prefix(key, prefix) String prefix
Suffix(key, suffix) String suffix
Contains(key, substr) String contains
Exists(key) Key exists in payload
GT(key, value) Greater than (numeric)
GTE(key, value) Greater than or equal
LT(key, value) Less than (numeric)
LTE(key, value) Less than or equal
Between(key, min, max) Value in range (inclusive)
And(filters...) All must match
Or(filters...) Any can match
Not(filter) Negate
Pagination

Use WithOffset and TopK (or WithLimit) to paginate search results:

// Page 1
page1, _ := coll.Search(query, veclite.TopK(10))

// Page 2
page2, _ := coll.Search(query, veclite.TopK(10), veclite.WithOffset(10))

// Page 3
page3, _ := coll.Search(query, veclite.TopK(10), veclite.WithOffset(20))
Iteration

Browse records without vector search:

// Iterator with offset and limit
it := coll.Iterate(veclite.IterOffset(0), veclite.IterLimit(100))
for {
    record, ok := it.Next()
    if !ok {
        break
    }
    fmt.Println(record.ID, record.Payload)
}
it.Close()

// ForEach (return false to stop early)
coll.ForEach(func(r *veclite.Record) bool {
    fmt.Println(r.ID)
    return true // continue
})

// Get all records
all := coll.All()
Upsert

Insert or update records:

// Upsert by ID (0 = generate new ID)
id, err := coll.Upsert(0, vector, payload)      // insert new
id, err := coll.Upsert(42, vector, payload)     // update if exists, insert otherwise

// Upsert by key field (useful for incremental indexing)
id, wasInsert, err := coll.UpsertByKey("file", "main.go", vector, map[string]any{
    "file": "main.go",
    "line": 100,
})
Updating Records
// Update payload only (keep vector)
err := coll.Update(id, newPayload)

// Update vector only (keep payload)
err := coll.UpdateVector(id, newVector)
Deleting Records
// Delete by ID
err := coll.Delete(42)

// Delete by filter
count, err := coll.DeleteWhere(veclite.Equal("type", "temp"))

// Clear all records
err := coll.Clear()
Auto-Embedding

Implement the Embedder interface to enable automatic text-to-vector conversion:

type Embedder interface {
    Embed(text string) ([]float32, error)
    EmbedBatch(texts []string) ([][]float32, error)
    Dimension() int
}

Usage:

coll, _ := db.CreateCollection("docs",
    veclite.WithEmbedder(myEmbedder),
)

// Insert text (auto-embeds to vector)
id, err := coll.InsertText("Go is a programming language", payload)

// Search by text (auto-embeds query)
results, err := coll.SearchText("programming languages", veclite.TopK(10))

The Embedder interface lives in the core library (zero dependencies). Implementations wrapping OpenAI, Ollama, or other providers belong in separate modules.

Local ONNX Embedder

VecLite provides an optional local embedding system using ONNX Runtime with the all-MiniLM-L6-v2 model. This enables:

  • Zero external API dependencies -- No Ollama or OpenAI required
  • Offline capability -- Text never leaves your machine
  • Low latency -- ~10ms per embedding vs ~50ms with external services
  • 384-dimensional vectors -- Compatible with most vector search use cases
Quick Start (macOS)

Complete setup in 4 steps:

# 1. Install ONNX Runtime
brew install onnxruntime

# 2. Download libtokenizers
mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

# 3. Download model files (~90MB)
mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx"

# 4. Build with ONNX support
CGO_LDFLAGS="-L./lib" go build -tags onnx ./...
Installation Details

The ONNX embedder requires native libraries and model files.

Step 1: Install ONNX Runtime

Platform Command
macOS (Homebrew) brew install onnxruntime
Linux Download from onnxruntime releases
Windows Download from onnxruntime releases

Step 2: Install libtokenizers

The github.com/daulet/tokenizers package requires the libtokenizers native library.

macOS ARM64 (Apple Silicon):

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-arm64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

macOS Intel:

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.darwin-amd64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

Linux:

mkdir -p lib
curl -L -o lib/libtokenizers.tar.gz \
  "https://github.com/daulet/tokenizers/releases/latest/download/libtokenizers.linux-amd64.tar.gz"
tar -xzf lib/libtokenizers.tar.gz -C lib && rm lib/libtokenizers.tar.gz

See github.com/daulet/tokenizers for other platforms.

Step 3: Download Model Files

Option A - Using curl (~90MB):

mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model.onnx"

Option B - Quantized model (~25MB, slightly lower quality):

mkdir -p models
curl -L -o models/tokenizer.json \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/tokenizer.json"
curl -L -o models/model.onnx \
  "https://huggingface.co/sentence-transformers/all-MiniLM-L6-v2/resolve/main/onnx/model_quantized.onnx"

Option C - Using Go (requires native libs installed first):

import "github.com/abdul-hamid-achik/veclite/embed/onnx"

// Full model
err := onnx.DownloadMiniLM("./models")

// Or quantized
err := onnx.DownloadMiniLMQuantized("./models")

Step 4: Build

# Set library path and build
CGO_LDFLAGS="-L./lib" go build -tags onnx ./...
Running Tests
# Run all ONNX tests
CGO_LDFLAGS="-L./lib" VECLITE_ONNX_MODEL_DIR=./models \
  go test -tags onnx -v ./embed/onnx/...

# Run benchmarks
CGO_LDFLAGS="-L./lib" VECLITE_ONNX_MODEL_DIR=./models \
  go test -tags onnx -bench=. -benchmem ./embed/onnx/...
Usage
import (
    "github.com/abdul-hamid-achik/veclite"
    "github.com/abdul-hamid-achik/veclite/embed/onnx"
)

// Create ONNX embedder
embedder, err := onnx.NewMiniLM("./models")
if err != nil {
    log.Fatal(err)
}
defer embedder.Close()

// Use with veclite
db, _ := veclite.Open("data.veclite")
coll, _ := db.CreateCollection("docs",
    veclite.WithDimension(384),  // MiniLM outputs 384 dimensions
    veclite.WithEmbedder(embedder),
)

// Insert and search by text
id, _ := coll.InsertText("Go is a statically typed language", nil)
results, _ := coll.SearchText("programming languages", veclite.TopK(5))
Batch Embedding

For better performance with multiple texts, use batch embedding:

texts := []string{
    "First document",
    "Second document",
    "Third document",
}

vectors, err := embedder.EmbedBatch(texts)
// Then insert with InsertBatch
Performance

Benchmarks on Apple M5:

Operation Time Throughput
Single embed ~12ms ~83 texts/sec
Batch 10 ~100ms ~100 texts/sec
Batch 100 ~875ms ~114 texts/sec

Batching improves throughput by ~37% compared to single embedding.

Custom Models

You can use other ONNX-exported sentence transformers:

embedder, err := onnx.NewEmbedder(
    "/path/to/model.onnx",
    "/path/to/tokenizer.json",
    onnx.WithDimension(768),    // For larger models
    onnx.WithMaxLength(512),    // Max token sequence length
)
Troubleshooting

"library 'tokenizers' not found"

# Ensure CGO_LDFLAGS points to the lib directory
CGO_LDFLAGS="-L/full/path/to/lib" go build -tags onnx ./...

"onnxruntime.so not found"

# The library auto-detects common paths. If not found, set manually:
export ONNXRUNTIME_LIB=/opt/homebrew/lib/libonnxruntime.dylib

"model.onnx not found"

# Ensure VECLITE_ONNX_MODEL_DIR points to the models directory
VECLITE_ONNX_MODEL_DIR=/full/path/to/models go test -tags onnx ./embed/onnx/...
HNSW Configuration

HNSW (Hierarchical Navigable Small World) provides approximate nearest neighbor search:

// Basic
coll, _ := db.CreateCollection("vectors",
    veclite.WithHNSW(16, 200),
)

// Full configuration
coll, _ := db.CreateCollection("vectors",
    veclite.WithHNSWConfig(veclite.HNSWConfig{
        M:              32,
        EfConstruction: 400,
        EfSearch:       100,
    }),
)
Parameter Default Range Trade-off
M 16 12-48 Higher = better recall, more memory
EfConstruction 200 100-500 Higher = better index quality, slower build
EfSearch 100 50-500 Higher = better recall, slower search

Without HNSW, search uses brute-force linear scan (exact results, slower on large datasets).

Observability
Logger

Implement the Logger interface to integrate with your logging library:

type Logger interface {
    Debug(msg string, keysAndValues ...any)
    Info(msg string, keysAndValues ...any)
    Error(msg string, keysAndValues ...any)
}

NopLogger is the default (zero overhead). Set a logger when opening the database:

db, _ := veclite.Open("data.veclite", veclite.WithLogger(myLogger))
Metrics

VecLite tracks operation counts and latency using atomic counters:

snapshot := db.Metrics()
fmt.Printf("Searches: %d, Inserts: %d, Deletes: %d, Avg Search: %v\n",
    snapshot.SearchCount,
    snapshot.InsertCount,
    snapshot.DeleteCount,
    snapshot.AvgSearchTime,
)

MetricsSnapshot fields:

Field Type Description
SearchCount int64 Total search operations
InsertCount int64 Total insert operations
DeleteCount int64 Total delete operations
AvgSearchTime time.Duration Average search latency
Statistics
// Database stats
dbStats := db.Stats()
fmt.Printf("Collections: %d, Total Records: %d\n",
    dbStats.Collections, dbStats.TotalRecords)

// Collection stats
collStats := coll.Stats()
fmt.Printf("Count: %d, Dimension: %d, Index: %s\n",
    collStats.Count, collStats.Dimension, collStats.IndexType)

CLI Usage

veclite <command> [arguments]
Commands
Read Commands
Command Description
version Show version information
info <file> Show database summary
collections <file> List all collections
stats <file> Show detailed statistics
dump <file> Export database as JSON
get <file> <collection> Get a vector by ID
Write Commands
Command Description
create-collection <file> <name> Create a new collection
drop-collection <file> <name> Drop a collection
insert <file> <collection> Insert a vector
batch-insert <file> <collection> Insert vectors from JSON file
delete <file> <collection> Delete a vector by ID
upsert <file> <collection> Insert or update a vector
update <file> <collection> Update payload for a record
delete-where <file> <collection> Delete records matching a filter
find <file> <collection> Find records by filter
search <file> <collection> Search for similar vectors
Server Mode
Command Description
serve <file> Start HTTP server
mcp <file> Start MCP tool server over stdio
Maintenance
Command Description
compact <file> Compact database and reclaim space
validate <file> Validate database integrity
benchmark <file> Run search performance benchmark

All commands support --json for JSON output.

CLI Examples
# Create a collection with HNSW index and text search
veclite create-collection data.veclite embeddings \
    --dimension=384 --distance=cosine --hnsw --text-index=title,body

# Insert a vector
veclite insert data.veclite embeddings \
    --vector='[0.1,0.2,0.3,...]' --payload='{"file":"main.go"}'

# Batch insert from JSON
veclite batch-insert data.veclite embeddings --input=vectors.json

# Search
veclite search data.veclite embeddings \
    --query='[0.1,0.2,0.3,...]' --top-k=10 --filter='type=code'

# Upsert
veclite upsert data.veclite embeddings \
    --id=42 --vector='[0.1,0.2,0.3,...]' --payload='{"file":"main.go"}'

# Find by filter
veclite find data.veclite embeddings --filter='type=code'

# Delete by filter
veclite delete-where data.veclite embeddings --filter='type=temp'

# Start HTTP server
veclite serve data.veclite --port=8080 --cors

# Start MCP server
veclite mcp data.veclite
Batch Insert File Format

JSON Array:

[
  {"vector": [0.1, 0.2, 0.3], "payload": {"file": "a.go"}},
  {"vector": [0.4, 0.5, 0.6], "payload": {"file": "b.go"}}
]

JSONL (one object per line):

{"vector": [0.1, 0.2, 0.3], "payload": {"file": "a.go"}}
{"vector": [0.4, 0.5, 0.6], "payload": {"file": "b.go"}}

HTTP Server

veclite serve data.veclite --port=8080 --host=127.0.0.1 --cors
Endpoints
Method Endpoint Description
GET /health Health check
GET /info Database info
GET /metrics Operation metrics
GET /collections List collections
POST /collections Create collection
GET /collections/{name} Collection info
DELETE /collections/{name} Drop collection
GET /collections/{name}/vectors List all vectors (paginated)
POST /collections/{name}/vectors Insert vector(s)
GET /collections/{name}/vectors/{id} Get vector by ID
PUT /collections/{name}/vectors/{id} Update vector and/or payload
DELETE /collections/{name}/vectors/{id} Delete vector
DELETE /collections/{name}/vectors Delete by filter (with body)
POST /collections/{name}/search Search vectors
POST /collections/{name}/upsert Upsert vector
POST /collections/{name}/find Find records by filter
POST /collections/{name}/compact Compact collection
POST /collections/{name}/validate Validate integrity
POST /sync Force sync to disk
Streaming Results

Set Accept: application/x-ndjson on search requests to receive results as newline-delimited JSON:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Accept: application/x-ndjson" \
  -H "Content-Type: application/json" \
  -d '{"query": [0.1, 0.2, 0.3], "top_k": 100}'

Each line is a JSON object:

{"id":1,"score":0.9821,"payload":{"file":"main.go"}}
{"id":5,"score":0.9654,"payload":{"file":"util.go"}}
API Examples

Create Collection:

curl -X POST http://localhost:8080/collections \
  -H "Content-Type: application/json" \
  -d '{"name": "embeddings", "dimension": 384, "distance": "cosine", "hnsw": true}'

Insert Vector:

curl -X POST http://localhost:8080/collections/embeddings/vectors \
  -H "Content-Type: application/json" \
  -d '{"vector": [0.1, 0.2, 0.3], "payload": {"file": "main.go"}}'

Batch Insert:

curl -X POST http://localhost:8080/collections/embeddings/vectors \
  -H "Content-Type: application/json" \
  -d '{"vectors": [[0.1,0.2,0.3], [0.4,0.5,0.6]], "payloads": [{"file":"a.go"}, {"file":"b.go"}]}'

Search:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Content-Type: application/json" \
  -d '{"query": [0.1, 0.2, 0.3], "top_k": 10}'

Search with Filters:

curl -X POST http://localhost:8080/collections/embeddings/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": [0.1, 0.2, 0.3],
    "top_k": 10,
    "filters": [{"key": "type", "op": "eq", "value": "code"}]
  }'

Upsert:

curl -X POST http://localhost:8080/collections/embeddings/upsert \
  -H "Content-Type: application/json" \
  -d '{"id": 42, "vector": [0.1, 0.2, 0.3], "payload": {"file": "main.go"}}'

Find by Filter:

curl -X POST http://localhost:8080/collections/embeddings/find \
  -H "Content-Type: application/json" \
  -d '{"filters": [{"key": "type", "op": "eq", "value": "code"}]}'

Update Vector:

curl -X PUT http://localhost:8080/collections/embeddings/vectors/42 \
  -H "Content-Type: application/json" \
  -d '{"vector": [0.4, 0.5, 0.6], "payload": {"file": "updated.go"}}'
Filter Operators
Operator Aliases Description
eq = Equal
neq != Not equal
gt > Greater than (numeric)
gte >= Greater than or equal
lt < Less than (numeric)
lte <= Less than or equal
glob Glob pattern match
prefix String prefix
suffix String suffix
contains String contains
exists Key exists
Python Client Example
import requests

base = "http://localhost:8080"

# Create collection
requests.post(f"{base}/collections", json={
    "name": "embeddings", "dimension": 384, "hnsw": True
})

# Insert
r = requests.post(f"{base}/collections/embeddings/vectors", json={
    "vector": [0.1] * 384,
    "payload": {"file": "main.py"}
})
print(r.json())  # {"status": "inserted", "id": 1}

# Search
r = requests.post(f"{base}/collections/embeddings/search", json={
    "query": [0.1] * 384, "top_k": 5
})
print(r.json())  # {"results": [...], "count": 5}

MCP Tool Server

VecLite can run as an MCP tool server, making it available to AI agents like Claude Code, Cursor, and other MCP-compatible clients.

veclite mcp data.veclite

The server communicates over stdio using JSON-RPC 2.0. It exposes these tools:

Tool Description
veclite_collections List all collections
veclite_stats Get collection statistics
veclite_search Vector similarity search
veclite_text_search BM25 full-text search
veclite_hybrid_search Combined vector + text search
veclite_find Find records by filter
veclite_insert Insert a vector
MCP Configuration

Add to your MCP client configuration (e.g., .claude/settings.json):

{
  "mcpServers": {
    "veclite": {
      "command": "veclite",
      "args": ["mcp", "/path/to/data.veclite"]
    }
  }
}

Examples

Runnable examples are in the examples/ directory:

go run ./examples/basic        # Open, insert, search, close
go run ./examples/hnsw         # HNSW index configuration and benchmarking
go run ./examples/filtering    # Rich filter expressions
go run ./examples/batch        # Batch operations, upsert, iteration, pagination
go run ./examples/http-client  # HTTP API client (start server first)

All examples use :memory: for zero-setup running (except http-client, which connects to a running server).

Performance

Benchmark results on 10,000 384-dimensional vectors:

Method Time Speedup
Brute Force ~2.5ms 1x
HNSW ~0.4ms 6x

HNSW provides >95% recall at 6-7x speedup over brute force.

Thread Safety

VecLite is safe for concurrent access:

  • Multiple goroutines can read simultaneously
  • Writes are serialized with sync.RWMutex
  • Metrics use atomic counters for lock-free reads
  • Use WithSyncOnWrite(true) for durability after each write

Persistence

Data is stored using Go's gob encoding in a single .veclite file:

  • Call db.Sync() to persist changes manually
  • Use WithSyncOnWrite(true) for automatic persistence after each write
  • db.Close() syncs before closing
  • HNSW indexes, BM25 inverted indexes, and record content are all persisted
  • File locking prevents concurrent process access
  • CRC32 checksums validate data integrity on load

Contributing

Contributions welcome. Please:

  1. Run go test -race ./... before submitting
  2. Add tests for new features
  3. Follow existing code style

License

MIT License - see LICENSE

Documentation

Overview

Package veclite provides an embeddable vector database with zero external dependencies. It stores vectors with metadata in a single file using gob encoding.

Basic usage: 

db, err := veclite.Open("data.veclite")
if err != nil {
    log.Fatal(err)
}
defer db.Close()

coll := db.Collection("embeddings")
id, err := coll.Insert(vector, map[string]any{"file": "main.go"})

results, err := coll.Search(queryVector, veclite.TopK(10))

Index

Constants

View Source 
const (
	// DistanceCosine uses cosine similarity (higher = more similar).
	DistanceCosine = floats.DistanceCosine
	// DistanceDot uses dot product (higher = more similar).
	DistanceDot = floats.DistanceDot
	// DistanceEuclidean uses Euclidean distance (lower = more similar).
	DistanceEuclidean = floats.DistanceEuclidean
)
View Source 
const Version = "0.2.0"

Version is the library version.

Variables

View Source 
var (
	// ErrNotFound is returned when a record or collection is not found.
	ErrNotFound = errors.New("veclite: not found")

	// ErrDimensionMismatch is returned when vector dimensions don't match.
	ErrDimensionMismatch = errors.New("veclite: dimension mismatch")

	// ErrEmptyVector is returned when an empty vector is provided.
	ErrEmptyVector = errors.New("veclite: empty vector")

	// ErrCollectionExists is returned when trying to create a collection that already exists.
	ErrCollectionExists = errors.New("veclite: collection already exists")

	// ErrDatabaseClosed is returned when operations are attempted on a closed database.
	ErrDatabaseClosed = errors.New("veclite: database closed")

	// ErrInvalidPath is returned when an invalid file path is provided.
	ErrInvalidPath = errors.New("veclite: invalid path")

	// ErrCorruptedFile is returned when the database file is corrupted.
	ErrCorruptedFile = errors.New("veclite: corrupted file")

	// ErrInvalidVersion is returned when the file version is not supported.
	ErrInvalidVersion = errors.New("veclite: unsupported file version")

	// ErrBatchSizeMismatch is returned when batch operation input sizes don't match.
	ErrBatchSizeMismatch = errors.New("veclite: batch size mismatch")

	// ErrReadOnly is returned when a write operation is attempted on a read-only database.
	ErrReadOnly = errors.New("veclite: database is read-only")
)

Sentinel errors for common conditions.

View Source 
var ErrChecksumMismatch = errors.New("veclite: checksum mismatch")

ErrChecksumMismatch is returned when the file checksum does not match.

View Source 
var ErrFileLocked = errors.New("veclite: database file is locked by another process")

ErrFileLocked is returned when the database file is already locked by another process.

View Source 
var ErrNoEmbedder = errors.New("veclite: no embedder configured")

ErrNoEmbedder is returned when an embedding operation is attempted without an embedder configured on the collection.

Functions

This section is empty.

Types

type Collection 

type Collection struct {
	// contains filtered or unexported fields
}

Collection represents a collection of vectors with the same dimension.

func (*Collection) All 

func (c *Collection) All() []*Record

All returns all records in the collection.

func (*Collection) Clear 

func (c *Collection) Clear() error

Clear removes all records from the collection. Returns an error if the database is read-only.

func (*Collection) Count 

func (c *Collection) Count() int

Count returns the number of records in the collection.

func (*Collection) Delete 

func (c *Collection) Delete(id uint64) error

Delete removes a record by ID.

func (*Collection) DeleteWhere 

func (c *Collection) DeleteWhere(filters ...Filter) (int, error)

DeleteWhere removes all records matching the filters. Returns the number of deleted records.

func (*Collection) Dimension 

func (c *Collection) Dimension() int

Dimension returns the vector dimension. Returns 0 if no vectors have been inserted yet.

func (*Collection) DistanceType 

func (c *Collection) DistanceType() floats.DistanceType

DistanceType returns the distance metric type.

func (*Collection) Find 

func (c *Collection) Find(filters ...Filter) ([]*Record, error)

Find retrieves all records matching the filters.

func (*Collection) FindOne 

func (c *Collection) FindOne(filters ...Filter) (*Record, error)

FindOne retrieves the first record matching the filters.

func (*Collection) ForEach added in v0.6.0 

func (c *Collection) ForEach(fn func(*Record) bool)

ForEach iterates over all records in the collection, calling fn for each. If fn returns false, iteration stops early. Records are cloned before being passed to fn.

func (*Collection) Get 

func (c *Collection) Get(id uint64) (*Record, error)

Get retrieves a record by ID.

func (*Collection) GetVector 

func (c *Collection) GetVector(id uint64) ([]float32, error)

GetVector retrieves just the vector for a record.

func (*Collection) HasIndex added in v0.2.0 

func (c *Collection) HasIndex() bool

HasIndex returns true if this collection has an index.

func (*Collection) HybridSearch added in v0.6.0 

func (c *Collection) HybridSearch(query []float32, text string, opts ...SearchOption) ([]Result, error)

HybridSearch performs both vector search and BM25 text search, then fuses results using Reciprocal Rank Fusion (RRF) with k=60. Requires text indexing to be enabled via WithTextIndex. Use WithVectorWeight and WithTextWeight to control the balance.

func (*Collection) IndexStats added in v0.2.0 

func (c *Collection) IndexStats() *hnsw.IndexStats

IndexStats returns statistics about the collection's index. Returns nil if no index is configured.

func (*Collection) IndexType added in v0.2.0 

func (c *Collection) IndexType() IndexType

IndexType returns the index type for this collection.

func (*Collection) Insert 

func (c *Collection) Insert(vector []float32, payload map[string]any) (uint64, error)

Insert adds a vector with optional payload to the collection. Returns the assigned record ID.

func (*Collection) InsertBatch 

func (c *Collection) InsertBatch(vectors [][]float32, payloads []map[string]any) ([]uint64, error)

InsertBatch adds multiple vectors with payloads to the collection. Returns the assigned record IDs. If payloads is nil or shorter than vectors, missing payloads are treated as nil.

func (*Collection) InsertDocument added in v0.6.0 

func (c *Collection) InsertDocument(vector []float32, content string, payload map[string]any) (uint64, error)

InsertDocument inserts a vector with content text and payload. Content is automatically indexed for BM25 text search when text indexing is enabled.

func (*Collection) InsertText added in v0.6.0 

func (c *Collection) InsertText(text string, payload map[string]any) (uint64, error)

InsertText embeds the text using the configured embedder and inserts the result. Requires an embedder to be set via WithEmbedder.

func (*Collection) Iterate added in v0.6.0 

func (c *Collection) Iterate(opts ...IterOption) *Iterator

Iterate returns an iterator over collection records. Options can control offset and limit for pagination.

func (*Collection) Name 

func (c *Collection) Name() string

Name returns the collection name.

func (*Collection) Search 

func (c *Collection) Search(query []float32, opts ...SearchOption) ([]Result, error)

Search finds the most similar vectors to the query vector.

func (*Collection) SearchExplain added in v0.2.0 

func (c *Collection) SearchExplain(query []float32, opts ...SearchOption) (*SearchExplanation, error)

SearchExplain performs a search and returns detailed statistics.

func (*Collection) SearchStream added in v0.6.0 

func (c *Collection) SearchStream(query []float32, fn SearchFunc, opts ...SearchOption) error

SearchStream performs a search and streams results to the callback function. The callback receives results one at a time and can return false to stop early.

func (*Collection) SearchText added in v0.6.0 

func (c *Collection) SearchText(text string, opts ...SearchOption) ([]Result, error)

SearchText embeds the text query using the configured embedder and searches. Requires an embedder to be set via WithEmbedder.

func (*Collection) Stats 

func (c *Collection) Stats() CollectionStats

Stats returns statistics about the collection.

func (*Collection) TextSearch added in v0.6.0 

func (c *Collection) TextSearch(query string, opts ...SearchOption) ([]Result, error)

TextSearch performs BM25 full-text search over indexed fields. Requires text indexing to be enabled via WithTextIndex.

func (*Collection) Update 

func (c *Collection) Update(id uint64, payload map[string]any) error

Update updates the payload for a record.

func (*Collection) UpdateVector added in v0.4.0 

func (c *Collection) UpdateVector(id uint64, vector []float32) error

UpdateVector updates the vector for a record.

func (*Collection) Upsert added in v0.4.0 

func (c *Collection) Upsert(id uint64, vector []float32, payload map[string]any) (uint64, error)

Upsert inserts a new record or updates an existing one by ID. If the ID is 0, a new record is created with an auto-generated ID. If the ID exists, the vector and payload are updated. If the ID doesn't exist, a new record is created with that ID. Returns the record ID (either the provided one or newly generated).

func (*Collection) UpsertByKey added in v0.4.0 

func (c *Collection) UpsertByKey(keyField string, keyValue any, vector []float32, payload map[string]any) (uint64, bool, error)

UpsertByKey inserts a new record or updates an existing one based on a key field. If a record with payload[keyField] == keyValue exists, it is updated. Otherwise, a new record is inserted. Returns the record ID and whether it was an insert (true) or update (false).

type CollectionOption 

type CollectionOption interface {
	// contains filtered or unexported methods
}

CollectionOption configures a collection.

func WithDimension 

func WithDimension(dim int) CollectionOption

WithDimension sets the vector dimension for the collection. If set, all vectors must match this dimension. If not set (0), the dimension is determined by the first insert.

func WithDistanceType 

func WithDistanceType(t floats.DistanceType) CollectionOption

WithDistanceType sets the distance metric for the collection. Default is cosine similarity.

func WithEmbedder added in v0.6.0 

func WithEmbedder(e Embedder) CollectionOption

WithEmbedder sets an auto-embedding plugin for the collection. When set, InsertText and SearchText methods become available.

func WithHNSW added in v0.2.0 

func WithHNSW(m, efConstruction int) CollectionOption

WithHNSW enables HNSW indexing for the collection. m is the maximum number of connections per node (default: 16, recommended: 12-48). efConstruction is the candidate list size during construction (default: 200).

func WithHNSWConfig added in v0.2.0 

func WithHNSWConfig(config HNSWConfig) CollectionOption

WithHNSWConfig enables HNSW indexing with custom configuration.

func WithTextIndex added in v0.6.0 

func WithTextIndex(fields ...string) CollectionOption

WithTextIndex enables BM25 full-text indexing on the specified payload fields. When enabled, string values in these fields are tokenized and indexed for text search. The Content field of records is always indexed when text indexing is enabled.

type CollectionSnapshot 

type CollectionSnapshot struct {
	// Name is the collection name.
	Name string

	// Dimension is the vector dimension.
	Dimension int

	// DistanceType is the distance metric.
	DistanceType floats.DistanceType

	// NextID is the next record ID to assign.
	NextID uint64

	// Records contains all records in the collection.
	Records []*RecordSnapshot

	// CreatedAt is when the collection was created.
	CreatedAt time.Time

	// UpdatedAt is when the collection was last modified.
	UpdatedAt time.Time

	// IndexType is the type of index (none, hnsw).
	IndexType IndexType

	// HNSWConfig holds the HNSW configuration (if IndexType is hnsw).
	HNSWConfig *HNSWConfig

	// HNSWSnapshot holds the HNSW index state (if IndexType is hnsw).
	HNSWSnapshot *hnsw.Snapshot

	// TextIndexSnapshot holds the BM25 text index state (if text indexing is enabled).
	TextIndexSnapshot *InvertedIndexSnapshot
}

CollectionSnapshot is the serializable state of a collection.

func NewCollectionSnapshot 

func NewCollectionSnapshot(name string, dimension int, distanceType floats.DistanceType) *CollectionSnapshot

NewCollectionSnapshot creates a new empty collection snapshot.

type CollectionStats 

type CollectionStats struct {
	// Name is the collection name.
	Name string

	// Count is the number of records in the collection.
	Count int

	// Dimension is the vector dimension (0 if not yet set).
	Dimension int

	// DistanceType is the distance metric used.
	DistanceType string

	// IndexType is the index type (none, hnsw).
	IndexType string
}

CollectionStats contains statistics about a collection.

type DB 

type DB struct {
	// contains filtered or unexported fields
}

DB represents a VecLite database.

func Open 

func Open(path string, opts ...Option) (*DB, error)

Open opens or creates a VecLite database at the given path. Use ":memory:" for an in-memory database that won't be persisted.

func (*DB) Close 

func (db *DB) Close() error

Close closes the database, syncing any pending changes.

func (*DB) Collection 

func (db *DB) Collection(name string) *Collection

Collection returns a collection by name, creating it if it doesn't exist. This is the preferred way to get collections for most use cases. In read-only mode, returns nil if the collection doesn't exist (cannot create).

func (*DB) Collections 

func (db *DB) Collections() []string

Collections returns the names of all collections.

func (*DB) CreateCollection 

func (db *DB) CreateCollection(name string, opts ...CollectionOption) (*Collection, error)

CreateCollection creates a new collection with the given options. Returns an error if the collection already exists.

func (*DB) DropCollection 

func (db *DB) DropCollection(name string) error

DropCollection removes a collection and all its data.

func (*DB) GetCollection 

func (db *DB) GetCollection(name string) (*Collection, error)

GetCollection returns an existing collection or ErrNotFound.

func (*DB) HasCollection 

func (db *DB) HasCollection(name string) bool

HasCollection returns true if a collection exists.

func (*DB) IsClosed 

func (db *DB) IsClosed() bool

IsClosed returns true if the database is closed.

func (*DB) Metrics added in v0.6.0 

func (db *DB) Metrics() MetricsSnapshot

Metrics returns the current metrics snapshot.

func (*DB) Path 

func (db *DB) Path() string

Path returns the database file path.

func (*DB) Stats 

func (db *DB) Stats() DatabaseStats

Stats returns statistics about the database.

func (*DB) Sync 

func (db *DB) Sync() error

Sync writes all pending changes to storage.

type DatabaseSnapshot 

type DatabaseSnapshot struct {
	// Version is the file format version.
	Version uint32

	// Collections maps collection names to their snapshots.
	Collections map[string]*CollectionSnapshot

	// CreatedAt is when the database was created.
	CreatedAt time.Time

	// UpdatedAt is when the database was last modified.
	UpdatedAt time.Time
}

DatabaseSnapshot is the serializable state of the database.

func NewDatabaseSnapshot 

func NewDatabaseSnapshot() *DatabaseSnapshot

NewDatabaseSnapshot creates a new empty database snapshot.

type DatabaseStats 

type DatabaseStats struct {
	// Path is the database file path (":memory:" for in-memory).
	Path string

	// Collections is the number of collections.
	Collections int

	// TotalRecords is the total number of records across all collections.
	TotalRecords int

	// CollectionStats contains stats for each collection.
	CollectionStats []CollectionStats
}

DatabaseStats contains statistics about the database.

type DimensionError 

type DimensionError struct {
	Expected int
	Got      int
}

DimensionError provides details about dimension mismatches.

func (*DimensionError) Error 

func (e *DimensionError) Error() string

func (*DimensionError) Unwrap 

func (e *DimensionError) Unwrap() error

type DistanceType added in v0.2.0 

type DistanceType = floats.DistanceType

Re-export distance types for external use.

type Embedder added in v0.6.0 

type Embedder interface {
	// Embed converts a single text into a vector embedding.
	Embed(text string) ([]float32, error)

	// EmbedBatch converts multiple texts into vector embeddings.
	EmbedBatch(texts []string) ([][]float32, error)

	// Dimension returns the output vector dimension.
	Dimension() int
}

Embedder is the interface for auto-embedding text to vectors. Implementations live in separate modules to maintain zero-dependency core.

type FileStorage 

type FileStorage struct {
	// contains filtered or unexported fields
}

FileStorage is a file-based storage implementation. Uses gob encoding with atomic writes for durability. Acquires an exclusive file lock to prevent concurrent access from multiple processes.

func NewFileStorage 

func NewFileStorage(path string) *FileStorage

NewFileStorage creates a new file storage for the given path.

func (*FileStorage) Close 

func (f *FileStorage) Close() error

Close releases the file lock.

func (*FileStorage) Delete 

func (f *FileStorage) Delete() error

Delete removes the database file and any backup/lock files.

func (*FileStorage) Exists 

func (f *FileStorage) Exists() bool

Exists returns true if the database file exists.

func (*FileStorage) Load 

func (f *FileStorage) Load() (*DatabaseSnapshot, error)

Load reads the database from the file. Returns nil, nil if the file doesn't exist yet.

func (*FileStorage) Lock added in v0.5.0 

func (f *FileStorage) Lock() error

Lock acquires an exclusive file lock on a .lock file adjacent to the database. This prevents multiple processes from opening the same database.

func (*FileStorage) Path 

func (f *FileStorage) Path() string

Path returns the file path.

func (*FileStorage) Save 

func (f *FileStorage) Save(snapshot *DatabaseSnapshot) error

Save writes the database to the file using atomic write pattern. Writes to .tmp file, fsyncs, then renames old to .bak, then renames .tmp to final.

func (*FileStorage) Unlock added in v0.5.0 

func (f *FileStorage) Unlock() error

Unlock releases the file lock.

type Filter 

type Filter interface {
	// Match returns true if the record matches the filter criteria.
	Match(r *Record) bool
}

Filter is an interface for filtering records based on payload values.

func And 

func And(filters ...Filter) Filter

And creates a filter that matches records matching all given filters.

func Between added in v0.4.0 

func Between(key string, min, max float64) Filter

Between creates a filter that matches records where min <= payload[key] <= max.

func Contains 

func Contains(key, substr string) Filter

Contains creates a filter that matches records where payload[key] contains the substring.

func Equal 

func Equal(key string, value any) Filter

Equal creates a filter that matches records where payload[key] equals value.

func Exists 

func Exists(key string) Filter

Exists creates a filter that matches records where payload[key] exists.

func GT added in v0.4.0 

func GT(key string, value float64) Filter

GT is an alias for GreaterThan.

func GTE added in v0.4.0 

func GTE(key string, value float64) Filter

GTE is an alias for GreaterThanOrEqual.

func Glob 

func Glob(key, pattern string) Filter

Glob creates a filter that matches records where payload[key] matches the glob pattern.

func GreaterThan added in v0.4.0 

func GreaterThan(key string, value float64) Filter

GreaterThan creates a filter that matches records where payload[key] > value.

func GreaterThanOrEqual added in v0.4.0 

func GreaterThanOrEqual(key string, value float64) Filter

GreaterThanOrEqual creates a filter that matches records where payload[key] >= value.

func In 

func In(key string, values ...any) Filter

In creates a filter that matches records where payload[key] is in the given values.

func LT added in v0.4.0 

func LT(key string, value float64) Filter

LT is an alias for LessThan.

func LTE added in v0.4.0 

func LTE(key string, value float64) Filter

LTE is an alias for LessThanOrEqual.

func LessThan added in v0.4.0 

func LessThan(key string, value float64) Filter

LessThan creates a filter that matches records where payload[key] < value.

func LessThanOrEqual added in v0.4.0 

func LessThanOrEqual(key string, value float64) Filter

LessThanOrEqual creates a filter that matches records where payload[key] <= value.

func Not 

func Not(filter Filter) Filter

Not creates a filter that negates the given filter.

func NotEqual 

func NotEqual(key string, value any) Filter

NotEqual creates a filter that matches records where payload[key] does not equal value.

func NotIn 

func NotIn(key string, values ...any) Filter

NotIn creates a filter that matches records where payload[key] is not in the given values.

func Or 

func Or(filters ...Filter) Filter

Or creates a filter that matches records matching any given filter.

func Prefix 

func Prefix(key, prefix string) Filter

Prefix creates a filter that matches records where payload[key] has the given prefix.

func Suffix 

func Suffix(key, suffix string) Filter

Suffix creates a filter that matches records where payload[key] has the given suffix.

type FilterFunc 

type FilterFunc func(r *Record) bool

FilterFunc is a function adapter for the Filter interface.

func (FilterFunc) Match 

func (f FilterFunc) Match(r *Record) bool

Match implements Filter interface.

type HNSWConfig added in v0.2.0 

type HNSWConfig struct {
	// M is the maximum number of connections per node.
	M int
	// EfConstruction is the size of the candidate list during index construction.
	EfConstruction int
	// EfSearch is the default size of the candidate list during search.
	EfSearch int
}

HNSWConfig holds HNSW index configuration.

type HNSWIndex added in v0.2.0 

type HNSWIndex struct {
	// contains filtered or unexported fields
}

HNSWIndex wraps the HNSW index to implement the Index interface.

func NewHNSWIndex added in v0.2.0 

func NewHNSWIndex(dimension int, distanceType floats.DistanceType, m, efConstruction int) *HNSWIndex

NewHNSWIndex creates a new HNSW index with the given parameters.

func NewHNSWIndexWithConfig added in v0.2.0 

func NewHNSWIndexWithConfig(dimension int, distanceType floats.DistanceType, config hnsw.Config) *HNSWIndex

NewHNSWIndexWithConfig creates a new HNSW index with a custom configuration.

func (*HNSWIndex) Count added in v0.2.0 

func (h *HNSWIndex) Count() int

Count returns the number of vectors in the index.

func (*HNSWIndex) Delete added in v0.2.0 

func (h *HNSWIndex) Delete(id uint64) error

Delete removes a vector from the index (soft delete).

func (*HNSWIndex) HardDelete added in v0.4.0 

func (h *HNSWIndex) HardDelete(id uint64) error

HardDelete removes a vector completely from the index. This is needed for update operations where we re-insert with the same ID.

func (*HNSWIndex) Insert added in v0.2.0 

func (h *HNSWIndex) Insert(id uint64, vector []float32) error

Insert adds a vector with the given ID to the index.

func (*HNSWIndex) Internal added in v0.2.0 

func (h *HNSWIndex) Internal() *hnsw.Index

Internal returns the underlying HNSW index (for serialization).

func (*HNSWIndex) Search added in v0.2.0 

func (h *HNSWIndex) Search(query []float32, k int) ([]IndexResult, error)

Search finds the k nearest neighbors to the query vector.

func (*HNSWIndex) SearchWithEf added in v0.2.0 

func (h *HNSWIndex) SearchWithEf(query []float32, k int, ef int) ([]IndexResult, error)

SearchWithEf searches with a custom ef parameter.

func (*HNSWIndex) SetInternal added in v0.2.0 

func (h *HNSWIndex) SetInternal(idx *hnsw.Index)

SetInternal sets the underlying HNSW index (for deserialization).

func (*HNSWIndex) Stats added in v0.2.0 

func (h *HNSWIndex) Stats() hnsw.IndexStats

Stats returns statistics about the index.

func (*HNSWIndex) Type added in v0.2.0 

func (h *HNSWIndex) Type() string

Type returns "hnsw".

type Index added in v0.2.0 

type Index interface {
	// Insert adds a vector with the given ID to the index.
	Insert(id uint64, vector []float32) error

	// Delete removes a vector from the index.
	Delete(id uint64) error

	// Search finds the k nearest neighbors to the query vector.
	// Returns IDs and distances/similarities.
	Search(query []float32, k int) ([]IndexResult, error)

	// SearchWithEf searches with a custom ef parameter (for HNSW).
	// For indexes that don't support ef, this should behave like Search.
	SearchWithEf(query []float32, k int, ef int) ([]IndexResult, error)

	// Count returns the number of vectors in the index.
	Count() int

	// Type returns the index type name.
	Type() string
}

Index is the interface for vector search indexes. Implementations can provide different algorithms (brute-force, HNSW, etc.).

type IndexResult added in v0.2.0 

type IndexResult struct {
	ID       uint64
	Distance float32
}

IndexResult represents a search result from an index.

type IndexType added in v0.2.0 

type IndexType string

IndexType represents the type of index. 

const (
	// IndexTypeNone means no index (brute force search).
	IndexTypeNone IndexType = "none"

	// IndexTypeHNSW means HNSW approximate nearest neighbor index.
	IndexTypeHNSW IndexType = "hnsw"
)

type InvertedIndexSnapshot added in v0.6.0 

type InvertedIndexSnapshot struct {
	Postings    map[string][]uint64
	DocLengths  map[uint64]int
	TotalDocLen int64
	DocCount    int
	Fields      []string
}

InvertedIndexSnapshot is the serializable state of the inverted index.

type IterOption added in v0.6.0 

type IterOption interface {
	// contains filtered or unexported methods
}

IterOption configures the iterator.

func IterLimit added in v0.6.0 

func IterLimit(n int) IterOption

IterLimit sets the maximum number of records to return.

func IterOffset added in v0.6.0 

func IterOffset(n int) IterOption

IterOffset sets the number of records to skip.

type Iterator added in v0.6.0 

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator allows iterating over collection records one at a time.

func (*Iterator) Close added in v0.6.0 

func (it *Iterator) Close()

Close releases resources held by the iterator.

func (*Iterator) Next added in v0.6.0 

func (it *Iterator) Next() (*Record, bool)

Next returns the next record and true, or nil and false if done.

type Logger added in v0.6.0 

type Logger interface {
	// Debug logs a debug message with key-value pairs.
	Debug(msg string, keysAndValues ...any)

	// Info logs an informational message with key-value pairs.
	Info(msg string, keysAndValues ...any)

	// Error logs an error message with key-value pairs.
	Error(msg string, keysAndValues ...any)
}

Logger is the interface for structured logging in VecLite. Implementations can bridge to any logging library (slog, zap, zerolog, etc.).

type MemoryStorage 

type MemoryStorage struct {
	// contains filtered or unexported fields
}

MemoryStorage is an in-memory storage implementation. Data is not persisted and will be lost when the database is closed.

func NewMemoryStorage 

func NewMemoryStorage() *MemoryStorage

NewMemoryStorage creates a new in-memory storage.

func (*MemoryStorage) Close 

func (m *MemoryStorage) Close() error

Close is a no-op for memory storage.

func (*MemoryStorage) Load 

func (m *MemoryStorage) Load() (*DatabaseSnapshot, error)

Load returns the stored snapshot or nil if none exists.

func (*MemoryStorage) Save 

func (m *MemoryStorage) Save(snapshot *DatabaseSnapshot) error

Save stores the snapshot in memory.

type Metrics added in v0.6.0 

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics provides observable counters for database operations. All operations are atomic and safe for concurrent access.

func (*Metrics) Snapshot added in v0.6.0 

func (m *Metrics) Snapshot() MetricsSnapshot

Snapshot returns a point-in-time snapshot of the metrics.

type MetricsSnapshot added in v0.6.0 

type MetricsSnapshot struct {
	SearchCount   int64         `json:"search_count"`
	InsertCount   int64         `json:"insert_count"`
	DeleteCount   int64         `json:"delete_count"`
	AvgSearchTime time.Duration `json:"avg_search_time_ns"`
}

MetricsSnapshot is a point-in-time snapshot of database metrics.

type NopLogger added in v0.6.0 

type NopLogger struct{}

NopLogger is a no-op logger that discards all messages. This is the default logger used when none is configured, ensuring zero overhead.

func (NopLogger) Debug added in v0.6.0 

func (NopLogger) Debug(string, ...any)

func (NopLogger) Error added in v0.6.0 

func (NopLogger) Error(string, ...any)

func (NopLogger) Info added in v0.6.0 

func (NopLogger) Info(string, ...any)

type NotFoundError 

type NotFoundError struct {
	Type string // "record", "collection", etc.
	ID   string // Identifier that was not found
}

NotFoundError provides details about what was not found.

func (*NotFoundError) Error 

func (e *NotFoundError) Error() string

func (*NotFoundError) Unwrap 

func (e *NotFoundError) Unwrap() error

type Option 

type Option interface {
	// contains filtered or unexported methods
}

Option configures the database.

func WithLogger added in v0.6.0 

func WithLogger(l Logger) Option

WithLogger sets a logger for the database. Pass nil or NopLogger{} to disable logging (default).

func WithReadOnly 

func WithReadOnly(enabled bool) Option

WithReadOnly opens the database in read-only mode. Write operations will return an error.

func WithSyncOnWrite 

func WithSyncOnWrite(enabled bool) Option

WithSyncOnWrite enables automatic sync after each write operation. This is slower but ensures durability.

type Record 

type Record struct {
	// ID is the unique identifier for this record.
	ID uint64

	// Vector is the embedding vector.
	Vector []float32

	// Payload contains arbitrary metadata associated with the vector.
	Payload map[string]any

	// Content is the optional original text content associated with this record.
	// Used for document-oriented storage and automatically indexed by BM25 when text indexing is enabled.
	Content string

	// CreatedAt is when the record was inserted.
	CreatedAt time.Time

	// UpdatedAt is when the record was last updated.
	UpdatedAt time.Time
}

Record represents a stored vector with its metadata.

func (*Record) Clone 

func (r *Record) Clone() *Record

Clone creates a deep copy of the record.

type RecordSnapshot 

type RecordSnapshot struct {
	// ID is the record's unique identifier.
	ID uint64

	// Vector is the embedding vector.
	Vector []float32

	// Payload contains arbitrary metadata.
	Payload map[string]any

	// Content is the optional text content.
	Content string

	// CreatedAt is when the record was inserted.
	CreatedAt time.Time

	// UpdatedAt is when the record was last updated.
	UpdatedAt time.Time
}

RecordSnapshot is the serializable state of a record.

type Result 

type Result struct {
	// Record is the matched record.
	Record *Record

	// Score is the similarity/distance score.
	// For cosine/dot: higher is more similar.
	// For euclidean: lower is more similar.
	Score float32
}

Result represents a search result with its similarity score.

type SearchExplanation added in v0.2.0 

type SearchExplanation struct {
	// Results contains the search results.
	Results []Result

	// IndexType is the type of index used (none, hnsw).
	IndexType string

	// NodesVisited is the number of nodes visited during search.
	// Only populated for HNSW searches.
	NodesVisited int

	// LayersVisited is the number of HNSW layers visited.
	// Only populated for HNSW searches.
	LayersVisited int

	// Duration is how long the search took.
	Duration time.Duration

	// BruteForce indicates whether brute-force search was used.
	BruteForce bool
}

SearchExplanation provides details about how a search was performed.

type SearchFunc added in v0.6.0 

type SearchFunc func(result Result) bool

SearchFunc is a callback for streaming search results. Return false to stop receiving results.

type SearchOption 

type SearchOption interface {
	// contains filtered or unexported methods
}

SearchOption configures search behavior.

func Threshold 

func Threshold(t float32) SearchOption

Threshold sets the minimum similarity score for results. For cosine/dot: results with score >= threshold are returned. For euclidean: results with score <= threshold are returned.

func TopK 

func TopK(k int) SearchOption

TopK sets the maximum number of results to return. Default is 10.

func WithContent added in v0.6.0 

func WithContent(include bool) SearchOption

WithContent controls whether the Content field is included in search results. By default, Content is included. Set to false to exclude it for smaller results.

func WithEfSearch added in v0.2.0 

func WithEfSearch(ef int) SearchOption

WithEfSearch sets the efSearch parameter for HNSW search. Higher values improve recall at the cost of speed. Has no effect on collections without HNSW index.

func WithFilter 

func WithFilter(f Filter) SearchOption

WithFilter adds a filter to the search. Multiple filters are combined with AND logic.

func WithFilters 

func WithFilters(filters ...Filter) SearchOption

WithFilters adds multiple filters to the search. All filters are combined with AND logic.

func WithLimit added in v0.6.0 

func WithLimit(n int) SearchOption

WithLimit sets the maximum number of results to return. This is an alias for TopK for use in pagination contexts.

func WithOffset added in v0.6.0 

func WithOffset(n int) SearchOption

WithOffset sets the number of results to skip before returning. Use with TopK for pagination: WithOffset(20), TopK(10) returns results 21-30.

func WithTextWeight added in v0.6.0 

func WithTextWeight(w float64) SearchOption

WithTextWeight sets the weight for the text search component in hybrid search. Default is 1.0.

func WithVectorWeight added in v0.6.0 

func WithVectorWeight(w float64) SearchOption

WithVectorWeight sets the weight for the vector search component in hybrid search. Default is 1.0.

type Storage 

type Storage interface {
	// Load reads the database from storage.
	// Returns nil, nil if the database doesn't exist yet.
	Load() (*DatabaseSnapshot, error)

	// Save writes the database to storage.
	Save(snapshot *DatabaseSnapshot) error

	// Close releases any resources held by the storage.
	Close() error
}

Storage is the interface for database persistence.

type StorageError 

type StorageError struct {
	Op  string // Operation that failed
	Err error  // Underlying error
}

StorageError wraps storage-related errors with context.

func (*StorageError) Error 

func (e *StorageError) Error() string

func (*StorageError) Unwrap 

func (e *StorageError) Unwrap() error

Source Files

View all Source files

Directories

Path Synopsis
cmd
veclite command
Command veclite provides a CLI for interacting with VecLite databases.
 Command veclite provides a CLI for interacting with VecLite databases.
examples
basic command
Example basic demonstrates the core VecLite operations: open a database, insert vectors, search, and close.
 Example basic demonstrates the core VecLite operations: open a database, insert vectors, search, and close.
batch command
Example batch demonstrates batch operations, upsert, and iteration.
 Example batch demonstrates batch operations, upsert, and iteration.
filtering command
Example filtering demonstrates VecLite's rich filter expressions.
 Example filtering demonstrates VecLite's rich filter expressions.
hnsw command
Example hnsw demonstrates HNSW index configuration and performance.
 Example hnsw demonstrates HNSW index configuration and performance.
http-client command
Example http-client demonstrates using VecLite's HTTP API.
 Example http-client demonstrates using VecLite's HTTP API.
internal
floats
Package floats provides optimized floating-point vector operations.
 Package floats provides optimized floating-point vector operations.
hnsw
Package hnsw implements the Hierarchical Navigable Small World graph algorithm for approximate nearest neighbor search.
 Package hnsw implements the Hierarchical Navigable Small World graph algorithm for approximate nearest neighbor search.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.