sqvect

README

sqvect

A lightweight, embeddable vector database LIBRARY for Go AI projects.

sqvect is a 100% pure Go library designed to be the storage kernel for your RAG applications. It provides vector storage, keyword search (FTS5), graph relationships, and chat memory management in a single SQLite file.

✨ Features

• 🪶 Lightweight – Single SQLite file, zero external dependencies.
• 🚀 RAG-Ready – Built-in tables for Documents, Chat Sessions, and Messages.
• 🔍 Hybrid Search – Combine Vector Search (HNSW) + Keyword Search (FTS5) with RRF fusion.
• 🛡️ Secure – Row-Level Security (RLS) via ACL fields and query filtering.
• 🧠 Memory Efficient – SQ8 Quantization reduces RAM usage by 75%.
• ⚡ High Performance – Optimized WAL mode, SIMD-ready distance calcs.
• 🎯 Zero Config – Works out of the box.

🚀 Quick Start

go get github.com/liliang-cn/sqvect

package main

import (
    "context"
    "fmt"
    "github.com/liliang-cn/sqvect/pkg/sqvect"
)

func main() {
    // 1. Open DB (auto-creates tables for vectors, docs, chat)
    db, _ := sqvect.Open(sqvect.DefaultConfig("rag.db"))
    defer db.Close()
    ctx := context.Background()

    // 2. Add a Document & Vector
    // sqvect manages the relationship between docs and chunks
    db.Vector().CreateDocument(ctx, &core.Document{ID: "doc1", Title: "Go Guide"})
    
    db.Quick().Add(ctx, []float32{0.1, 0.2, 0.9}, "Go is awesome")

    // 3. Search
    results, _ := db.Quick().Search(ctx, []float32{0.1, 0.2, 0.8}, 1)
    fmt.Printf("Found: %s\n", results[0].Content)
}

🏗 Enterprise RAG Capabilities

sqvect goes beyond simple vector storage. It provides the schema and APIs needed for complex RAG apps.

1. Hybrid Search (Vector + Keyword)

Combine semantic understanding with precise keyword matching using Reciprocal Rank Fusion (RRF).

// Search for "apple" (keyword) AND vector similarity
results, _ := db.Vector().HybridSearch(ctx, queryVec, "apple", core.HybridSearchOptions{
    TopK: 5,
    RRFK: 60, // Fusion parameter
})

2. Chat Memory Management

Store conversation history directly alongside your data.

// 1. Create a session
db.Vector().CreateSession(ctx, &core.Session{ID: "sess_1", UserID: "user_123"})

// 2. Add messages (User & Assistant)
db.Vector().AddMessage(ctx, &core.Message{
    SessionID: "sess_1",
    Role:      "user",
    Content:   "What is sqvect?",
})

// 3. Retrieve history for context window
history, _ := db.Vector().GetSessionHistory(ctx, "sess_1", 10)

3. Row-Level Security (ACL)

Enforce permissions at the database level.

// Insert restricted document
db.Vector().Upsert(ctx, &core.Embedding{
    ID: "secret_doc", 
    Vector: vec, 
    ACL: []string{"group:admin", "user:alice"}, // Only admins and Alice
})

// Search with user context (auto-filters results)
results, _ := db.Vector().SearchWithACL(ctx, queryVec, []string{"user:bob"}, opts)
// Returns nothing for Bob!

4. Document Management

Track source files, versions, and metadata. Deleting a document automatically deletes all its vector chunks (Cascading Delete).

db.Vector().CreateDocument(ctx, &core.Document{
    ID: "manual_v1", 
    Title: "User Manual",
    Version: 1,
})
// ... add embeddings linked to "manual_v1" ...

// Delete document and ALL its embeddings in one call
db.Vector().DeleteDocument(ctx, "manual_v1")

📚 Database Schema

sqvect manages these tables for you:

Table	Description
embeddings	Vectors, content, JSON metadata, ACLs.
documents	Parent records for embeddings (Title, URL, Version).
sessions	Chat sessions/threads.
messages	Chat logs (Role, Content, Timestamp).
collections	Logical namespaces (Multi-tenancy).
chunks_fts	FTS5 virtual table for keyword search.

📊 Performance (128-dim)

Index Type	Insert Speed	Search QPS	Memory (1M vecs)
HNSW	~580 ops/s	~720 QPS	~1.2 GB (SQ8)
IVF	~14,500 ops/s	~1,230 QPS	~1.0 GB (SQ8)

Tested on Apple M2 Pro.

⚖️ License

MIT License. See LICENSE file.

Documentation

Overview

Package sqvect provides a lightweight, embeddable vector store using SQLite.

sqvect is a 100% pure Go library designed for AI applications that need fast, reliable vector storage without external dependencies. Built on SQLite using modernc.org/sqlite (pure Go implementation - NO CGO REQUIRED!), it's perfect for RAG (Retrieval-Augmented Generation) systems, semantic search, knowledge graphs, and any Go AI project that needs embedding storage.

Features

• 100% Pure Go - No CGO dependencies, easy cross-compilation
• SQLite-based storage with single .db file
• Multiple similarity functions (cosine, dot product, Euclidean distance)
• Collections support for multi-tenant namespacing
• Knowledge graphs with advanced graph operations
• Batch operations for efficient data loading
• Thread-safe operations with concurrent read/write support
• Rich metadata support with JSON storage
• Automatic dimension adaptation for any embedding model
• HNSW indexing for high-performance search

Quick Start

Create a new vector store and perform basic operations:

package main

import (
    "context"
    "log"
    "github.com/liliang-cn/sqvect/pkg/sqvect"
)

func main() {
    // Initialize database
    config := sqvect.Config{
        Path:       "embeddings.db",
        Dimensions: 768, // or 0 for auto-detect
    }

    db, err := sqvect.Open(config)
    if err != nil {
        log.Fatal(err)
    }
    defer db.Close()

    ctx := context.Background()
    quick := db.Quick()

    // Add an embedding
    vector := []float32{0.1, 0.2, 0.3, ...} // 768 dimensions
    id, err := quick.Add(ctx, vector, "Sample text content")
    if err != nil {
        log.Fatal(err)
    }

    // Search for similar vectors
    query := []float32{0.1, 0.25, 0.28, ...} // 768 dimensions
    results, err := quick.Search(ctx, query, 5)
    if err != nil {
        log.Fatal(err)
    }

    for _, result := range results {
        log.Printf("ID: %s, Score: %.3f, Content: %s\n",
            result.ID, result.Score, result.Content)
    }
}

Advanced Usage

Using collections and vector store directly:

import (
    "github.com/liliang-cn/sqvect/pkg/core"
    "github.com/liliang-cn/sqvect/pkg/sqvect"
)

// Create collections for different data types
vectorStore := db.Vector()

_, err := vectorStore.CreateCollection(ctx, "products", 256)
_, err = vectorStore.CreateCollection(ctx, "users", 128)

// Add to specific collection
emb := &core.Embedding{
    ID:         "product_123",
    Collection: "products",
    Vector:     productVector,
    Content:    "Product description",
    Metadata: map[string]string{
        "category": "electronics",
        "price":    "99.99",
    },
}
err = vectorStore.Upsert(ctx, emb)

// Search within collection
results, err := vectorStore.Search(ctx, queryVector, core.SearchOptions{
    Collection: "products",
    TopK:       10,
    Threshold:  0.7,
})

Graph Operations

Using the graph store for knowledge graphs:

import "github.com/liliang-cn/sqvect/pkg/graph"

graphStore := db.Graph()
err := graphStore.InitGraphSchema(ctx)

// Create nodes
node := &graph.GraphNode{
    ID:       "doc_1",
    Vector:   docVector,
    Content:  "Document content",
    NodeType: "document",
}
err = graphStore.UpsertNode(ctx, node)

// Create relationships
edge := &graph.GraphEdge{
    ID:         "edge_1",
    FromNodeID: "doc_1",
    ToNodeID:   "doc_2",
    EdgeType:   "references",
    Weight:     0.8,
}
err = graphStore.UpsertEdge(ctx, edge)

// Hybrid search (vector + graph)
results, err := graphStore.HybridSearch(ctx, &graph.HybridQuery{
    Vector:      queryVector,
    StartNodeID: "doc_1",
    TopK:        5,
    Weights: graph.HybridWeights{
        VectorWeight: 0.5,
        GraphWeight:  0.3,
        EdgeWeight:   0.2,
    },
})

Similarity Functions

sqvect provides three built-in similarity functions:

• CosineSimilarity: Best for text embeddings (default)
• DotProduct: Fast computation for normalized vectors
• EuclideanDist: Good for spatial data and image embeddings

Configure via:

config := sqvect.Config{
    Path:         "data.db",
    Dimensions:   384,
    SimilarityFn: core.CosineSimilarity, // or core.DotProduct, core.EuclideanDist
}

Performance

sqvect is optimized for common vector operations:

• Cosine similarity: ~1.2M operations/second
• Vector encoding/decoding: ~38K operations/second
• Single upsert: ~20K operations/second
• Batch search (1K vectors): ~60 operations/second
• Pure Go implementation enables easy deployment and cross-compilation

Thread Safety

All operations are thread-safe. Multiple goroutines can safely read and write to the same store instance concurrently.

Error Handling

sqvect uses wrapped errors with operation context. Check for specific errors:

err := vectorStore.Delete(ctx, "non-existent")
if errors.Is(err, core.ErrNotFound) {
    // Handle not found case
}

Common errors include:

• ErrInvalidDimension: Vector dimension mismatch
• ErrInvalidVector: Invalid vector data (nil, empty, NaN, Inf)
• ErrNotFound: Embedding not found
• ErrStoreClosed: Operation on closed store

Why Pure Go?

sqvect uses modernc.org/sqlite, a pure Go SQLite implementation, which means:

• No CGO required - simplifies builds and deployments
• Cross-compilation to any platform Go supports
• Single binary distribution
• Better compatibility with serverless and container environments
• Easier debugging and profiling

Examples

See the examples/ directory for comprehensive examples:

• semantic_search: Full-text semantic search
• document_clustering: K-means clustering
• hybrid_search: Combined vector + graph search
• multi_collection: Multi-tenant data management
• image_search: Multi-modal CLIP-like search
• knowledge_graph: Graph-based knowledge management
• rag_system: Retrieval-augmented generation
• benchmark: Performance testing

Index

• Constants

Constants

const Version = "1.4.0"

Version represents the current version of the sqvect library.