micrographrag

package module
v0.0.0-...-fdb8fd5 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 16, 2026 License: MIT Imports: 13 Imported by: 0

README

micrographrag-go

A low-memory embedded memory/database layer for Go agents.

It combines SQLite + FTS5 + graph tables + sqlite-vec + local static embeddings in a single SQLite database. The target is an agent that already runs around 12 MB RSS and should remain below roughly 30 MB total under normal embedded workloads.

Architecture

                     Go Agent
                        |
                  +-----+------+
                  | MicroStore |
                  +-----+------+
                        |
           +------------+-------------+
           |            |             |
          KV           FTS5          Graph
                                      nodes/edges
           |            |             |
           |       Local Embedder      |
           |       Model2Vec 64D       |
           |            |             |
           |       INT8 quantize       |
           |            |             |
           |        sqlite-vec         |
           |            |             |
           +--------- RRF -------------+
                        |
                 graph expansion
                    1-2 hops
                        |
                  Agent Context

Design goals

  • one agent.db, no database server;
  • SQLite is also the KV/document/state store;
  • FTS5 provides lexical/BM25 retrieval;
  • nodes + edges + recursive CTE provide bounded graph traversal;
  • chunk_nodes connects graph entities to textual memories;
  • sqlite-vec stores compact int8[64] vectors;
  • Model2Vec/Potion generates 64D embeddings locally;
  • embeddings are generated lazily after the memory transaction commits;
  • RRF merges lexical and semantic ranks without mixing incompatible score scales;
  • every expensive operation has small hard limits;
  • FTS/database keep working if embeddings are unavailable.

Requirements

  • Go 1.25+
  • C compiler (CGO)
  • build with the sqlite_fts5 tag

The sqlite-vec CGO binding compiles the extension into the application; there is no runtime extension file or vector server.

Quick start

go mod download
make test
make build

Minimal database-only usage:

cfg := micrographrag.DefaultConfig("agent.db")
cfg.EnableVector = false

store, err := micrographrag.Open(ctx, cfg, nil)
if err != nil { panic(err) }
defer store.Close()

_ = store.PutKV(ctx, "agent", "state", []byte("ready"))

With local embeddings:

embedder, err := micrographrag.NewPotionEmbedder(ctx)
if err != nil { panic(err) }

store, err := micrographrag.Open(
    ctx,
    micrographrag.DefaultConfig("agent.db"),
    embedder,
)

Fully offline model deployment

go-potion normally downloads potion-base-2M on the first run. For an embedded/offline deployment, preload these files:

$GO_POTION_HOME/BASE2M/model.safetensors
$GO_POTION_HOME/BASE2M/tokenizer.json

When both files exist, model loading performs no download. A custom compatible 64D Model2Vec PT/EN model can be placed in the same location. See scripts/build_embedding_model/.

Memory insertion

m, err := store.AddMemory(ctx, micrographrag.MemoryInput{
    Source:  "tool",
    Title:   "OAuth failure",
    Content: "O provider recusou a autenticação OAuth.",
})

The transaction writes the document/chunks and FTS index first. Vector embedding happens afterward through a single lazy worker, so a slow embedder does not keep the write transaction open.

Graph

provider, _ := store.UpsertNode(ctx, micrographrag.Node{
    Kind: 1, Canonical: "provider", Display: "Provider",
})

oauth, _ := store.UpsertNode(ctx, micrographrag.Node{
    Kind: 2, Canonical: "oauth", Display: "OAuth",
})

_ = store.UpsertEdge(ctx, micrographrag.Edge{
    Src: provider, Dst: oauth, Relation: 1, Weight: 1, Confidence: 1,
})

_ = store.LinkChunkNode(ctx, m.ChunkIDs[0], oauth, 1)

Traversal is bounded to depth <= 3 and <= 128 nodes; defaults are depth 2 and 64 nodes.

results, err := store.Search(
    ctx,
    "problema de autenticação do provider",
    micrographrag.SearchOptions{},
)

Pipeline:

FTS5 top 16 ---------+
                     +--> RRF(k=60) --> top 8 --> graph expansion --> final
Vector top 16 -------+

Vector failure is fail-soft: the same query can still return FTS + graph results.

Important resource defaults

Resource Default
SQLite page cache 1536 KiB
Open SQLite connections 1
FTS candidates 16 (hard max 64)
Vector candidates 16 (hard max 64)
Final results 8 (hard max 12)
Graph depth 2 (hard max 3)
Graph visited nodes 64 (hard max 128)
Vector dimension 64
Vector representation INT8
Embedding workers 1
temp_store FILE
mmap_size for SQLite 0

The Potion model itself uses mmap where supported. RSS must be measured on the actual target device; mmap-backed pages are not the same thing as Go heap.

FTS consistency

FTS5 uses an external-content table backed by chunks and triggers keep it in sync. RebuildFTS and CheckIntegrity are included for recovery/diagnostics.

Model changes

Do not mix vectors from different embedding models. v1 fixes the vector space to 64 dimensions. When replacing the local model, clear/rebuild chunk_vec and mark existing chunks pending before using vector retrieval again.

Build targets

CGO_ENABLED=1 go test -tags sqlite_fts5 ./...
CGO_ENABLED=1 go build -tags sqlite_fts5 ./...

Cross-compilation requires a C toolchain for the target architecture.

Status

This is an initial embedded-focused implementation. sqlite-vec is pre-v1, so its usage is intentionally isolated behind this package. Benchmark RSS and latency on your target hardware before setting production limits.

Documentation

Index

Constants

View Source 
const (
	EmbeddingPending = iota
	EmbeddingIndexed
	EmbeddingRetry
	EmbeddingUnavailable
	EmbeddingDisabled
)
View Source 
const VectorDimensions = 64

Variables

View Source 
var (
	ErrNotFound       = errors.New("micrographrag: not found")
	ErrZeroVector     = errors.New("micrographrag: zero vector")
	ErrVectorDisabled = errors.New("micrographrag: vector search disabled")
	ErrFTSDisabled    = errors.New("micrographrag: fts disabled")
	ErrGraphDisabled  = errors.New("micrographrag: graph disabled")
)

Functions

func ChunkText 

func ChunkText(text string, maxRunes, overlap int) []string

func QuantizeI8 

func QuantizeI8(v []float32) ([]byte, error)

Types

type Config 

type Config struct {
	DBPath string

	SQLiteCacheKB int
	BusyTimeout   time.Duration
	EnableWAL     bool

	EnableFTS    bool
	EnableVector bool
	EnableGraph  bool

	FTSLimit    int
	VectorLimit int
	HybridLimit int

	GraphDepth    int
	GraphMaxNodes int

	MaxChunkRunes int
	ChunkOverlap  int

	EmbeddingPollInterval time.Duration
	EmbeddingMaxAttempts  int
	EnableEmbeddingWorker bool
}

func DefaultConfig 

func DefaultConfig(path string) Config

type DBStats 

type DBStats struct {
	Documents int64
	Chunks    int64
	Nodes     int64
	Edges     int64
	Vectors   int64
	Pending   int64
}

type Edge 

type Edge struct {
	Src        int64
	Dst        int64
	Relation   int
	Weight     float64
	Confidence float64
}

type Embedder 

type Embedder interface {
	Dimensions() int
	Encode(ctx context.Context, text string) ([]float32, error)
	Close() error
}

type IntegrityReport 

type IntegrityReport struct {
	SQLiteOK          bool
	FTSOK             bool
	OrphanVectors     int64
	OrphanChunkNodes  int64
	PendingEmbeddings int64
}

type MemoryInput 

type MemoryInput struct {
	Kind       int
	Source     string
	Title      string
	Content    string
	Metadata   []byte
	Importance float64
}

type MemoryResult 

type MemoryResult struct {
	DocumentID int64
	ChunkIDs   []int64
}

type Node 

type Node struct {
	ID        int64
	Kind      int
	Canonical string
	Display   string
	Metadata  []byte
}

type NoopEmbedder 

type NoopEmbedder struct{}

func (NoopEmbedder) Close 

func (NoopEmbedder) Close() error

func (NoopEmbedder) Dimensions 

func (NoopEmbedder) Dimensions() int

func (NoopEmbedder) Encode 

func (NoopEmbedder) Encode(context.Context, string) ([]float32, error)

type PotionEmbedder 

type PotionEmbedder struct {
	// contains filtered or unexported fields
}

func NewPotionEmbedder 

func NewPotionEmbedder(ctx context.Context) (*PotionEmbedder, error)

NewPotionEmbedder loads potion-base-2M (64 dimensions). If GO_POTION_HOME already contains BASE2M/model.safetensors and BASE2M/tokenizer.json, no network access is performed. This is the recommended deployment mode.

func (*PotionEmbedder) Close 

func (p *PotionEmbedder) Close() error

func (*PotionEmbedder) Dimensions 

func (p *PotionEmbedder) Dimensions() int

func (*PotionEmbedder) Encode 

func (p *PotionEmbedder) Encode(ctx context.Context, text string) ([]float32, error)

type SearchOptions 

type SearchOptions struct {
	Limit int

	EnableFTS    bool
	EnableVector bool
	EnableGraph  bool

	FTSLimit    int
	VectorLimit int
	GraphDepth  int

	FTSWeight    float64
	VectorWeight float64
}

type SearchResult 

type SearchResult struct {
	ChunkID    int64
	DocumentID int64
	Content    string
	Score      float64

	FTSRank        int
	VectorRank     int
	VectorDistance float64
	GraphDepth     int

	FromFTS    bool
	FromVector bool
	FromGraph  bool
}

type Store 

type Store struct {
	// contains filtered or unexported fields
}

func Open 

func Open(ctx context.Context, cfg Config, embedder Embedder) (*Store, error)

func (*Store) AddMemory 

func (s *Store) AddMemory(ctx context.Context, in MemoryInput) (MemoryResult, error)

func (*Store) CheckIntegrity 

func (s *Store) CheckIntegrity(ctx context.Context) (IntegrityReport, error)

func (*Store) Close 

func (s *Store) Close() error

func (*Store) DB 

func (s *Store) DB() *sql.DB

func (*Store) DefaultSearchOptions 

func (s *Store) DefaultSearchOptions() SearchOptions

func (*Store) DeleteDocument 

func (s *Store) DeleteDocument(ctx context.Context, documentID int64) error

func (*Store) DeleteKV 

func (s *Store) DeleteKV(ctx context.Context, namespace, key string) error

func (*Store) GetKV 

func (s *Store) GetKV(ctx context.Context, namespace, key string) ([]byte, error)

func (*Store) LinkChunkNode 

func (s *Store) LinkChunkNode(ctx context.Context, chunkID, nodeID int64, weight float64) error

func (*Store) Neighbors 

func (s *Store) Neighbors(ctx context.Context, nodeID int64, relation *int, limit int) ([]Node, error)

func (*Store) Optimize 

func (s *Store) Optimize(ctx context.Context) error

func (*Store) PutKV 

func (s *Store) PutKV(ctx context.Context, namespace, key string, value []byte) error

func (*Store) RebuildFTS 

func (s *Store) RebuildFTS(ctx context.Context) error

func (*Store) Search 

func (s *Store) Search(ctx context.Context, query string, opts SearchOptions) ([]SearchResult, error)

func (*Store) Stats 

func (s *Store) Stats(ctx context.Context) (DBStats, error)

func (*Store) UpsertEdge 

func (s *Store) UpsertEdge(ctx context.Context, e Edge) error

func (*Store) UpsertNode 

func (s *Store) UpsertNode(ctx context.Context, n Node) (int64, error)

func (*Store) Vacuum 

func (s *Store) Vacuum(ctx context.Context) error

func (*Store) Walk 

func (s *Store) Walk(ctx context.Context, startID int64, depth, limit int) ([]Node, error)

Source Files

View all Source files

Directories

Path Synopsis
cmd
demo command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.