reliquary

README

reliquary

reliquary is a library-only, boundary-first toolkit of provider-neutral primitives and thin opt-in adapters for building local-first retrieval and content pipelines in Go. Callers own all policy — models, databases, prompts, scheduling, business logic. reliquary owns the neutral contracts and the deterministic math.

cd examples
go run ./cmd/examples list
go run ./cmd/examples next
go run ./cmd/examples run quickstart-rag

app := reliquary.Quickstart()

if _, err := app.Ingest(ctx, docs...); err != nil {
	return err
}
hits, err := app.Search(ctx, "how does Go reclaim memory", reliquary.TopK(5), reliquary.WithMMR(0.5))
if err != nil {
	return err
}
_ = hits

Quickstart uses deterministic demo/test embeddings and an in-memory store so the example runs without provider setup. It is not production retrieval quality. For production systems, bring your own embedder and store:

app, err := reliquary.New(
	reliquary.WithEmbedder(myEmbedder),
	reliquary.WithStore(myStore),
)

Manual composition remains available when callers need each boundary explicitly:

scorer := retrieval.NewScorer(retrieval.DefaultWeights())

results, err := retrieval.ResultsFromDocuments(docs, chunking.SmartBoundary, 1200, 100)
if err != nil {
	return err
}
if err := retrieval.AttachEmbeddings(results, embedded.Vectors); err != nil {
	return err
}
ranked := scorer.RerankEmbedding(queryEmbedding.Vectors[0], "how does Go reclaim memory", results)
top := retrieval.Diversify(ranked, 5, 0.5)
_ = top

A package belongs in reliquary only if it is library-only, works in a caller-owned semantic space (no models/DBs/clients/servers/prompts in its core), is a neutral primitive or thin adapter, and is domain-free. See ARCHITECTURE.md for the full inclusion rule and boundary contract.

go get github.com/dotcommander/reliquary

The production embeddings module remains separate. Outside of the demo hashing embedder, reliquary expects callers to produce embeddings elsewhere and pass vectors into these packages. If you use github.com/dotcommander/embeddings, import the separate adapter module so model/runtime dependencies stay out of the primitive module:

go get github.com/dotcommander/reliquary/providers/dotcommander/embeddings

OpenAI/OpenRouter-compatible provider client wiring is isolated in a nested adapter module; generic config-map parsing lives in a small runtime module:

go get github.com/dotcommander/reliquary/providers/openai
go get github.com/dotcommander/reliquary/runtime/config

Domain and product code that does not meet the inclusion rule lives in sibling modules, not in reliquary:

go get github.com/dotcommander/mlmetrics    # ML-model evaluation metrics
go get github.com/dotcommander/genealogy    # genealogy/GEDCOM graph extraction
go get github.com/dotcommander/memory-core  # agent-memory library

Use pipeline/retrieval metrics for information-retrieval quality: Recall@K, Precision@K, MRR, NDCG@K, segment summaries, layer reports, and ranking-weight tuning. Use mlmetrics or caller-owned evaluation code for numeric prediction losses such as RMSE, RMSLE, MAE, MAPE, Huber loss, or quantile/pinball loss. Those metrics measure regression estimates in the caller's semantic space, not retrieval rank quality.

The root module is primitive-only. It has no production CLI or server entrypoint. Start with examples/quickstart-rag, then use examples/ for package-specific runnable examples. Runtime dependencies, provider clients, databases, migrations, HTTP servers, and CLI startup code belong in nested modules or sibling repositories.

Packages

Package	Use it for
support/hash, support/validate, support/diff	Small shared helpers for stable hashes, primitive validation, and deterministic metadata diffs.
document	Provider-neutral document, section, span, format, metadata, normalized text, and byte/rune offset primitives.
provenance	Source, artifact, claim, lineage, hash, generated-artifact metadata, and source-to-output links.
ingest	Generic reader, decoder, mapper, sink, batch, report, and cursor contracts for caller-owned pipelines.
workflow	Durable run, stage, attempt, checkpoint, cursor, state, and idempotency key value types.
events	Generic event, listener, dispatcher, outbox, and append-only event log interfaces. App repos own event names.
embeddings	Provider-neutral embedding model references, vectors, batch requests/results, dimension validation, and cache keys.
llm	Provider-neutral language-model messages, requests, responses, stream events, tool calls, usage, and provider interface.
webfetch	URL normalization, fetch request/result metadata, cache validators, rate-limit/robots contracts, and snapshots.
media	Audio, video, image, transcript, segment, time range, and derivative artifact metadata.
chunking	Split text for search, retrieval, summarization, and LLM context assembly. Chunks carry byte spans, heading paths, metadata, token counts, and content hashes where the strategy can preserve them.
lexical	Normalize lexical queries, count terms, compute local BM25 scores, and adapt ranked lexical results without owning database schemas, SQL tokenizers, stemming, trigram behavior, or business boosts.
vectors	Run primitive vector math: cosine, dot products, normalization, quantization, Hamming distance, in-memory exact search, binary pre-filtering, index manifests, RRF, k-means, and semantic-boundary helpers.
retrieval	Score and rerank local search results with embedding, keyword, filename, metadata, MMR diversification, score-reference percentiles, tuning grids, and evaluation metrics.
clustering	Assign documents to caller-supplied embedding categories with centroid updates, confidence bands, drift-recorder hooks, merging, and TF-IDF helpers.
dedup	Detect exact and near-duplicate text-like values with generic detector APIs and hash strategies.
graph	Build and query directed graphs, extract entities and relationships from text chunks, run deterministic community detection, and model temporal graph facts.
runtime/config	Typed map[string]any parsing, defaults, required-field checks, and primitive config accessors.
storage	Driver-free storage identifiers and minimal store contracts.
storage/clustering	Optional filesystem adapter for category seeding and centroid drift persistence.
storage/postgres	PostgreSQL identifier validation, quoting, DSN, and pool config helpers for SQL-building adapters and migrations.
textutil	Extract keywords, normalize titles, locate fragments, score fuzzy aliases, detect simple themes, and work with stop words.
observability	Small logging, metrics, and tracing interfaces with caller-owned metric/span names.
observability/logging	Minimal slog-compatible logger interface and discard logger helper.
testkit	Fake clock, fake LLM, fake embedder, fake event log, and fixture documents for downstream tests.

Package Map

• Primitive value packages: document, provenance, workflow, media, storage, and support/*.
• Provider-neutral contracts: ingest, events, embeddings, llm, webfetch, and observability.
• Retrieval and ML primitives: chunking, lexical, vectors, retrieval, clustering, dedup, graph, and textutil.
• Test support: testkit.
• Optional root adapters: storage/clustering and observability/logging.
• Nested adapter/runtime modules: providers/dotcommander/embeddings, providers/openai, runtime/cli, runtime/config, runtime/web, storage/migrate, storage/postgres, storage/redis, and storage/sqlite.
• Nested tool module: tools/gokart.

Common Flows

Chunk text

chunker, err := chunking.NewChunker(chunking.MarkdownAware)
if err != nil {
	panic(err)
}

chunks := chunker.Chunk(markdownText, 1200, 100)

Use SentenceBoundary, CodeBlockAware, ParagraphAware, HeadingAware, MarkdownAware, TokenBased, or semantic chunking depending on the input. MarkdownAware and HeadingAware use Goldmark for structural Markdown parsing.

Score and diversify retrieval results

scorer := retrieval.NewScorer(retrieval.DefaultWeights())
ranked := scorer.RerankEmbedding(queryEmbedding, "machine learning", candidates)
diverse := retrieval.Diversify(ranked, 10, 0.5)
_ = ranked
_ = diverse

retrieval does not embed text. Pass in embeddings from your own model or embedding service.

Adapt lexical search

lexical keeps FTS providers at the app boundary. SQLite/PostgreSQL callers own schema, tokenizer, stemming, trigram, and boost choices, then adapt ranked IDs:

fts5 := lexical.RankByOrder([]lexical.Result{
	{ID: "doc-a", Score: -0.7},
	{ID: "doc-b", Score: -0.2},
}, "sqlite_fts5", lexical.ScoreSpaceRankOnly)

pg := lexical.RankByScore([]lexical.Candidate{
	{ID: "doc-a", Score: 0.5, Source: "postgres_fts", ScoreSpace: lexical.ScoreSpaceProvider},
	{ID: "doc-b", Score: 0.2, Source: "postgres_fts", ScoreSpace: lexical.ScoreSpaceProvider},
})

fused := lexical.FuseRRFByID([]lexical.FusionInput{
	{Source: "sqlite_fts5", Candidates: fts5},
	{Source: "postgres_fts", Candidates: pg},
}, lexical.FusionOptions{K: 60, Limit: 10})

_ = retrieval.EvaluateLayers(queryFixture, retrieval.LayeredResults{
	Candidates: lexical.ToRetrievalResults(fts5),
	Final:      lexical.ToRetrievalResults(fused),
}, 10)

For stable score semantics across runs, fit a score-reference cohort and apply it explicitly after validating the scoring identity:

reference, err := retrieval.FitScoreReference(referenceScores, retrieval.ScoreReferenceIdentity{
	ScoreVersion: "retrieval-v1",
	ModelID:      "local-embedder-v1",
	ConfigHash:   "weights-v1",
})
if err != nil {
	panic(err)
}
if err := reference.Validate(retrieval.ScoreReferenceIdentity{ScoreVersion: "retrieval-v1"}); err != nil {
	panic(err)
}
ranked = scorer.RerankWithReference(queryEmbedding, "machine learning", candidates, reference)

Use evaluation helpers when tuning retrieval quality. The example variables are caller-owned query fixtures and stage outputs from your retrieval pipeline:

segments := retrieval.EvaluateSegments(query, rankedResults, 10, func(docID string) string {
	return sourceTypeByDoc[docID]
})

layers := retrieval.EvaluateLayers(query, retrieval.LayeredResults{
	Candidates:  candidateResults,
	Reranked:    rerankedResults,
	Diversified: diversifiedResults,
	Final:       finalResults,
}, 10)

report := retrieval.TuneWeights(evalCases, retrieval.TuneConfig{
	K: 10,
	Weights: []retrieval.Weights{
		retrieval.DefaultWeights(),
		{Embedding: 0.8, Keyword: 0.2},
	},
	MMRLambdas: []float64{0.3, 0.5, 0.7},
	Constraints: retrieval.TuneConstraints{
		MinRecallAtK: 0.8,
		MinNDCGAtK:   0.7,
	},
})
_ = segments
_ = layers
_ = report

Build a two-stage vector search

binaryIndex, report := vectors.NewBinaryIndexChecked(blobs, groups, chunkIndexes, dims)
if report.IndexedRows == 0 {
	return
}

candidates, ok := binaryIndex.SearchCandidates(queryVec)
if !ok {
	return
}

keys := make([]vectors.IndexKey, len(candidates))
for i, c := range candidates {
	keys[i] = vectors.IndexKey{Group: c.Group, ChunkIndex: c.ChunkIndex}
}

exactIndex := vectors.NewExactIndex(dims, chunks, arena)
results, ok := exactIndex.SearchKeys(queryVec, 10, 0.7, keys)
_ = results
_ = ok

Use BinaryIndex as a compact Hamming-distance pre-filter and ExactIndex for exact reranking. Checked constructors return IndexBuildReport so callers can log skipped rows, bad spans, missing metadata, and dimension mismatches.

Persist an index manifest alongside any caller-owned serialized index so stale artifacts can be rejected before search:

manifest := binaryIndex.Manifest(vectors.IndexManifestIdentity{
	EmbeddingModelID: "local-embedder-v1",
	ChunkStrategy:    "markdown-aware",
	ChunkSize:        1200,
	ChunkOverlap:     100,
})
if err := manifest.Validate(vectors.IndexManifestIdentity{Dims: dims}); err != nil {
	panic(err)
}

Assign documents to categories

manager := clustering.NewCategoryManager(0.7)
manager.AddCategory(clustering.NewCategory("technology", []float64{1, 0, 0}, "seed.md"))

result := manager.FindBestCategoryWithConfidence([]float64{0.98, 0.2, 0}, 0.7)
if result.Category != nil {
	_ = result.Category.UpdateCentroid([]float64{0.98, 0.2, 0})
}

The clustering package performs no network calls and has no model dependency. You supply normalized vectors.

Extract and graph relationships

result := extract.FromChunk("note-001", 0, "Daily Note", "See [[Project Alpha#Risks]].")
graph := graphbuild.FromExtractResult(result)
_ = graph

graph/extract ships wikilink and code-symbol extractors. Domain extractors register through the public Builder API — the external github.com/dotcommander/genealogy module calls extract.RegisterExtractor in its init to contribute person/parentage edges. Import graph subpackages directly:

import (
	"github.com/dotcommander/reliquary/graph/digraph"
	"github.com/dotcommander/reliquary/graph/extract"
	"github.com/dotcommander/reliquary/graph/graphbuild"
)

Verify

Core packages and optional root adapters:

./scripts/check-boundaries.sh
just verify-core
go test ./storage/...
go vet ./storage/...
go test ./support/... ./document ./provenance ./ingest ./workflow ./events ./embeddings ./llm ./webfetch ./media ./storage ./testkit ./chunking ./lexical ./vectors/... ./retrieval ./clustering ./dedup ./graph/... ./textutil ./observability ./observability/logging
go vet ./support/... ./document ./provenance ./ingest ./workflow ./events ./embeddings ./llm ./webfetch ./media ./storage ./testkit ./chunking ./lexical ./vectors/... ./retrieval ./clustering ./dedup ./graph/... ./textutil ./observability ./observability/logging

Adapter/runtime sibling modules:

just verify-adapters
cd providers/dotcommander/embeddings && go test ./...
cd providers/dotcommander/embeddings && go vet ./...
cd providers/openai && go test ./...
cd providers/openai && go vet ./...
cd runtime/config && go test ./...
cd runtime/config && go vet ./...
cd runtime/cli && go test ./...
cd runtime/cli && go vet ./...
cd runtime/web && go test ./...
cd runtime/web && go vet ./...
cd storage/migrate && go test ./...
cd storage/migrate && go vet ./...
cd storage/postgres && go test ./...
cd storage/postgres && go vet ./...
cd storage/redis && go test ./...
cd storage/redis && go vet ./...
cd storage/sqlite && go test ./...
cd storage/sqlite && go vet ./...

Tool sibling modules:

cd tools/gokart && go test ./...
cd tools/gokart && go vet ./...

Vulnerability checks must cover both Go modules:

govulncheck ./...
cd providers/dotcommander/embeddings && govulncheck ./...

Everything in this workspace:

just verify

For vector hot paths:

go test -bench . -benchmem ./vectors

Notes

• Production code in vectors is standard-library only.
• chunking uses github.com/yuin/goldmark for Markdown-aware strategies and github.com/pkoukk/tiktoken-go for token-based chunking.
• The root module is a library module, not a CLI. Agent-memory, ML-metrics, and genealogy extraction live in their own sibling modules (memory-core, mlmetrics, genealogy).

Documentation

Overview

Package reliquary is the high-level facade for the reliquary retrieval toolkit. It wires the chunking, embedding, and retrieval pipelines behind a small App value with sensible defaults, so a caller can ingest documents and run hybrid search without assembling each lower-level package by hand.

Construct an App with New (bring your own embedder) or with the zero-config Quickstart/InMemory conventions. The App owns no resources and starts nothing; every call takes an explicit context.

Package reliquary is the high-level facade for local retrieval pipelines.

It wires document chunking, caller-owned embedding, in-memory storage, hybrid reranking, and optional MMR diversification behind a small App. Provider SDKs, storage drivers, model runtimes, prompts, and product policy stay outside the root package.

Index

• Variables
• type App
  • func InMemory(dim int) *App
  • func New(opts ...Option) (*App, error)
  • func Quickstart() *App
  • func (a *App) Ingest(ctx context.Context, docs ...document.Document) (int, error)
  • func (a *App) Search(ctx context.Context, query string, opts ...SearchOption) ([]*retrieval.Result, error)
• type Option
  • func WithChunker(strategy chunking.Strategy, size, overlap int) Option
  • func WithEmbedder(e embeddings.Embedder) Option
  • func WithStore(s Store) Option
  • func WithWeights(w retrieval.Weights) Option
• type SearchOption
  • func TopK(k int) SearchOption
  • func WithMMR(lambda float64) SearchOption
• type Store
  • func NewMemoryStore() Store

Examples

• Quickstart

Variables

var ErrNilApp = errors.New("reliquary: nil app")

ErrNilApp is returned when a method is called on a nil App.

var ErrNoEmbedder = errors.New("reliquary: an embedder is required (use WithEmbedder)")

ErrNoEmbedder is returned by New when no embedder is supplied. There is no sensible default embedding model, so the embedder is the one required dependency.

Types

type App

type App struct {
	// contains filtered or unexported fields
}

App is a wired retrieval pipeline. Construct it with New, Quickstart, or InMemory.

func InMemory

func InMemory(dim int) *App

InMemory returns a ready-to-use RAG App backed entirely by in-process, dependency-free defaults: the deterministic hashing embedder at the given dimension plus New's defaults (in-memory store, smart-boundary chunking, and the default scoring weights). It is ideal for demos and tests; the hashing embedder is not production retrieval quality.

Every input is known-good, so InMemory does not return an error; the only way New can fail is a missing embedder, which InMemory always supplies.

func New

func New(opts ...Option) (*App, error)

New builds an App from the supplied options. An embedder is required; all other settings default: an in-memory store, smart-boundary chunking at 220 characters with no overlap, and the default hybrid-scoring weights.

func Quickstart

func Quickstart() *App

Quickstart is InMemory(256) — the one-obvious-default entry point for trying reliquary with zero configuration.

Example

package main

import (
	"context"
	"fmt"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/pipeline/document"
)

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	_, _ = app.Ingest(ctx, document.Document{
		ID:   "doc-1",
		Text: "Go uses a concurrent garbage collector.",
	})

	hits, _ := app.Search(ctx, "garbage collector", reliquary.TopK(1))
	fmt.Println(len(hits), hits[0].ID != "")
}

Output:
1 true

func (*App) Ingest

func (a *App) Ingest(ctx context.Context, docs ...document.Document) (int, error)

Ingest chunks each document with the App's strategy, embeds every chunk with the App's embedder, and stores the embedded chunks. It returns the number of chunks produced.

func (*App) Search

func (a *App) Search(ctx context.Context, query string, opts ...SearchOption) ([]*retrieval.Result, error)

Search embeds the query, hybrid-scores every stored chunk, and applies the supplied search options (TopK, WithMMR). It returns ranked results, best first.

type Option

type Option func(*App)

Option configures an App. Every default is explicit and overridable.

func WithChunker

func WithChunker(strategy chunking.Strategy, size, overlap int) Option

WithChunker overrides the chunking strategy and chunk sizing.

func WithEmbedder

func WithEmbedder(e embeddings.Embedder) Option

WithEmbedder sets the embedder used for ingestion and queries (required).

func WithStore

func WithStore(s Store) Option

WithStore overrides the default in-memory store.

func WithWeights

func WithWeights(w retrieval.Weights) Option

WithWeights overrides the default hybrid-scoring weights.

type SearchOption

type SearchOption func(*searchConfig)

SearchOption tunes a single Search call. Options never mutate the App.

func TopK

func TopK(k int) SearchOption

TopK limits Search to the k highest-ranked results. k <= 0 returns no results. Without it, Search returns all scored results.

func WithMMR

func WithMMR(lambda float64) SearchOption

WithMMR enables maximal-marginal-relevance diversification with the given lambda (0 = maximize diversity, 1 = maximize relevance).

type Store

type Store interface {
	// Put appends embedded chunks to the store.
	Put(ctx context.Context, items []*retrieval.Result) error
	// All returns every stored chunk.
	All(ctx context.Context) ([]*retrieval.Result, error)
}

Store holds embedded, scorable chunks between Ingest and Search. It is deliberately tiny: the facade needs somewhere to keep ingested chunks. Callers who already own a vector database implement this over their driver in their own module — the root reliquary package never imports storage drivers.

func NewMemoryStore

func NewMemoryStore() Store

NewMemoryStore returns an in-process Store backed by a slice. It is the default store for New and Quickstart.