reliquary

README

Reliquary

Reliquary helps Go applications turn documents into useful context for AI features. Start in memory, then bring your own embedder and index when you need production storage and retrieval quality.

go get github.com/dotcommander/reliquary@v0.11.0

package main

import (
	"context"
	"fmt"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/document"
)

func main() {
	ctx := context.Background()
	app := reliquary.Quickstart()

	_, err := app.Ingest(ctx, document.Document{
		ID: "notes",
		Text: "The Go build cache lives in ~/Library/Caches/go-build.",
	})
	if err != nil {
		panic(err)
	}

	hits, err := app.Search(ctx, "where is the Go build cache?", reliquary.TopK(1))
	if err != nil {
		panic(err)
	}
	fmt.Println(hits[0].Content)
}

Quickstart uses a deterministic in-memory embedder and index. It is useful for examples and tests, but it is not a production embedding model.

What you can build

• Give an AI assistant useful context from project notes, READMEs, runbooks, or issue summaries. App.Ingest chunks and indexes each document; App.Search returns the best passages to add to your model prompt.

• Keep retrieval inside the current project or tenant with scalar metadata:

  _, err := app.Ingest(ctx, document.Document{
      ID:       "runbook",
      Text:     "Restart the worker with launchctl kickstart.",
      Metadata: document.Metadata{"project": "reliquary"},
  })
  hits, err := app.Search(ctx, "restart the worker",
      reliquary.WithFilter(map[string]any{"project": "reliquary"}),
      reliquary.TopK(3),
  )
  

  Filters scope retrieval; they are not an authorization boundary.

• Build a less repetitive context pack by fetching more candidates than you return and using MMR to balance relevance with diversity:

  hits, err := app.Search(ctx, question,
      reliquary.CandidateLimit(20),
      reliquary.TopK(5),
      reliquary.WithMMR(0.5),
  )
  

• Fuse independent vector and lexical rankings before TopK or MMR:

  hits, err := app.Search(ctx, question,
      reliquary.CandidateLimit(50),
      reliquary.WithRRF(60),
      reliquary.TopK(5),
  )
  

  WithRRF asks the configured index for one vector-only ranking and one text-only ranking, then combines their result IDs with reciprocal rank fusion. It does not add BM25 or another lexical engine; your index supplies both rankings. Each lane receives the candidate limit independently. Values at or below zero and non-finite values use the standard RRF constant 60.

• Add a per-search cross-encoder after hybrid scoring and before TopK or MMR:

  hits, err := app.Search(ctx, question,
      reliquary.CandidateLimit(50),
      reliquary.WithReranker(bgeReranker),
      reliquary.TopK(5),
  )
  

  The reranker receives hybrid-ranked candidates by default or RRF-ranked candidates when WithRRF is enabled, and returns one finite score in [0,1] for each candidate. A reranker failure or malformed response fails the search; Reliquary does not fall back to the preceding ranking.

• Inspect how retained candidates moved through hybrid scoring, RRF, an external reranker, and MMR:

  hits, err := app.Search(ctx, question,
      reliquary.CandidateLimit(50),
      reliquary.WithExplain(),
      reliquary.TopK(5),
  )
  trace := hits[0].Explain
  fmt.Println(trace.Hybrid.Raw.Keyword, trace.FinalRank)
  

  Explain is nil unless WithExplain is supplied. Explanations are typed, ephemeral result data; they are not written to the index or source metadata. The keyword value is token overlap computed by the hybrid scorer, not a backend-native BM25 score.

• Embed several questions in one call while preserving blank positions, then render selected passages as neutral context:

  rows, err := app.SearchBatch(ctx, []string{question, followup})
  hits := rows[0]
  promptBlock, err := retrieval.FormatContext(hits,
      retrieval.WithHeader("[Source: %s, Lines: %d-%d]"),
      retrieval.WithMaxTokens(2048, tokenCounter),
  )
  

  The token counter is caller-supplied, so it can match the model that will consume the prompt. Context includes only a contiguous prefix of complete result blocks; headers and separators count toward the budget.

• Construct bounded text documents from streams with document.FromReader. Input defaults to a 16 MiB limit and must be valid UTF-8; filenames label documents but do not select parsers or infer formats.

• Read a local directory in deterministic, resumable batches with pipeline/ingest/fs, decode each file with its path metadata through pipeline/ingest.NewRecordPipeline, then persist mapped results with pipeline/indexsink.

Production wiring

Production applications inject an embedding.Embedder, an index.Index, and an explicit index identity:

app, err := reliquary.New(
	reliquary.WithEmbedder(embedder),
	reliquary.WithIndex(idx),
	reliquary.WithIndexIdentity("text-embedding-3-small:v1#smart-boundary"),
)

The identity names the embedding space and chunking policy stored in the index. Reliquary rejects reads and writes with a different identity even when the vector dimensions match. Change the identity and call ResetIndex before a deliberate rebuild.

Reliquary includes opt-in adapters for OpenAI and Ollama embeddings, PostgreSQL/pgvector, and SQLite/FTS5. The Ollama adapter targets the native /api/embed endpoint. Adapter constructors perform no network I/O. Database constructors validate configuration but do not run migrations; call the adapter's Migrate(ctx) explicitly. Clients, database handles, credentials, transports, and retry policy remain caller-owned.

App.Ingest treats each document as a complete revision and atomically replaces its previous chunks. Custom indexes must implement index.Index, including ReplaceDocuments, and should run the shared index/indextest contract suite. Custom embedders should run embedding/embeddingtest.

Packages

Package	Purpose
reliquary	High-level App facade, options, and constructors
document	Document value type and bounded UTF-8 reader construction
embedding	Provider-neutral Embedder, request, result, and vector contracts
embed	Deterministic hashing embedder for demos and tests
index	Candidate retrieval contract, in-memory implementation, and indextest suite
chunking	Boundary-aware text, code, sentence, and heading splitters
retrieval	Hybrid scoring, MMR, filtering, evaluation, and neutral context rendering
pipeline/ingest	Generic resumable ingestion contracts and runner
pipeline/ingest/fs	Deterministic, bounded local-directory reader
pipeline/indexsink	pipeline/ingest sink backed by index.Index
pipeline/lexical	Lexical analysis, BM25, and result fusion
dedup, textutil, vector	Retrieval primitives, vector math, quantization, and clustering
adapter/openai	OpenAI embedding adapter
adapter/ollama	Native Ollama embedding adapter
adapter/postgres, adapter/sqlite	Persistent candidate-retrieval adapters

Documentation

• Technical specification and invariants
• Scoring, fusion, reranking, and MMR guide
• Architecture and package boundaries
• Migration to v0.10
• Migration to v0.6
• Migration to v0.5
• Runnable examples

Verify

GOWORK=off go build ./...
GOWORK=off go test ./...
GOWORK=off go vet ./...
go test -race . ./index/... ./adapter/...
./scripts/check-boundaries.sh
./scripts/verify-modules.sh

Contributing · Security · MIT License

Documentation

Overview

Package reliquary is the high-level facade for the reliquary retrieval toolkit. It wires the chunking, embedding, and retrieval pipelines behind a small App value with sensible defaults, so a caller can ingest documents and run hybrid search without assembling each lower-level package by hand.

Construct an App with New (bring your own embedder) or with the zero-config Quickstart/InMemory conventions. The App owns no resources and starts nothing; every call takes an explicit context.

Package reliquary is the high-level facade for local retrieval pipelines.

It wires document chunking, caller-owned embedding, in-memory storage, hybrid reranking, and optional MMR diversification behind a small App. Provider SDKs, storage drivers, model runtimes, prompts, and product policy stay outside the root package.

Index

• Variables
• type App
  • func InMemory(dim int) *App
  • func New(opts ...Option) (*App, error)
  • func Quickstart() *App
  • func (a *App) Ingest(ctx context.Context, docs ...document.Document) (int, error)
  • func (a *App) ResetIndex(ctx context.Context) error
  • func (a *App) Search(ctx context.Context, query string, opts ...SearchOption) ([]*retrieval.Result, error)
  • func (a *App) SearchBatch(ctx context.Context, queries []string, opts ...SearchOption) ([][]*retrieval.Result, error)
• type DocumentReplacement
• type Index
  • func NewMemoryIndex() Index
• type IndexQuery
• type Option
  • func WithChunker(strategy chunking.Strategy, size, overlap int) Option
  • func WithEmbedder(e embedding.Embedder) Option
  • func WithIndex(index Index) Option
  • func WithIndexIdentity(identity string) Option
  • func WithWeights(w retrieval.Weights) Option
• type SearchOption
  • func CandidateLimit(n int) SearchOption
  • func TopK(k int) SearchOption
  • func WithExplain() SearchOption
  • func WithFilter(filter map[string]any) SearchOption
  • func WithMMR(lambda float64) SearchOption
  • func WithRRF(k float64) SearchOption
  • func WithReranker(r retrieval.Reranker) SearchOption

Examples

• App.SearchBatch
• Quickstart
• WithExplain
• WithRRF
• WithReranker

Variables

var ErrDuplicateDocumentID = indexcontract.ErrDuplicateDocumentID

ErrDuplicateDocumentID reports duplicate document identifiers in one Ingest call.

var ErrIdentityMismatch = indexcontract.ErrIdentityMismatch

ErrIdentityMismatch reports incompatible index identities.

var ErrInvalidDocumentID = indexcontract.ErrInvalidDocumentID

ErrInvalidDocumentID reports a blank document identifier passed to Ingest.

var ErrInvalidIndexIdentity = errors.New("reliquary: index identity must not be empty")

ErrInvalidIndexIdentity is returned when WithIndexIdentity is empty.

var ErrNilApp = errors.New("reliquary: nil app")

ErrNilApp is returned when a method is called on a nil App.

var ErrNoEmbedder = errors.New("reliquary: an embedder is required (use WithEmbedder)")

ErrNoEmbedder is returned by New when no embedder is supplied. There is no sensible default embedding model, so the embedder is the one required dependency.

var ErrResetUnsupported = errors.New("reliquary: index does not support reset")

ErrResetUnsupported reports an Index without the optional reset contract.

var ErrResultIDConflict = indexcontract.ErrResultIDConflict

ErrResultIDConflict reports an invalid replacement result-ID collision.

Types

type App

type App struct {
	// contains filtered or unexported fields
}

App is a wired retrieval pipeline. Construct it with New, Quickstart, or InMemory.

func InMemory

func InMemory(dim int) *App

InMemory returns a ready-to-use RAG App backed entirely by in-process, dependency-free defaults: the deterministic hashing embedder at the given dimension plus New's defaults (in-memory index, smart-boundary chunking, and the default scoring weights). It is ideal for demos and tests; the hashing embedder is not production retrieval quality.

Every input is known-good, so InMemory does not return an error; it supplies both required New options itself.

func New

func New(opts ...Option) (*App, error)

New builds an App from the supplied options. An embedder and a non-empty index identity are required; all other settings default: an in-memory index, smart-boundary chunking at 220 characters with no overlap, and the default hybrid-scoring weights.

func Quickstart

func Quickstart() *App

Quickstart is InMemory(256) — the one-obvious-default entry point for trying reliquary with zero configuration.

Example

package main

import (
	"context"
	"fmt"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/document"
)

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	_, _ = app.Ingest(ctx, document.Document{
		ID:   "doc-1",
		Text: "Go uses a concurrent garbage collector.",
	})

	hits, _ := app.Search(ctx, "garbage collector", reliquary.TopK(1))
	fmt.Println(len(hits), hits[0].ID != "")
}

Output:
1 true

func (*App) Ingest

func (a *App) Ingest(ctx context.Context, docs ...document.Document) (int, error)

Ingest chunks each document with the App's strategy, embeds every chunk, and atomically replaces all prior chunks for every supplied document. A document that produces no chunks deletes its prior revision. Document IDs must be non-blank and unique within the call. It returns the number of chunks produced.

func (*App) ResetIndex

func (a *App) ResetIndex(ctx context.Context) error

ResetIndex destructively removes every indexed chunk. It is intended as the explicit rebuild escape hatch after changing index identity; source documents must be ingested again. Custom indexes may opt in via index.Resetter.

func (*App) Search

func (a *App) Search(ctx context.Context, query string, opts ...SearchOption) ([]*retrieval.Result, error)

Search embeds the query, asks the index for candidates, applies weighted hybrid ordering or optional reciprocal rank fusion, then applies an optional external reranker, TopK, and MMR diversification.

func (*App) SearchBatch

func (a *App) SearchBatch(ctx context.Context, queries []string, opts ...SearchOption) ([][]*retrieval.Result, error)

SearchBatch searches multiple queries while preserving their input order. Blank queries produce nil rows. All nonblank queries are embedded in one ordered call, validated as a complete batch, and searched sequentially.

Example

package main

import (
	"context"
	"fmt"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/document"
)

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	_, _ = app.Ingest(ctx, document.Document{ID: "doc-1", Text: "Go uses a concurrent garbage collector."})

	rows, _ := app.SearchBatch(ctx, []string{"garbage collector", "", "Go memory"}, reliquary.TopK(1))
	fmt.Println(len(rows), len(rows[0]), rows[1] == nil, len(rows[2]))
}

Output:
3 1 true 1

type DocumentReplacement

type DocumentReplacement = indexcontract.DocumentReplacement

DocumentReplacement describes one complete document revision for an Index.

type Index

type Index = indexcontract.Index

Index is the candidate-retrieval boundary used by App.

func NewMemoryIndex

func NewMemoryIndex() Index

NewMemoryIndex returns the default concurrency-safe in-process Index.

type IndexQuery

type IndexQuery = indexcontract.IndexQuery

IndexQuery describes candidate retrieval.

type Option

type Option func(*App)

Option configures an App. Every default is explicit and overridable.

func WithChunker

func WithChunker(strategy chunking.Strategy, size, overlap int) Option

WithChunker overrides the chunking strategy and chunk sizing.

func WithEmbedder

func WithEmbedder(e embedding.Embedder) Option

WithEmbedder sets the embedder used for ingestion and queries (required).

func WithIndex

func WithIndex(index Index) Option

WithIndex sets the index used for ingestion and candidate retrieval. A nil or typed-nil Index is unset and falls back to the default in-memory index.

func WithIndexIdentity

func WithIndexIdentity(identity string) Option

WithIndexIdentity identifies the embedding vector space and chunking policy used by this App. Index implementations reject reads and writes against a different identity, even when vector dimensions match.

func WithWeights

func WithWeights(w retrieval.Weights) Option

WithWeights overrides the default hybrid-scoring weights.

type SearchOption

type SearchOption func(*searchConfig)

SearchOption tunes a Search or SearchBatch call. Options never mutate the App.

func CandidateLimit

func CandidateLimit(n int) SearchOption

CandidateLimit bounds index candidate retrieval independently of final TopK. Values less than or equal to zero request implementation-defined or unbounded candidate retrieval.

func TopK

func TopK(k int) SearchOption

TopK limits Search to the k highest-ranked results. k <= 0 returns no results. Without it, Search returns all scored results.

func WithExplain

func WithExplain() SearchOption

WithExplain attaches an ephemeral, typed ranking explanation to each returned result. Explanations are never stored in an Index and cover only candidates retained for facade-level ranking.

Example

package main

import (
	"context"
	"fmt"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/document"
)

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	_, _ = app.Ingest(ctx, document.Document{ID: "doc-1", Text: "Go uses a concurrent garbage collector."})

	hits, _ := app.Search(ctx, "garbage collector", reliquary.TopK(1), reliquary.WithExplain())
	explanation := hits[0].Explain
	fmt.Println(explanation.FinalRank, explanation.Hybrid.Raw.Keyword > 0)
}

Output:
1 true

func WithFilter

func WithFilter(filter map[string]any) SearchOption

WithFilter restricts index candidates by reserved result fields (id, document_id, and filename) or JSON-scalar metadata values. Reserved fields match strings only. Metadata keys must be present; explicit nil matches only JSON null, strings and booleans are type-exact, and finite numbers compare by exact JSON numeric value across accepted Go numeric types and json.Number. The filter is snapshotted when the option is created; later caller mutations have no effect. Compound and non-finite values cause Search to return an error before embedding the query.

func WithMMR

func WithMMR(lambda float64) SearchOption

WithMMR enables maximal-marginal-relevance diversification with the given lambda (0 = maximize diversity, 1 = maximize relevance).

func WithRRF

func WithRRF(k float64) SearchOption

WithRRF enables reciprocal-rank fusion over independent vector-only and text-only candidate searches. Non-finite values and k <= 0 use the standard constant 60. The last WithRRF option wins.

Example

package main

import (
	"context"

	"github.com/dotcommander/reliquary"
)

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	query := "garbage collector"

	hits, err := app.Search(ctx, query,
		reliquary.CandidateLimit(50),
		reliquary.WithRRF(60),
		reliquary.TopK(5),
	)
	_, _ = hits, err
}

func WithReranker

func WithReranker(r retrieval.Reranker) SearchOption

WithReranker adds an external reranking stage after hybrid scoring or RRF and before TopK and MMR. The last WithReranker option wins; nil and typed-nil values disable the stage. Concurrent Search calls may invoke the same Reranker concurrently, so implementations must provide any required synchronization.

Example

package main

import (
	"context"

	"github.com/dotcommander/reliquary"
	"github.com/dotcommander/reliquary/retrieval"
)

type exampleReranker struct{}

func (exampleReranker) Rerank(_ context.Context, _ string, candidates []*retrieval.Result) ([]float64, error) {
	scores := make([]float64, len(candidates))
	for i := range scores {
		scores[i] = 1 - float64(i)/float64(len(scores))
	}
	return scores, nil
}

func main() {
	app := reliquary.Quickstart()
	ctx := context.Background()
	bgeReranker := exampleReranker{}

	hits, err := app.Search(
		ctx,
		"garbage collector",
		reliquary.CandidateLimit(50),
		reliquary.WithReranker(bgeReranker),
		reliquary.TopK(5),
	)
	_, _ = hits, err
}