README
¶
Reliquary
Reliquary is a Go toolkit for document ingestion and retrieval. The root
facade combines chunking, embeddings, candidate retrieval, hybrid scoring, and
optional MMR reranking behind a small App API.
go get github.com/dotcommander/reliquary@v0.8.0
package main
import (
"context"
"log"
"github.com/dotcommander/reliquary"
"github.com/dotcommander/reliquary/document"
)
func main() {
ctx := context.Background()
app := reliquary.Quickstart()
if _, err := app.Ingest(ctx, document.Document{
ID: "doc-1", Text: "Go uses a concurrent garbage collector.",
}); err != nil {
log.Fatal(err)
}
results, err := app.Search(ctx, "garbage collector", reliquary.TopK(5))
if err != nil {
log.Fatal(err)
}
log.Printf("found %d result(s)", len(results))
}
This quickstart uses deterministic in-memory embeddings for demos and tests.
Production applications should inject their own embedder and index with
New, WithEmbedder, WithIndex, and WithIndexIdentity.
The index identity is required for New and must change whenever the embedding
space or chunking policy changes. A mismatched populated index is rejected even
when dimensions match. To rebuild intentionally, call app.ResetIndex(ctx);
this permanently deletes every indexed chunk before re-ingestion. Quickstart
and InMemory supply a deterministic identity for their built-in hashing setup.
Restrict candidate retrieval by reserved fields (id, document_id, or
filename) or scalar metadata while retaining final reranking and MMR:
results, err := app.Search(ctx, "search text",
reliquary.WithFilter(map[string]any{"project": "reliquary"}),
reliquary.TopK(5),
)
The default index is concurrency-safe and in-memory. Use WithIndex to inject
another implementation.
Adapters
adapter/openaiadapts an injectedopenai.Clientto the embedding contract.adapter/postgresprovides bounded pgvector candidate retrieval.adapter/sqliteprovides bounded FTS5 candidate retrieval with final ranking performed by Reliquary.
Database constructors validate configuration and perform no migrations. Call
the adapter's Migrate(ctx) method explicitly before use. Callers retain
ownership of database pools, connections, credentials, transports, and retry
policy.
Ownership
Product memory, graph behavior, and generic application infrastructure are intentionally outside this module. See the v0.5 migration guide for removed ownership surfaces and the v0.6 migration guide for the current public import paths.
Reliquary v0.7 uses Index as its only persistence seam. The deprecated
Store compatibility API was removed.
Project policies
Verify
GOWORK=off go build ./...
GOWORK=off go test ./...
GOWORK=off go vet ./...
go test -race . ./index/... ./adapter/...
./scripts/check-boundaries.sh
./scripts/verify-modules.sh
Documentation
¶
Overview ¶
Package reliquary is the high-level facade for the reliquary retrieval toolkit. It wires the chunking, embedding, and retrieval pipelines behind a small App value with sensible defaults, so a caller can ingest documents and run hybrid search without assembling each lower-level package by hand.
Construct an App with New (bring your own embedder) or with the zero-config Quickstart/InMemory conventions. The App owns no resources and starts nothing; every call takes an explicit context.
Package reliquary is the high-level facade for local retrieval pipelines.
It wires document chunking, caller-owned embedding, in-memory storage, hybrid reranking, and optional MMR diversification behind a small App. Provider SDKs, storage drivers, model runtimes, prompts, and product policy stay outside the root package.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrIdentityMismatch = indexcontract.ErrIdentityMismatch
ErrIdentityMismatch reports incompatible index identities.
var ErrInvalidIndexIdentity = errors.New("reliquary: index identity must not be empty")
ErrInvalidIndexIdentity is returned when WithIndexIdentity is empty.
var ErrNilApp = errors.New("reliquary: nil app")
ErrNilApp is returned when a method is called on a nil App.
var ErrNoEmbedder = errors.New("reliquary: an embedder is required (use WithEmbedder)")
ErrNoEmbedder is returned by New when no embedder is supplied. There is no sensible default embedding model, so the embedder is the one required dependency.
var ErrResetUnsupported = errors.New("reliquary: index does not support reset")
ErrResetUnsupported reports an Index without the optional reset contract.
Functions ¶
This section is empty.
Types ¶
type App ¶
type App struct {
// contains filtered or unexported fields
}
App is a wired retrieval pipeline. Construct it with New, Quickstart, or InMemory.
func InMemory ¶
InMemory returns a ready-to-use RAG App backed entirely by in-process, dependency-free defaults: the deterministic hashing embedder at the given dimension plus New's defaults (in-memory index, smart-boundary chunking, and the default scoring weights). It is ideal for demos and tests; the hashing embedder is not production retrieval quality.
Every input is known-good, so InMemory does not return an error; it supplies both required New options itself.
func New ¶
New builds an App from the supplied options. An embedder and a non-empty index identity are required; all other settings default: an in-memory index, smart-boundary chunking at 220 characters with no overlap, and the default hybrid-scoring weights.
func Quickstart ¶
func Quickstart() *App
Quickstart is InMemory(256) — the one-obvious-default entry point for trying reliquary with zero configuration.
Example ¶
package main
import (
"context"
"fmt"
"github.com/dotcommander/reliquary"
"github.com/dotcommander/reliquary/document"
)
func main() {
app := reliquary.Quickstart()
ctx := context.Background()
_, _ = app.Ingest(ctx, document.Document{
ID: "doc-1",
Text: "Go uses a concurrent garbage collector.",
})
hits, _ := app.Search(ctx, "garbage collector", reliquary.TopK(1))
fmt.Println(len(hits), hits[0].ID != "")
}
Output: 1 true
func (*App) Ingest ¶
Ingest chunks each document with the App's strategy, embeds every chunk with the App's embedder, and stores the embedded chunks. It returns the number of chunks produced.
func (*App) ResetIndex ¶ added in v0.7.0
ResetIndex destructively removes every indexed chunk. It is intended as the explicit rebuild escape hatch after changing index identity; source documents must be ingested again. Custom indexes may opt in via index.Resetter.
type Index ¶ added in v0.7.0
type Index = indexcontract.Index
Index is the candidate-retrieval boundary used by App.
func NewMemoryIndex ¶ added in v0.7.0
func NewMemoryIndex() Index
NewMemoryIndex returns the default concurrency-safe in-process Index.
type IndexQuery ¶ added in v0.7.0
type IndexQuery = indexcontract.IndexQuery
IndexQuery describes candidate retrieval.
type Option ¶
type Option func(*App)
Option configures an App. Every default is explicit and overridable.
func WithChunker ¶
WithChunker overrides the chunking strategy and chunk sizing.
func WithEmbedder ¶
func WithEmbedder(e embeddings.Embedder) Option
WithEmbedder sets the embedder used for ingestion and queries (required).
func WithIndex ¶ added in v0.7.0
WithIndex sets the index used for ingestion and candidate retrieval. A nil Index is unset and falls back to the default in-memory index.
func WithIndexIdentity ¶ added in v0.7.0
WithIndexIdentity identifies the embedding vector space and chunking policy used by this App. Index implementations reject reads and writes against a different identity, even when vector dimensions match.
func WithWeights ¶
WithWeights overrides the default hybrid-scoring weights.
type SearchOption ¶
type SearchOption func(*searchConfig)
SearchOption tunes a single Search call. Options never mutate the App.
func CandidateLimit ¶ added in v0.7.0
func CandidateLimit(n int) SearchOption
CandidateLimit bounds index candidate retrieval independently of final TopK. Values less than or equal to zero request implementation-defined or unbounded candidate retrieval.
func TopK ¶
func TopK(k int) SearchOption
TopK limits Search to the k highest-ranked results. k <= 0 returns no results. Without it, Search returns all scored results.
func WithFilter ¶ added in v0.7.0
func WithFilter(filter map[string]any) SearchOption
WithFilter restricts index candidates by reserved result fields (id, document_id, and filename) or scalar metadata values. The filter is snapshotted when the option is created; later caller mutations have no effect. Compound values such as slices and maps cause Search to return an error before embedding the query.
func WithMMR ¶
func WithMMR(lambda float64) SearchOption
WithMMR enables maximal-marginal-relevance diversification with the given lambda (0 = maximize diversity, 1 = maximize relevance).
Source Files
Directories
¶
| Path | Synopsis |
|---|---|
|
adapter
|
|
|
openai
Package openai adapts the official OpenAI SDK to Reliquary's embedding contract.
|
Package openai adapts the official OpenAI SDK to Reliquary's embedding contract. |
|
postgres
Package postgres provides a PostgreSQL pgvector-backed Reliquary index.
|
Package postgres provides a PostgreSQL pgvector-backed Reliquary index. |
|
sqlite
Package sqlite provides a SQLite-backed Reliquary candidate index.
|
Package sqlite provides a SQLite-backed Reliquary candidate index. |
|
Package chunking splits text into reusable chunks for search, retrieval, summarization, and LLM context assembly.
|
Package chunking splits text into reusable chunks for search, retrieval, summarization, and LLM context assembly. |
|
contracts
|
|
|
provenance
Package provenance defines source, artifact, claim, evidence-link, and lineage primitives.
|
Package provenance defines source, artifact, claim, evidence-link, and lineage primitives. |
|
Package dedup provides generic duplicate and near-duplicate detection for text-like values using string hash strategies.
|
Package dedup provides generic duplicate and near-duplicate detection for text-like values using string hash strategies. |
|
Package document defines provider-neutral document, section, and span primitives.
|
Package document defines provider-neutral document, section, and span primitives. |
|
Package embed provides deterministic, dependency-free embedding helpers for demos, tests, and reliquary.Quickstart.
|
Package embed provides deterministic, dependency-free embedding helpers for demos, tests, and reliquary.Quickstart. |
|
Package embeddings defines provider-neutral embedding model, request, result, vector, cache-key, and dimension-validation contracts.
|
Package embeddings defines provider-neutral embedding model, request, result, vector, cache-key, and dimension-validation contracts. |
|
examples
|
|
|
internal/examplekit
Package examplekit holds demo-only helpers for examples.
|
Package examplekit holds demo-only helpers for examples. |
|
postgres-index
command
|
|
|
provider-openai-wiring
command
|
|
|
quickstart-rag
command
Command quickstart-rag is the short, copyable retrieval path.
|
Command quickstart-rag is the short, copyable retrieval path. |
|
rag-ingest-retrieve
command
Command rag-ingest-retrieve demonstrates an end-to-end retrieval pipeline assembled from reliquary packages the way a real consumer would:
|
Command rag-ingest-retrieve demonstrates an end-to-end retrieval pipeline assembled from reliquary packages the way a real consumer would: |
|
retrieval-calibration-tune
command
Command retrieval-calibration-tune demonstrates the retrieval quality-control layer: captured runs, threshold checks, score-reference calibration, and deterministic weight tuning.
|
Command retrieval-calibration-tune demonstrates the retrieval quality-control layer: captured runs, threshold checks, score-reference calibration, and deterministic weight tuning. |
|
Package index defines the candidate-retrieval boundary used by Reliquary.
|
Package index defines the candidate-retrieval boundary used by Reliquary. |
|
indextest
Package indextest provides a reusable contract suite for Index implementations.
|
Package indextest provides a reusable contract suite for Index implementations. |
|
inmem
Package inmem provides Reliquary's concurrency-safe in-process index.
|
Package inmem provides Reliquary's concurrency-safe in-process index. |
|
internal
|
|
|
hash
Package hash provides stable content hash helpers for reliquary primitives.
|
Package hash provides stable content hash helpers for reliquary primitives. |
|
sqltx
Package sqltx provides generic transaction lifecycle semantics shared by storage adapters.
|
Package sqltx provides generic transaction lifecycle semantics shared by storage adapters. |
|
validate
Package validate provides small validation helpers for primitive packages.
|
Package validate provides small validation helpers for primitive packages. |
|
pipeline
|
|
|
ingest
Package ingest defines generic reader, decoder, mapper, sink, batch, report, and cursor contracts for caller-owned ingestion pipelines.
|
Package ingest defines generic reader, decoder, mapper, sink, batch, report, and cursor contracts for caller-owned ingestion pipelines. |
|
lexical
Package lexical provides provider-neutral lexical search primitives.
|
Package lexical provides provider-neutral lexical search primitives. |
|
Package retrieval is a hybrid scoring and reranking layer for local retrieval pipelines.
|
Package retrieval is a hybrid scoring and reranking layer for local retrieval pipelines. |
|
Package textutil provides lightweight, stdlib-only text helpers for keyword extraction, theme detection, fuzzy alias matching, fragment location, and title normalization.
|
Package textutil provides lightweight, stdlib-only text helpers for keyword extraction, theme detection, fuzzy alias matching, fragment location, and title normalization. |
|
Package vectors provides small vector math helpers for embedding and retrieval systems.
|
Package vectors provides small vector math helpers for embedding and retrieval systems. |
|
clustering
Package clustering performs online and offline clustering over caller-supplied vector embeddings.
|
Package clustering performs online and offline clustering over caller-supplied vector embeddings. |
|
pq
Package pq implements Product Quantization for efficient vector compression.
|
Package pq implements Product Quantization for efficient vector compression. |