Documentation
¶
Overview ¶
Package amoxtli provides a multi-backend document indexing and file-ingestion library: full-text search (bleve), vector search (sqlite-vec), weighted result merging, markdown chunking, file conversion and snapshot/restore of indexes.
Index ¶
- Variables
- type Codex
- func (c *Codex) Backup(ctx context.Context) (io.ReadCloser, error)
- func (c *Codex) CancelTask(ctx context.Context, id task.ID) error
- func (c *Codex) CheckGrounding(ctx context.Context, query string, results []*index.SearchResult) (*retrieval.GroundingResult, error)
- func (c *Codex) CleanupIndex(ctx context.Context, collections ...model.CollectionID) (task.ID, error)
- func (c *Codex) Close() error
- func (c *Codex) CreateCollection(ctx context.Context, label string) (model.CollectionID, error)
- func (c *Codex) DeleteBySource(ctx context.Context, source *url.URL) error
- func (c *Codex) GetSectionsByIDs(ctx context.Context, ids []model.SectionID) (map[model.SectionID]model.Section, error)
- func (c *Codex) Index() index.Index
- func (c *Codex) IndexFile(ctx context.Context, collectionID model.CollectionID, filename string, ...) (task.ID, error)
- func (c *Codex) ListTasks(ctx context.Context) ([]task.StateHeader, error)
- func (c *Codex) Manager() *ingest.Manager
- func (c *Codex) Reindex(ctx context.Context) (task.ID, error)
- func (c *Codex) ReindexCollection(ctx context.Context, collectionID model.CollectionID) (task.ID, error)
- func (c *Codex) Restore(ctx context.Context, r io.Reader) error
- func (c *Codex) Search(ctx context.Context, query string, funcs ...SearchOption) ([]*index.SearchResult, error)
- func (c *Codex) SearchIterative(ctx context.Context, query string, funcs ...SearchOption) (*retrieval.Result, error)
- func (c *Codex) SearchPage(ctx context.Context, query string, funcs ...SearchOption) (*SearchPage, error)
- func (c *Codex) TaskState(ctx context.Context, id task.ID) (*task.State, error)
- type IndexFileOption
- type IndexFileOptions
- type Indexer
- type Option
- func WithCloseTimeout(d time.Duration) Option
- func WithDisableHyDE() Option
- func WithDisableJudge() Option
- func WithFileConverter(fc convert.Converter) Option
- func WithGroundingCheck() Option
- func WithGroundingFailOpen() Option
- func WithGroundingMinScore(minScore float64) Option
- func WithGroundingMode(mode retrieval.GroundingMode) Option
- func WithIndex(idx index.Index) Option
- func WithIndexers(indexers ...Indexer) Option
- func WithIterativeRetrieval(rounds int) Option
- func WithLLMClient(client llm.Client) Option
- func WithMaxTotalWords(n int) Option
- func WithMaxWordsPerSection(n int) Option
- func WithObservability() Option
- func WithPersistentTasks(stagingDir string) Option
- func WithQueryDecomposition(maxSubQueries int) Option
- func WithReranking() Option
- func WithSnapshotBoundary(boundary string) Option
- func WithSourceCode(registry *sourcecode.Registry) Option
- func WithStore(store ingest.Store) Option
- func WithTaskParallelism(n int) Option
- func WithTaskRunner(runner task.Runner) Option
- type SearchOption
- type SearchOptions
- type SearchPage
Constants ¶
This section is empty.
Variables ¶
var ( ErrNotFound = ingest.ErrNotFound ErrCanceled = task.ErrCanceled )
Functions ¶
This section is empty.
Types ¶
type Codex ¶
type Codex struct {
// contains filtered or unexported fields
}
Codex is the main embedded instance: a store, index pipeline and task runner behind a single API.
func New ¶
New creates a new embedded Codex instance.
A store (WithStore) and an index (WithIndex or WithIndexers) are required; build them with the dedicated constructors, e.g. gorm.NewSQLiteStore / gorm.NewPostgresStore for the store and bleve.OpenOrCreate / sqlitevec.NewIndex / postgres.NewIndex for the indexers. The caller owns the resources it constructs and must Close them; Codex.Close only stops the task runner.
func (*Codex) Backup ¶
Backup streams a snapshot of the index and the document store as a multipart archive.
func (*Codex) CancelTask ¶ added in v0.0.3
CancelTask cancels a scheduled or running task.
func (*Codex) CheckGrounding ¶ added in v0.0.2
func (c *Codex) CheckGrounding(ctx context.Context, query string, results []*index.SearchResult) (*retrieval.GroundingResult, error)
CheckGrounding evaluates whether the given search results support a reliable, grounded answer to the query, returning a verdict (status + score + explanation). It is a standalone step, fully decoupled from Search: run Search first, then pass its results here to decide whether to trust them or abstain. Requires WithGroundingCheck and an LLM client; otherwise it returns an error.
func (*Codex) CleanupIndex ¶
func (c *Codex) CleanupIndex(ctx context.Context, collections ...model.CollectionID) (task.ID, error)
CleanupIndex schedules a cleanup of orphaned documents and obsolete index entries.
func (*Codex) Close ¶
Close stops the task runner, waiting up to the configured close timeout (WithCloseTimeout, default 30s) for in-flight indexing tasks to drain, then removes the ingestion staging directory. Resources passed in through WithStore, WithIndex/WithIndexers and WithTaskRunner are owned by the caller and must be closed by them.
func (*Codex) CreateCollection ¶
CreateCollection creates a new collection and returns its ID.
func (*Codex) DeleteBySource ¶
DeleteBySource removes all documents and index entries for the given source URL.
func (*Codex) GetSectionsByIDs ¶
func (c *Codex) GetSectionsByIDs(ctx context.Context, ids []model.SectionID) (map[model.SectionID]model.Section, error)
GetSectionsByIDs returns the sections matching the given IDs, typically used to retrieve the content behind search results.
func (*Codex) IndexFile ¶
func (c *Codex) IndexFile(ctx context.Context, collectionID model.CollectionID, filename string, r io.Reader, funcs ...IndexFileOption) (task.ID, error)
IndexFile indexes a file into the given collection. Returns a task.ID that can be used to track progress via TaskState.
func (*Codex) ListTasks ¶ added in v0.0.3
ListTasks returns the headers of every task known to the runner.
func (*Codex) ReindexCollection ¶
func (c *Codex) ReindexCollection(ctx context.Context, collectionID model.CollectionID) (task.ID, error)
ReindexCollection schedules a rebuild of the index for a single collection.
func (*Codex) Search ¶
func (c *Codex) Search(ctx context.Context, query string, funcs ...SearchOption) ([]*index.SearchResult, error)
Search performs a semantic search across the indexed documents. It returns a single page of results (metadata filtering via WithSearchFilter and reranking via WithReranking still apply); use SearchPage when the pagination cursor is needed.
func (*Codex) SearchIterative ¶ added in v0.0.2
func (c *Codex) SearchIterative(ctx context.Context, query string, funcs ...SearchOption) (*retrieval.Result, error)
SearchIterative runs the explicit re-retrieval orchestration on top of Search: optional query decomposition and, when a grounding checker and reformulator are configured, grounding-gated iterative re-retrieval. It returns the fused evidence together with the final grounding verdict (retrieval.Result). It is a separate entry point from Search (which stays a plain, single-shot retrieval) and from CheckGrounding; with none of the orchestration options enabled it is equivalent to Search.
func (*Codex) SearchPage ¶ added in v0.0.3
func (c *Codex) SearchPage(ctx context.Context, query string, funcs ...SearchOption) (*SearchPage, error)
SearchPage performs a search and returns a page of results plus the cursor to resume from (WithSearchCursor). It supports metadata filtering (WithSearchFilter), reranking (WithReranking) and cursor pagination. Unlike Search it always over-fetches a bounded candidate window so that a next cursor can be produced from the first page.
type IndexFileOption ¶
type IndexFileOption func(*IndexFileOptions)
IndexFileOption configures an IndexFile call.
func WithIndexFileCollections ¶
func WithIndexFileCollections(ids ...model.CollectionID) IndexFileOption
WithIndexFileCollections associates the indexed file with the given collection IDs.
func WithIndexFileETag ¶
func WithIndexFileETag(etag string) IndexFileOption
WithIndexFileETag sets the ETag for the indexed file (used for deduplication).
func WithIndexFileMetadata ¶ added in v0.0.3
func WithIndexFileMetadata(metadata map[string]any) IndexFileOption
WithIndexFileMetadata attaches arbitrary metadata to the indexed document, used for metadata filtering at search time (see WithSearchFilter).
func WithIndexFileSource ¶
func WithIndexFileSource(source *url.URL) IndexFileOption
WithIndexFileSource sets the source URL for the indexed file.
type IndexFileOptions ¶
type IndexFileOptions struct {
Source *url.URL
ETag string
Collections []model.CollectionID
// Metadata is arbitrary document metadata used for filtering at search time.
Metadata map[string]any
}
IndexFileOptions holds options for IndexFile calls.
type Indexer ¶
type Indexer struct {
// ID identifies the indexer in the pipeline (e.g. "bleve", "postgres").
ID string
// Index is any implementation of the index.Index contract.
Index index.Index
// Weight is the relative weight of this indexer when merging scores.
Weight float64
}
Indexer identifies a weighted index.Index inside the search pipeline.
type Option ¶
type Option func(*options)
Option is a function that configures a Codex instance.
func WithCloseTimeout ¶ added in v0.0.3
WithCloseTimeout bounds how long Close waits for in-flight indexing tasks to drain before giving up (default 30s). A non-positive duration keeps the default.
func WithDisableHyDE ¶
func WithDisableHyDE() Option
WithDisableHyDE disables the HyDE query transformer.
func WithDisableJudge ¶
func WithDisableJudge() Option
WithDisableJudge disables the Judge results transformer.
func WithFileConverter ¶
WithFileConverter sets a file converter for converting files before indexing.
func WithGroundingCheck ¶ added in v0.0.2
func WithGroundingCheck() Option
WithGroundingCheck enables the fused evidence evaluator: a single LLM call that both relevance-filters the retrieved evidence and judges whether it supports a reliable answer (the grounding γ verdict). It makes CheckGrounding available as a standalone step and gates the re-retrieval loop of SearchIterative. When enabled it replaces the Judge results transformer in the pipeline (Search then relies on the evaluator for relevance filtering), avoiding a redundant LLM pass over the same evidence. Requires an LLM client (WithLLMClient). Disabled by default.
func WithGroundingFailOpen ¶ added in v0.0.3
func WithGroundingFailOpen() Option
WithGroundingFailOpen makes Search degrade gracefully when the grounding evidence evaluator (an LLM call) fails: instead of returning the error, Search logs a warning and returns the retrieved results unfiltered. Without it, a transient LLM failure in the evaluator fails the whole Search. Only meaningful together with WithGroundingCheck. Disabled by default (fail-closed).
func WithGroundingMinScore ¶ added in v0.0.2
WithGroundingMinScore sets the grounding score threshold below which the verdict is considered not confident (default 0.4). This gates iterative re-retrieval only — whether SearchIterative reformulates and searches again — and does NOT control the relevance filtering/demotion of the evidence itself (that is WithGroundingMode). Only meaningful together with WithGroundingCheck.
func WithGroundingMode ¶ added in v0.0.3
func WithGroundingMode(mode retrieval.GroundingMode) Option
WithGroundingMode selects what the evaluator's relevance signal does to the evidence: retrieval.GroundingFilter (default) drops the sections judged irrelevant — maximising precision but truncating recall; retrieval.GroundingDemote keeps them but ranks them last — preserving recall@k while still surfacing the grounded evidence first. Only meaningful together with WithGroundingCheck.
func WithIndex ¶
WithIndex provides a ready-made index.Index, bypassing pipeline composition entirely (including the HyDE/Judge/dedup transformers). Mutually exclusive with WithIndexers. The caller owns and must close the index.
func WithIndexers ¶
WithIndexers declares the set of indexers composing the search pipeline, each with its relative weight. It can be called several times; indexers accumulate.
Any implementation of index.Index can be plugged in; conformance can be verified with the index/testsuite package. Build the backends with their own constructors, e.g. bleve.OpenOrCreate(...), sqlitevec.NewIndex(...) or postgres.NewIndex(...), and wrap each in an Indexer.
func WithIterativeRetrieval ¶ added in v0.0.2
WithIterativeRetrieval enables grounding-driven re-retrieval in SearchIterative: when the evidence is not confidently grounded the query is reformulated and searched again, up to rounds times (rounds <= 0 means 1). Requires WithGroundingCheck and an LLM client.
func WithLLMClient ¶
WithLLMClient sets the LLM client used by the HyDE and Judge transformers.
func WithMaxTotalWords ¶
WithMaxTotalWords bounds the prompt size (in words) shared by the LLM retrieval stages: the Judge transformer, the LLM reranker and the grounding evidence evaluator. Words are only a coarse proxy for tokens, so keep it low enough that the resulting prompt fits the chat endpoint's context window.
func WithMaxWordsPerSection ¶
WithMaxWordsPerSection sets the maximum number of words per document section.
func WithObservability ¶ added in v0.0.3
func WithObservability() Option
WithObservability wraps the configured LLM client (WithLLMClient) with the OpenTelemetry decorator so the HyDE, Judge, grounding and reranker LLM calls emit spans and token/latency metrics under the amoxtli instrumentation scope. Search latency is instrumented unconditionally. It has no effect unless an LLM client is set. Embeddings issued by an index built by the caller (sqlitevec/postgres) are not covered here — wrap that client yourself with llmx.NewObservableClient if you want their metrics too.
func WithPersistentTasks ¶ added in v0.0.3
WithPersistentTasks enables a persistent, gorm-backed task runner that shares the document store's database. Pending indexing tasks — and tasks left running by a previous, interrupted process — are resumed on startup. It requires a gorm-backed store (gorm.NewSQLiteStore / gorm.NewPostgresStore) and a stable stagingDir where files awaiting indexing are kept across restarts (it must be a durable path, not a temporary directory). It is ignored when a runner is supplied explicitly with WithTaskRunner.
func WithQueryDecomposition ¶ added in v0.0.2
WithQueryDecomposition enables splitting a complex question into at most maxSubQueries sub-questions in SearchIterative, searching each and fusing their evidence. Requires an LLM client. maxSubQueries <= 0 keeps the default (3).
func WithReranking ¶ added in v0.0.3
func WithReranking() Option
WithReranking enables LLM-based reranking of the fused search results before pagination: the retrieved candidates are reordered by relevance to the query, reusing the WithMaxTotalWords budget to bound the prompt size. Requires an LLM client (WithLLMClient). Disabled by default.
func WithSnapshotBoundary ¶
WithSnapshotBoundary overrides the multipart boundary used by Backup/Restore.
func WithSourceCode ¶ added in v0.0.4
func WithSourceCode(registry *sourcecode.Registry) Option
WithSourceCode enables source-code indexing for the file extensions registered in the registry (see sourcecode.DefaultRegistry). Source files are parsed with tree-sitter into declaration-level sections and tagged with `type=code` and `language=<name>` metadata, filterable at search time with WithSearchFilter.
func WithStore ¶
WithStore sets the document store. It is required. Build it with gorm.NewSQLiteStore or gorm.NewPostgresStore (or any ingest.Store). The caller owns and must close the store.
func WithTaskParallelism ¶
WithTaskParallelism sets the number of concurrent tasks allowed.
func WithTaskRunner ¶
WithTaskRunner provides a custom task.Runner implementation.
type SearchOption ¶
type SearchOption func(*SearchOptions)
SearchOption configures a Search call.
func WithSearchCollections ¶
func WithSearchCollections(ids ...model.CollectionID) SearchOption
WithSearchCollections restricts the search to the given collection IDs.
func WithSearchCursor ¶ added in v0.0.3
func WithSearchCursor(cursor string) SearchOption
WithSearchCursor resumes pagination after the given opaque cursor (the NextCursor returned by a previous SearchPage).
func WithSearchFilter ¶ added in v0.0.3
func WithSearchFilter(conditions ...index.Condition) SearchOption
WithSearchFilter restricts results to documents whose metadata satisfies the given filter (see index.Eq/Ne/Gt/Gte/Lt/Lte/In). Requires a store implementing ingest.MetadataProvider.
func WithSearchMaxResults ¶
func WithSearchMaxResults(n int) SearchOption
WithSearchMaxResults sets the maximum number of search results (page size).
type SearchOptions ¶
type SearchOptions struct {
// MaxResults is the page size (number of results per page).
MaxResults int
Collections []model.CollectionID
// Filter restricts results to documents whose metadata matches every
// condition. Requires a store implementing ingest.MetadataProvider (the
// gorm stores do).
Filter index.Filter
// Cursor resumes pagination after a previous SearchPage (empty = first page).
Cursor string
}
SearchOptions holds options for Search calls.
type SearchPage ¶ added in v0.0.3
type SearchPage struct {
Results []*index.SearchResult
NextCursor string
// Grounding is the sufficiency verdict computed by the evidence evaluator
// during this search (nil when grounding is not configured or the evaluator
// failed under fail-open). It is the same verdict CheckGrounding would
// return, exposed here so callers need not re-run the evaluation.
Grounding *retrieval.GroundingResult
}
SearchPage is a page of search results together with the opaque cursor used to fetch the next page (empty when the last page has been reached).
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
amoxtli
command
Command amoxtli is the command line interface to the amoxtli library: per-project document indexing, hybrid search and an MCP server for agents.
|
Command amoxtli is the command line interface to the amoxtli library: per-project document indexing, hybrid search and an MCP server for agents. |
|
Package eval provides an offline retrieval-quality harness: it runs a golden dataset (query → expected relevant sources) through a Retriever and reports the standard ranking metrics — Recall@k, Mean Reciprocal Rank (MRR) and nDCG@k — so that changes to the retrieval stack (fusion, reranking, grounding) can be validated objectively rather than by feel.
|
Package eval provides an offline retrieval-quality harness: it runs a golden dataset (query → expected relevant sources) through a Retriever and reports the standard ranking metrics — Recall@k, Mean Reciprocal Rank (MRR) and nDCG@k — so that changes to the retrieval stack (fusion, reranking, grounding) can be validated objectively rather than by feel. |
|
beir
Package beir loads datasets in the BEIR format (the de-facto information retrieval benchmark) into an amoxtli evaluation corpus + query set.
|
Package beir loads datasets in the BEIR format (the de-facto information retrieval benchmark) into an amoxtli evaluation corpus + query set. |
|
hfqa
Package hfqa loads Hugging Face extractive-QA datasets in the SQuAD JSON format into an amoxtli evaluation corpus + query set.
|
Package hfqa loads Hugging Face extractive-QA datasets in the SQuAD JSON format into an amoxtli evaluation corpus + query set. |
|
example
|
|
|
convert
command
Command convert demonstrates two things at once:
|
Command convert demonstrates two things at once: |
|
postgres
command
Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database.
|
Command postgres demonstrates amoxtli backed entirely by PostgreSQL: the document store and the hybrid index (index/postgres) share a single database. |
|
sourcecode
command
Command sourcecode demonstrates indexing source code alongside documentation and searching across both.
|
Command sourcecode demonstrates indexing source code alongside documentation and searching across both. |
|
sqlite
command
Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index.
|
Command sqlite demonstrates amoxtli backed entirely by local files: a SQLite document store and a Bleve full-text index. |
|
postgres
Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance).
|
Package postgres provides a hybrid index backed by PostgreSQL, combining native full-text search (tsvector + ts_rank) with vector similarity search (pgvector, cosine distance). |
|
internal
|
|
|
build
Package build carries version information injected at build time via -ldflags "-X github.com/bornholm/amoxtli/internal/build.Version=...".
|
Package build carries version information injected at build time via -ldflags "-X github.com/bornholm/amoxtli/internal/build.Version=...". |
|
cli
Package cli implements the amoxtli command line interface: a thin layer over the library that manages a per-project workspace (.amoxtli directory) holding the configuration and the indexed data.
|
Package cli implements the amoxtli command line interface: a thin layer over the library that manages a per-project workspace (.amoxtli directory) holding the configuration and the indexed data. |
|
cli/config
Package config defines the amoxtli workspace configuration file (.amoxtli/config.yaml): parsing, environment variable interpolation and validation.
|
Package config defines the amoxtli workspace configuration file (.amoxtli/config.yaml): parsing, environment variable interpolation and validation. |
|
cli/mcpserver
Package mcpserver exposes an amoxtli workspace over the Model Context Protocol (stdio), so an LLM agent can search the local corpus.
|
Package mcpserver exposes an amoxtli workspace over the Model Context Protocol (stdio), so an LLM agent can search the local corpus. |
|
cli/runtime
Package runtime turns a workspace configuration into a live amoxtli.Codex, owning (and closing) the resources the library constructors require: the document store, the index backends and the LLM client.
|
Package runtime turns a workspace configuration into a live amoxtli.Codex, owning (and closing) the resources the library constructors require: the document store, the index backends and the LLM client. |
|
cli/workspace
Package workspace locates and describes an amoxtli workspace: a project directory holding a .amoxtli/ configuration directory, discovered by walking up the filesystem from a starting point (like git does).
|
Package workspace locates and describes an amoxtli workspace: a project directory holding a .amoxtli/ configuration directory, discovered by walking up the filesystem from a starting point (like git does). |
|
ignore
Package ignore implements .amoxtlignore support: gitignore-style exclusion rules that decide whether a file should be skipped by "amoxtli add".
|
Package ignore implements .amoxtlignore support: gitignore-style exclusion rules that decide whether a file should be skipped by "amoxtli add". |
|
ollamatest
Package ollamatest provides shared helpers for the integration tests that run against an Ollama testcontainer.
|
Package ollamatest provides shared helpers for the integration tests that run against an Ollama testcontainer. |
|
Package llmx provides reusable decorators around github.com/bornholm/genai llm.Client.
|
Package llmx provides reusable decorators around github.com/bornholm/genai llm.Client. |
|
Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition.
|
Package retrieval implements grounding-driven retrieval orchestration ported from Corpus' MothRAG-derived mechanisms: an evidence evaluator (relevance filtering fused with a grounding (γ) verdict), query reformulation for iterative re-retrieval, and query decomposition. |
|
Package telemetry wires amoxtli to OpenTelemetry.
|
Package telemetry wires amoxtli to OpenTelemetry. |