ragkit

package module
v0.1.9 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 19, 2026 License: MIT Imports: 1 Imported by: 0

README

ragkit

Reusable RAG (retrieval-augmented generation) building blocks for Go, originally extracted from the rag-ttc research codebase.

The package tree is contracts-first: rag holds the domain types and narrow component interfaces; everything else is a replaceable implementation.

ragkit/
  rag/                 Document/Chunk/Representation/Hit/Evidence types,
                       Chunker/Embedder/Generator/Searcher/Reranker interfaces,
                       validation, deterministic hit ordering
  rag/chunking         fixed, markdown, and heading-aware chunkers
  rag/representations  raw/breadcrumbs/small-to-big + generated (batched) kinds,
                       PromptSet (prompts are cache identity)
  rag/embedding        embedding fan-out, caching, budget caps, hash fixture
  rag/lexical          in-memory BM25; rag/lexical/bleve persistent index
  rag/vector           exact cosine; rag/vector/sqliteexact persistent index
  rag/indexbundle      immutable content-addressed bundles (atomic publication)
  rag/retrieval        collapse, weighted RRF fusion, hydration, filters
  rag/reranking        term-overlap + cached reranker decorator
  rag/provider/geppetto validated Geppetto embedding/reranking adapters
  rag/generation       generation caching/observation + flow adapters
  rag/answering        retrieval strategies (bm25/vector/rrf/rrf-reranked/
                       multi-query/hyde), context policy, grounded-answer contract
  rag/evaluation       precision/recall@k, MRR, nDCG, target-level validation
  rag/dataset          corpus/evaluation loading with digest validation
  digest, text, vector, execution, flow   domain-neutral infrastructure

Invariants preserved from upstream:

  • Chunk text is verbatim source: ValidateChunk asserts the chunk equals the exact byte slice of its document.
  • Representations are retrieval material; chunks are evidence. Search returns representation hits, Collapse maps them to chunks, Hydrate produces evidence.
  • HitRanksBefore defines a complete ordering (score, document ID, chunk ID, representation ID) for deterministic retrieval results.

Changes relative to rag-ttc:

  • Module path github.com/go-go-golems/ragkit; logcopter areas renamed.
  • Representation prompts moved from package constants to an injectable representations.PromptSet; callers construct and own their domain prompts explicitly.
  • The grounded-answer contract kind is injectable via answering.Service.ContractKind (default ttc-grounded-answer-v1).
  • Dataset tests use self-contained fixtures instead of the TTC corpora.
  • A boundary test forbids geppetto/pinocchio/glazed/cobra/bubbletea in the ragkit core. The explicit rag/provider/... adapter tree is the only place model-framework dependencies may enter; CLI frameworks remain forbidden.
  • rag/provider/geppetto wraps already-configured providers. Applications keep deployment settings, task prefixes, candidate rendering, cache policy, and artifact identity; ragkit validates and projects provider responses.

Extraction is a cache epoch: execution cache keys are semantically compatible with upstream, but no attempt is made to share cache directories with rag-ttc installations.

Opinionated Geppetto adapters

import raggeppetto "github.com/go-go-golems/ragkit/rag/provider/geppetto"

embedder, err := raggeppetto.NewEmbedder(embeddingProvider)
reranker, err := raggeppetto.NewReranker(rerankProvider)

The constructors accept providers configured by the application. They add no endpoints, credentials, model defaults, task prefixes, document formatting, or cache policy. This keeps the dependency direction strict:

application → ragkit/rag/provider/geppetto → Geppetto

Development

make ci-check

License

MIT

Documentation

Overview

Package ragkit provides repository-wide generation metadata for logcopter.

Source Files

View all Source files

Directories

Path Synopsis
Package digest provides deterministic SHA-256 identities for bytes, text, and JSON-serializable values.
 Package digest provides deterministic SHA-256 identities for bytes, text, and JSON-serializable values.
internal
fsutil
Package fsutil provides private durable-file and lexical path primitives shared by experiment infrastructure.
 Package fsutil provides private durable-file and lexical path primitives shared by experiment infrastructure.
jsonutil
Package jsonutil provides strict JSON syntax mechanics shared by internal caches and experiment commands.
 Package jsonutil provides strict JSON syntax mechanics shared by internal caches and experiment commands.
rag
Package rag defines the small, typed building blocks used by RAG experiments.
 Package rag defines the small, typed building blocks used by RAG experiments.
answering
Package answering composes retrieval, context construction, generation, and grounded-answer validation into one observable RAG operation.
 Package answering composes retrieval, context construction, generation, and grounded-answer validation into one observable RAG operation.
chunking
Package chunking provides source-preserving chunk implementations.
 Package chunking provides source-preserving chunk implementations.
content
Package content defines bounded source-content lookup used by serving retrieval indexes.
 Package content defines bounded source-content lookup used by serving retrieval indexes.
content/sqlite
Package sqlite implements the immutable read-only content store used by a serving bundle.
 Package sqlite implements the immutable read-only content store used by a serving bundle.
dataset
Package dataset loads the two supported experiment input shapes into the small, provider-independent RAG domain types.
 Package dataset loads the two supported experiment input shapes into the small, provider-independent RAG domain types.
embedding
Package embedding provides embedding helpers and deterministic local embeddings for reproducible examples.
 Package embedding provides embedding helpers and deterministic local embeddings for reproducible examples.
evaluation
Package evaluation computes transparent per-query retrieval metrics.
 Package evaluation computes transparent per-query retrieval metrics.
flowpolicy
Package flowpolicy contains ragkit-specific policy for Flowkit runs.
 Package flowpolicy contains ragkit-specific policy for Flowkit runs.
generation
Package generation provides small generation helpers and deterministic providers for self-contained experiments.
 Package generation provides small generation helpers and deterministic providers for self-contained experiments.
gochunk
Package gochunk converts committed Go source files into source-preserving RAG chunks and declaration metadata.
 Package gochunk converts committed Go source files into source-preserving RAG chunks and declaration metadata.
indexbundle
Package indexbundle builds and opens immutable, identity-checked persistent retrieval indexes.
 Package indexbundle builds and opens immutable, identity-checked persistent retrieval indexes.
lexical
Package lexical provides an in-memory BM25 correctness baseline.
 Package lexical provides an in-memory BM25 correctness baseline.
lexical/bleve
Package bleve provides a persistent Bleve BM25 index.
 Package bleve provides a persistent Bleve BM25 index.
provider/geppetto
Package geppetto adapts configured Geppetto model-service providers to ragkit's retrieval-domain interfaces.
 Package geppetto adapts configured Geppetto model-service providers to ragkit's retrieval-domain interfaces.
representations
Package representations generates non-raw searchable representations of chunks: summaries and synthetic questions.
 Package representations generates non-raw searchable representations of chunks: summaries and synthetic questions.
reranking
Package reranking provides reranker helpers and deterministic baselines.
 Package reranking provides reranker helpers and deterministic baselines.
retrieval
Package retrieval provides collapse, fusion, and source hydration.
 Package retrieval provides collapse, fusion, and source hydration.
vector
Package vector provides an exact cosine-search correctness oracle.
 Package vector provides an exact cosine-search correctness oracle.
vector/hnswcandidate
Package hnswcandidate provides a deterministic in-memory HNSW candidate index over generic RAG vector records.
 Package hnswcandidate provides a deterministic in-memory HNSW candidate index over generic RAG vector records.
vector/sqliteexact
Package sqliteexact provides persistent exhaustive cosine vector search.
 Package sqliteexact provides persistent exhaustive cosine vector search.
Package text provides small deterministic text-analysis primitives shared by local experiment baselines.
 Package text provides small deterministic text-analysis primitives shared by local experiment baselines.
Package vector provides deterministic numerical operations for float32 embedding vectors.
 Package vector provides deterministic numerical operations for float32 embedding vectors.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.