rag

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 20, 2026 License: MIT Imports: 24 Imported by: 0

Documentation

Overview

Package rag implements a fast graph RAG store. Chunks are embedded, quantized with TurboQuant, and indexed in an HNSW graph for sublinear nearest-neighbor search and a BM25 index for lexical matching. Retrieval fuses dense and sparse hits, seeds Personalized PageRank over a chunk similarity graph, blends graph propagation with direct similarity, and optionally diversifies with MMR. The result is context relevant by association as well as by direct match.

Index

Constants

View Source 
const (
	StrategyRecursive = "recursive" // separator hierarchy, the default
	StrategyWord      = "word"      // fixed overlapping word windows
	StrategyMarkdown  = "markdown"  // split on headings, attach breadcrumbs
	StrategySentence  = "sentence"  // pack whole sentences to the budget
)

Chunking strategy names accepted by ChunkConfig.Strategy.

View Source 
const DefaultLexicalWeight = 0.25

DefaultLexicalWeight is the BM25 weight used when RetrieveParams.LexicalWeight is unset and the store indexes BM25. It is deliberately small: a light lexical boost on top of the dense ranking improves keyword and entity matching while staying near-neutral on dense-dominant corpora. See docs/benchmarks.md.

Variables

This section is empty.

Functions

func ExportJSON 

func ExportJSON(r io.Reader, w io.Writer, includeVectors bool) error

ExportJSON reads a gob-encoded .tg snapshot from r and writes an equivalent, indented JSON document to w. The on-disk format is Go gob, which is Go-specific; this is the supported interop path for other languages and tools, producing a plain JSON view of the same data (config, chunks with their document offsets, embeddings, per-document metadata, version history, and the entity graph). Set includeVectors to false to omit the embeddings, which dominate the size, when only the text and structure are needed.

func ShouldAbstain 

func ShouldAbstain(res []Retrieved, minTopSim float32) bool

ShouldAbstain reports whether retrieval is too weak to answer from the corpus. It uses the raw cosine Similarity of the top hit (an objective signal), not the blended Score, so the threshold is comparable across queries. A store with no results, or whose best hit is below minTopSim, should abstain rather than let the model answer from parametric memory.

func ValidBucketName 

func ValidBucketName(name string) bool

ValidBucketName reports whether name is an acceptable bucket identifier. The restriction also prevents path traversal, since names become blob keys.

Types

type Chunk 

type Chunk struct {
	ID    string `json:"id"` // stable identifier, "doc#pos"
	DocID string `json:"doc_id"`
	Pos   int    `json:"pos"` // ordinal within the document
	Text  string `json:"text"`
	// Start and End are the [start,end) rune offsets of this chunk's body within
	// the original document text, giving an exact document-to-chunk mapping that
	// callers use to preview a document with its retrieved chunks highlighted.
	// They are best-effort: both are -1 when a chunk's text cannot be located
	// verbatim in the source (for example a custom Chunker that rewrites text).
	Start int `json:"start"`
	End   int `json:"end"`
	// Kind labels non-text chunks. "" (text) is the default; "image" marks a chunk
	// whose Text is a model-written caption of an image, figure, or table.
	Kind string `json:"kind,omitempty"`
	// ImageRef is the asset id of the source image for an image chunk, served by
	// the host application (for example GET /api/asset/<ref>). Empty for text.
	ImageRef string `json:"image_ref,omitempty"`
}

Chunk is a unit of retrievable text with provenance.

type ChunkConfig 

type ChunkConfig struct {
	// Strategy names the built-in chunker: "recursive" (default), "word",
	// "markdown", or "sentence". See NewChunker and the Strategy* constants.
	Strategy string
	// TargetWords is the desired chunk size in whitespace-delimited tokens.
	TargetWords int
	// OverlapWords is how many tokens consecutive chunks share, preserving
	// context across boundaries.
	OverlapWords int
}

ChunkConfig controls how documents are split.

func DefaultChunkConfig 

func DefaultChunkConfig() ChunkConfig

DefaultChunkConfig returns balanced defaults for prose: the recursive splitter at a modest size, which keeps paragraphs and sentences intact.

type ChunkSpan 

type ChunkSpan struct {
	ID    string `json:"id"`
	Pos   int    `json:"pos"`
	Start int    `json:"start"` // rune offset, -1 if the chunk could not be located
	End   int    `json:"end"`   // rune offset (exclusive)
}

ChunkSpan locates one chunk inside its document for highlighting.

type Chunker 

type Chunker interface {
	// Split returns the document's pieces in order. It must never return empty
	// pieces and must be deterministic for a given input.
	Split(text string) []Piece
}

Chunker splits a document's text into ordered pieces. It is the seam for document segmentation: the built-in strategies implement it with pure string operations, and a caller can supply their own by implementing this one method (set Config.Chunker). Dependencies for richer strategies are injected at construction, so the interface stays minimal.

func NewChunker 

func NewChunker(cfg ChunkConfig) Chunker

NewChunker builds the chunker named by cfg.Strategy, sized by cfg.TargetWords and cfg.OverlapWords. An unknown or empty strategy uses the recursive splitter, the pragmatic default that keeps paragraphs and sentences intact.

type CommunityOptions 

type CommunityOptions struct {
	Workers     int // concurrent summarizers; 0 uses GOMAXPROCS
	MaxPassages int // cap member passages per community in the prompt (0 = 12)
	MinSize     int // skip communities smaller than this (0 = 1, summarize all)
	OnProgress  func(done, total int)
}

CommunityOptions configures BuildCommunitySummaries.

type CommunitySummary 

type CommunitySummary struct {
	Label   int      `json:"label"`
	Size    int      `json:"size"`    // number of chunks in the community
	Summary string   `json:"summary"` // the generated thematic summary
	Chunks  []string `json:"chunks"`  // member chunk ids
	DocIDs  []string `json:"doc_ids"` // distinct source documents
}

CommunitySummary is a natural-language description of one community of chunks, generated once at index time so global, thematic, corpus-wide questions can be answered from the summaries instead of from raw passages. The communities are the existing label-propagation partitions of the chunk similarity graph; the summary is the missing piece that lets turbograph answer "what are the main themes" style questions (the GraphRAG community-report idea), without giving up its cheap default: summaries are built only when asked for.

type Config 

type Config struct {
	Chunk ChunkConfig
	// Chunker, if set, overrides Chunk.Strategy with a caller-supplied splitter
	// (bring your own). It is not persisted; after loading a store you must set it
	// again to ingest further documents with a custom chunker.
	Chunker Chunker

	// Quantization.
	Bits         int
	ResidualDims int
	Seed         uint64

	// Vector index (HNSW).
	HNSW     index.HNSWConfig
	EfSearch int // query-time search width (default 64)

	// Graph construction.
	GraphKNN         int
	MinSimilarity    float32
	SequentialWeight float32

	// Hybrid lexical fusion. BM25 + RRF is on by default because it reliably
	// improves recall on exact and rare terms; set DisableLexical to turn it off.
	DisableLexical bool
	RRFK           int // reciprocal rank fusion constant (default 60)
}

Config parameterizes the store. Zero values select sensible defaults.

type DocInfo 

type DocInfo struct {
	ID     string `json:"id"`
	Chunks int    `json:"chunks"`
	Bytes  int    `json:"bytes"` // total chunk text length
}

DocInfo summarizes one ingested document.

type DocVersion 

type DocVersion struct {
	N       int    `json:"n"`       // 1-based version number, oldest is 1
	Hash    string `json:"hash"`    // short hex content hash
	Time    int64  `json:"time"`    // unix seconds when recorded
	Bytes   int    `json:"bytes"`   // document size at this version
	Chunks  int    `json:"chunks"`  // chunks this version produced
	Current bool   `json:"current"` // whether this is the live version
}

DocVersion is a single version's metadata for listing, without the full text.

type DocView 

type DocView struct {
	ID    string          `json:"id"`
	Text  string          `json:"text"`
	Meta  json.RawMessage `json:"meta,omitempty"`
	Spans []ChunkSpan     `json:"spans"`
}

DocView is a document with everything needed to preview it and highlight the chunks a query retrieved: the original text, the document's metadata, and the span of every chunk within the text.

type Document 

type Document struct {
	ID   string
	Text string
	// Meta is arbitrary user metadata attached to the document. It is stored as
	// canonical JSON, propagated to every chunk of the document, and returned with
	// each retrieved result, so callers can decide how to use it (parse it, filter
	// on it, or feed selected fields to the model). nil means no metadata.
	Meta map[string]any
	// Kind and ImageRef mark an image-derived document: Text is then a caption of
	// the image, Kind is "image", and ImageRef is the asset id of the source image.
	// Both are empty for an ordinary text document.
	Kind     string
	ImageRef string
}

Document is an input document.

type Embedder 

type Embedder interface {
	Embed(ctx context.Context, texts []string) ([][]float32, error)
}

Embedder produces embeddings for a batch of texts, preserving order. These are document embeddings: the source-of-truth vectors that are indexed.

type EntityBuildOptions 

type EntityBuildOptions struct {
	Workers int
	// BatchSize groups this many chunks into a single model call when the extractor
	// implements entity.BatchExtractor, cutting the number of LLM round trips by
	// roughly this factor. 0 or 1 extracts one chunk per call. Large batches are
	// faster but can dilute a small model's accuracy; 4 to 8 is a good range.
	BatchSize  int
	OnProgress func(EntityProgress)
}

EntityBuildOptions configures BuildEntityGraph.

type EntityProgress 

type EntityProgress struct {
	Done      int
	Total     int
	Entities  int
	Relations int
}

EntityProgress reports the state of an entity-graph build.

type Generator 

type Generator interface {
	Generate(ctx context.Context, system, prompt string) (string, error)
}

Generator produces a completion for a system and user prompt. It is the minimal surface the reranker and other LLM-assisted steps need; the Ollama client satisfies it once a model is bound.

type GraphEdge 

type GraphEdge struct {
	Source int     `json:"source"`
	Target int     `json:"target"`
	Weight float32 `json:"weight"`
}

GraphEdge is an undirected similarity edge between two chunk indices.

type GraphNode 

type GraphNode struct {
	Index     int    `json:"index"`
	ID        string `json:"id"`
	DocID     string `json:"doc_id"`
	Community int    `json:"community"`
	Degree    int    `json:"degree"`
	Snippet   string `json:"snippet"`
}

GraphNode is a chunk as seen by a visualization: its identity, the community it belongs to, its degree in the similarity graph, and a short text preview.

type GraphView 

type GraphView struct {
	Nodes []GraphNode `json:"nodes"`
	Edges []GraphEdge `json:"edges"`
}

GraphView is a serializable snapshot of the similarity graph for rendering.

type IngestOptions 

type IngestOptions struct {
	// Workers is how many documents are embedded concurrently. Embedding is the
	// bottleneck, so this is the main parallelism knob. Defaults to GOMAXPROCS.
	Workers int
	// Journal, if set, records durably-ingested documents so an interrupted run
	// resumes without re-embedding completed work.
	Journal *Journal
	// Save, if set, checkpoints the store to durable storage. It is called every
	// CheckpointEvery documents and once at the end, always before the matching
	// journal entries are written so a "done" record always implies a saved store.
	Save func() error
	// CheckpointEvery bounds how much embedding work a crash can lose. 0 disables
	// intermediate checkpoints (only a final save). Ignored if Save is nil.
	CheckpointEvery int
	// OnProgress, if set, is called after each document with a snapshot.
	OnProgress func(Progress)
}

IngestOptions configures a bulk ingestion.

type Journal 

type Journal struct {
	// contains filtered or unexported fields
}

Journal is an append-only record of which documents have been durably ingested. It lets an interrupted ingestion resume without re-embedding work that is already saved: a document is marked done only after the store containing it has been checkpointed to disk, so a "done" entry always implies the document is recoverable. Failed documents are recorded for visibility but are retried on the next run.

func OpenJournal 

func OpenJournal(path string) (*Journal, error)

OpenJournal opens (or creates) a journal at path, replaying existing entries to rebuild the set of completed documents.

func (*Journal) Close 

func (j *Journal) Close() error

Close flushes and closes the journal.

func (*Journal) Done 

func (j *Journal) Done(id string) bool

Done reports whether a document has been durably ingested.

func (*Journal) DoneCount 

func (j *Journal) DoneCount() int

DoneCount returns the number of completed documents.

func (*Journal) MarkDone 

func (j *Journal) MarkDone(ids ...string) error

MarkDone records documents as durably ingested and flushes to disk, so the record survives a crash.

func (*Journal) MarkFailed 

func (j *Journal) MarkFailed(id, errMsg string) error

MarkFailed records that a document failed to ingest. It is informational; the document will be retried on the next run.

type Manager 

type Manager struct {
	// contains filtered or unexported fields
}

Manager owns a set of named, independent stores ("buckets"). Each bucket is a separate corpus with its own quantizer, indexes, similarity graph, and communities, so they can be kept apart for multitenancy or simply to organize different document sets. Buckets are persisted as separate blobs through a storage.Blob, which can be the local filesystem or any S3-compatible service. A nil blob keeps buckets in memory only.

The Manager is safe for concurrent use. Individual stores are themselves concurrency-safe for reads.

func NewManager 

func NewManager(dir string, embedder Embedder, cfg Config) (*Manager, error)

NewManager creates a manager that persists to a local directory, loading any existing buckets. A directory of "" keeps buckets in memory.

func NewManagerBlob 

func NewManagerBlob(blob storage.Blob, embedder Embedder, cfg Config) (*Manager, error)

NewManagerBlob creates a manager persisting through an arbitrary blob store (for example S3), loading any existing buckets.

func (*Manager) Config 

func (m *Manager) Config() Config

Config returns the manager's current new-bucket configuration.

func (*Manager) Create 

func (m *Manager) Create(name string) (*Store, error)

Create makes a new empty bucket, erroring if the name is invalid or taken.

func (*Manager) Delete 

func (m *Manager) Delete(name string) error

Delete removes a bucket and its persisted blob.

func (*Manager) Get 

func (m *Manager) Get(name string) (*Store, bool)

Get returns the store for name, if it exists.

func (*Manager) GetOrCreate 

func (m *Manager) GetOrCreate(name string) (*Store, error)

GetOrCreate returns the bucket, creating it if absent.

func (*Manager) List 

func (m *Manager) List() []string

List returns the bucket names in sorted order.

func (*Manager) Path 

func (m *Manager) Path(name string) string

Path returns the blob key for a bucket (empty for an in-memory manager).

func (*Manager) Put 

func (m *Manager) Put(name string, st *Store)

Put inserts an externally constructed store under name. It is used to wrap a single store as a one-bucket manager.

func (*Manager) Save 

func (m *Manager) Save(name string) error

Save persists one bucket. It is a no-op for an in-memory manager.

func (*Manager) SaveAll 

func (m *Manager) SaveAll() error

SaveAll persists every bucket.

func (*Manager) SetConfig 

func (m *Manager) SetConfig(cfg Config)

SetConfig updates the configuration used when new buckets are created (for example to change the chunking strategy). Existing buckets keep the config they were built with. Safe for concurrent use.

func (*Manager) SetEmbedder 

func (m *Manager) SetEmbedder(e Embedder)

SetEmbedder swaps the embedder used for new buckets. Existing buckets keep theirs, since their stored vectors come from the original embedder; changing the embedding model for a populated bucket would mix incompatible vectors.

type Piece 

type Piece struct {
	Text     string
	Headings []string
}

Piece is one unit produced by a Chunker: the text plus an optional heading breadcrumb (for example ["Title", "Section"]). When headings are present the store prepends them to the chunk before embedding and lexical indexing, a cheap "contextual chunk header" that situates the passage and measurably improves retrieval on structured documents.

type Progress 

type Progress struct {
	Total   int // documents offered, if known (0 means unknown/streaming)
	Done    int // documents durably ingested this run
	Failed  int // documents that errored
	Skipped int // documents already present (resumed)
	Chunks  int // chunks added this run
}

Progress reports the running state of a bulk ingestion.

type QueryEmbedder 

type QueryEmbedder interface {
	EmbedQuery(ctx context.Context, texts []string) ([][]float32, error)
}

QueryEmbedder is an optional extension of Embedder for asymmetric, instruction tuned models that encode a query differently from a document. When the store's embedder implements it, retrieval embeds the query through EmbedQuery instead of Embed; otherwise the same Embed path is used for both, so plain embedders keep working unchanged.

type RetrieveParams 

type RetrieveParams struct {
	TopK  int // number of chunks to return
	SeedK int // dense/sparse hits used to seed PageRank (default 3*TopK)
	// GraphMix is how strongly the personalized-PageRank graph signal is added on
	// top of direct relevance. The score is relevance + GraphMix*pagerank, so the
	// graph can lift an associated chunk (one hop from a strong hit) into the
	// results without demoting a strong direct match. It is off by default (0):
	// graph reranking measurably lowers precision on standard retrieval, so it is
	// opt-in for thematic or associative queries. Negative is clamped to 0.
	GraphMix float32
	// LexicalWeight is how strongly the BM25 score is added to the dense cosine in
	// the direct relevance term (relevance = dense + LexicalWeight*bm25, both
	// normalized to their per-query max). It preserves the dense ranking and lets
	// an exact lexical match lift a chunk, which helps on keyword- and entity-heavy
	// corpora and is near-neutral where dense already dominates. 0 is pure dense; a
	// negative value is treated as 0. Ignored when the store has lexical disabled.
	LexicalWeight float32
	MMRLambda     float32 // MMR relevance/diversity tradeoff; 0 disables diversification
	EntityMix     float32 // weight of the entity-graph signal in [0,1]; 0 ignores it
	// PRF enables pseudo-relevance feedback: an initial dense search of this many
	// chunks is run, their vectors are averaged into the query (Rocchio in
	// embedding space), and the expanded query drives retrieval. It surfaces
	// chunks that share the topic's vocabulary but not the query's exact words,
	// which helps recall on multi-hop and underspecified queries. 0 disables it.
	PRF int
	// PRFWeight is how strongly the feedback centroid is mixed into the query
	// (Rocchio beta). The original query is always kept at full weight so feedback
	// refines rather than replaces it. Defaults to 0.5 when PRF is set.
	PRFWeight float32
	Filter    func(Chunk) bool // optional metadata filter
	PPR       graph.PPRParams
}

RetrieveParams controls a retrieval.

type Retrieved 

type Retrieved struct {
	Chunk      Chunk
	Score      float32         // blended retrieval score
	Similarity float32         // direct cosine similarity to the query (0 if not a seed)
	Meta       json.RawMessage // the source document's metadata, if any
}

Retrieved is a scored chunk.

func Rerank 

func Rerank(ctx context.Context, gen Generator, query string, res []Retrieved, topK int) []Retrieved

Rerank reorders retrieved results with a single pointwise LLM call, then blends the model score with the original fused score so the model refines rather than overrides retrieval. It is fail-open: any error or unparseable output returns the input truncated to topK, so enabling it can never make results worse than the base ranking. Passages are truncated to keep the prompt bounded.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store holds the indexed corpus, its vector and lexical indexes, the similarity graph, and its community structure. It is safe for concurrent retrieval.

func Load 

func Load(embedder Embedder, r io.Reader) (*Store, error)

Load reconstructs a store from r, attaching the given embedder for queries. Indexes are rebuilt from the stored embeddings, so loading is as fast as indexing minus the embedding step (the expensive part is already done).

func New 

func New(embedder Embedder, cfg Config) *Store

New creates an empty store.

func (*Store) AddDocuments 

func (s *Store) AddDocuments(ctx context.Context, docs []Document) error

AddDocuments incrementally indexes documents. Re-adding identical content is a no-op (deduped by content hash). A document whose id already exists but whose content has changed is updated in place: it is re-chunked and only the chunks whose text actually changed are re-embedded, the rest reuse their existing embeddings. New documents are added directly. The graph and communities are then refreshed.

func (*Store) AddEmbedded 

func (s *Store) AddEmbedded(chunks []Chunk, vecs [][]float32) error

AddEmbedded indexes already-embedded chunks without rebuilding the graph. The caller is expected to call Reindex once after a batch of AddEmbedded calls. This is the low-level entry point used by the bulk ingestion engine so that embedding (the slow part) happens off the write lock and graph reconstruction is deferred to the end.

func (*Store) Build 

func (s *Store) Build(ctx context.Context, docs []Document) error

Build indexes the documents from scratch, replacing any previous contents.

func (*Store) BuildCommunitySummaries 

func (s *Store) BuildCommunitySummaries(ctx context.Context, summarize Summarizer, opt CommunityOptions) error

BuildCommunitySummaries generates a summary for every community of the chunk similarity graph using summarize, replacing any previous summaries. It is the global-query counterpart to the entity graph: more expensive than the embedding only default (one model call per community, far fewer than per chunk), and entirely opt-in.

func (*Store) BuildEntityGraph 

func (s *Store) BuildEntityGraph(ctx context.Context, ex entity.Extractor, opt EntityBuildOptions) error

BuildEntityGraph extracts an entity-relationship knowledge graph from every chunk using ex (typically an LLM), replacing any previous entity graph. This is the GraphRAG-style alternative to the chunk-similarity graph: it is more expensive to build but connects passages by shared entities and typed relationships rather than by similarity. Extraction runs in parallel; the accumulation and graph construction happen once at the end.

func (*Store) Chunk 

func (s *Store) Chunk(i int) Chunk

Chunk returns the chunk at ordinal i.

func (*Store) ChunkDocument 

func (s *Store) ChunkDocument(d Document) []Chunk

ChunkDocument splits a document using the store's chunk configuration. It is exposed so an ingestion engine can chunk and embed off the write path.

func (*Store) Communities 

func (s *Store) Communities() *graph.Communities

Communities returns the detected community structure (may be nil before build).

func (*Store) CommunitySummaries 

func (s *Store) CommunitySummaries() []CommunitySummary

CommunitySummaries returns the generated summaries, largest community first, each enriched with its current member chunk and document ids.

func (*Store) Config 

func (s *Store) Config() Config

Config returns the store's configuration (the custom Chunker, if any, is omitted as it does not round-trip).

func (*Store) ContentOwner 

func (s *Store) ContentOwner(h [32]byte) (string, bool)

ContentOwner returns the id of the document with the given content hash, if any.

func (*Store) DeleteDocument 

func (s *Store) DeleteDocument(id string) int

DeleteDocument removes a document and all of its chunks from the store, dropping its metadata and version history, and rebuilds the indexes. It returns the number of chunks removed (0 if the document was not present).

func (*Store) DocCount 

func (s *Store) DocCount() int

DocCount returns the number of distinct documents ingested.

func (*Store) DocMeta 

func (s *Store) DocMeta(id string) json.RawMessage

DocMeta returns the raw JSON metadata attached to a document, or nil if none.

func (*Store) DocVersionText 

func (s *Store) DocVersionText(id string, n int) (string, bool)

DocVersionText returns the stored text of version n (1-based) of a document.

func (*Store) DocVersions 

func (s *Store) DocVersions(id string) []DocVersion

DocVersions returns the version history of a document, oldest first, marking the newest as the current (live) version. It returns nil for an unknown document or one ingested before version tracking existed.

func (*Store) DocumentView 

func (s *Store) DocumentView(id string) (DocView, bool)

DocumentView returns the original text of a document, its metadata, and the span of each of its chunks, so a caller can render the whole document with the retrieved chunks highlighted. It reports false for an unknown document or one whose original text was not retained (ingested before text was tracked).

func (*Store) Documents 

func (s *Store) Documents() []DocInfo

Documents lists the ingested documents in first-seen order, with each one's chunk count and size. It lets a client reconstruct the document list after loading a store from disk, where the in-memory client has no record of what was ingested in a previous session.

func (*Store) Embedder 

func (s *Store) Embedder() Embedder

Embedder returns the embedder the store was created with.

func (*Store) EntityCount 

func (s *Store) EntityCount() int

EntityCount returns the number of entities in the knowledge graph.

func (*Store) EntityGraphView 

func (s *Store) EntityGraphView() GraphView

EntityGraphView exports the entity graph for visualization, reusing the chunk graph view shape: id is the entity name, doc_id carries its type, and snippet carries its description.

func (*Store) GraphView 

func (s *Store) GraphView() GraphView

GraphView exports the current similarity graph. Each undirected edge is emitted once (source < target). It is safe for concurrent use.

func (*Store) HasCommunitySummaries 

func (s *Store) HasCommunitySummaries() bool

HasCommunitySummaries reports whether community summaries have been generated.

func (*Store) HasDoc 

func (s *Store) HasDoc(id string) bool

HasDoc reports whether a document id has been ingested.

func (*Store) HasEntityGraph 

func (s *Store) HasEntityGraph() bool

HasEntityGraph reports whether an entity-relationship graph has been built.

func (*Store) Ingest 

func (s *Store) Ingest(ctx context.Context, docs <-chan Document, total int, opt IngestOptions) (Progress, error)

Ingest indexes a stream of documents with bounded parallelism, error tolerance, resume support, and periodic checkpointing. Embedding runs across Workers goroutines off the write lock; indexing is serialized; the graph is rebuilt once at the end. Cancelling ctx stops the run cleanly after checkpointing what has completed, and returns ctx.Err().

total is the document count if known (for progress display); pass 0 when streaming an unknown number.

func (*Store) Len 

func (s *Store) Len() int

Len returns the number of indexed chunks.

func (*Store) Reindex 

func (s *Store) Reindex()

Reindex discovers similarity edges for any chunks added since the last reindex and rebuilds the graph and communities. It is cheap to call once after a bulk ingestion and idempotent if nothing changed.

func (*Store) RelevantCommunities 

func (s *Store) RelevantCommunities(ctx context.Context, query string, k int) ([]CommunitySummary, error)

RelevantCommunities ranks the community summaries by similarity of their text to the query and returns the top k, the seed set for a global, map-reduce style answer. It embeds the query and the summaries with the store's embedder.

func (*Store) Retrieve 

func (s *Store) Retrieve(ctx context.Context, query string, p RetrieveParams) ([]Retrieved, error)

Retrieve runs hybrid graph retrieval for the query.

func (*Store) Save 

func (s *Store) Save(w io.Writer) error

Save serializes the store to w.

func (*Store) SetDocMeta 

func (s *Store) SetDocMeta(id string, meta map[string]any) error

SetDocMeta replaces the metadata for a document. A nil or empty map clears it. Metadata is independent of content, so this updates it without re-ingesting.

type Summarizer 

type Summarizer func(ctx context.Context, passages []string) (string, error)

Summarizer turns a community's member passages into a short thematic summary, typically by calling a language model. It is supplied by the caller so the rag package stays free of any model dependency.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.