raggo

README

Raggo - Retrieval Augmented Generation Library

A flexible RAG (Retrieval Augmented Generation) library for Go, designed to make document processing and context-aware AI interactions simple and efficient.

🔍 Smart Document Search • 💬 Context-Aware Responses • 🤖 Intelligent RAG

Quick Start

package main

import (
	"context"
	"fmt"
	"github.com/teilomillet/raggo"
)

func main() {
	// Initialize RAG with default settings
	rag, err := raggo.NewSimpleRAG(raggo.DefaultConfig())
	if err != nil {
		fmt.Printf("Error: %v\n", err)
		return
	}
	defer rag.Close()

	// Add documents from a directory
	err = rag.AddDocuments(context.Background(), "./docs")
	if err != nil {
		fmt.Printf("Error: %v\n", err)
		return
	}

	// Search with natural language
	response, _ := rag.Search(context.Background(), "What are the key features?")
	fmt.Printf("Answer: %s\n", response)
}

Configuration

Raggo provides a flexible configuration system that can be loaded from multiple sources (environment variables, JSON files, or programmatic defaults):

// Load configuration (automatically checks standard paths)
cfg, err := config.LoadConfig()
if err != nil {
    log.Fatal(err)
}

// Or create a custom configuration
cfg := &config.Config{
    Provider:   "milvus",           // Vector store provider
    Model:      "text-embedding-3-small",
    Collection: "my_documents",
    
    // Search settings
    DefaultTopK:     5,      // Number of similar chunks to retrieve
    DefaultMinScore: 0.7,    // Similarity threshold
    
    // Document processing
    DefaultChunkSize:    300,  // Size of text chunks
    DefaultChunkOverlap: 50,   // Overlap between chunks
}

// Create RAG instance with config
rag, err := raggo.NewSimpleRAG(cfg)

Configuration can be saved for reuse:

err := cfg.Save("~/.raggo/config.json")

Environment variables (take precedence over config files):

• RAGGO_PROVIDER: Service provider
• RAGGO_MODEL: Model identifier
• RAGGO_COLLECTION: Collection name
• RAGGO_API_KEY: Default API key

Table of Contents

Part 1: Core Components

1. Quick Start
2. Building Blocks
   • Document Loading
   • Text Parsing
   • Text Chunking
   • Embeddings
   • Vector Storage

Part 2: RAG Implementations

1. Simple RAG
   • Basic Usage
   • Document Q&A
   • Configuration
2. Contextual RAG
   • Advanced Features
   • Context Window
   • Hybrid Search
3. Memory Context
   • Chat Applications
   • Memory Management
   • Context Enhancement
4. Advanced Use Cases
   • Full Processing Pipeline
   • Concurrent Processing
   • Rate Limiting

Part 1: Core Components

Quick Start

Prerequisites

# Set API key
export OPENAI_API_KEY=your-api-key

# Install Raggo
go get github.com/teilomillet/raggo

Building Blocks

Document Loading

loader := raggo.NewLoader(raggo.SetTimeout(1*time.Minute))
doc, err := loader.LoadURL(context.Background(), "https://example.com/doc.pdf")

Text Parsing

parser := raggo.NewParser()
doc, err := parser.Parse("document.pdf")

Text Chunking

chunker := raggo.NewChunker(raggo.ChunkSize(100))
chunks := chunker.Chunk(doc.Content)

Embeddings

embedder := raggo.NewEmbedder(
    raggo.SetProvider("openai"),
    raggo.SetModel("text-embedding-3-small"),
)

Vector Storage

db := raggo.NewVectorDB(raggo.WithMilvus("collection"))

Part 2: RAG Implementations

Simple RAG

Best for straightforward document Q&A:

package main

import (
    "context"
    "log"
    "github.com/teilomillet/raggo"
)

func main() {
    // Initialize SimpleRAG
    rag, err := raggo.NewSimpleRAG(raggo.SimpleRAGConfig{
        Collection: "docs",
        Model:      "text-embedding-3-small",
        ChunkSize:  300,
        TopK:       3,
    })
    if err != nil {
        log.Fatal(err)
    }
    defer rag.Close()

    // Add documents
    err = rag.AddDocuments(context.Background(), "./documents")
    if err != nil {
        log.Fatal(err)
    }

    // Search with different strategies
    basicResponse, _ := rag.Search(context.Background(), "What is the main feature?")
    hybridResponse, _ := rag.SearchHybrid(context.Background(), "How does it work?", 0.7)
    
    log.Printf("Basic Search: %s\n", basicResponse)
    log.Printf("Hybrid Search: %s\n", hybridResponse)
}

Contextual RAG

For complex document understanding and context-aware responses:

package main

import (
	"context"
	"fmt"
	"os"
	"path/filepath"

	"github.com/teilomillet/raggo"
)

func main() {
	// Initialize RAG with default settings
	rag, err := raggo.NewDefaultContextualRAG("basic_contextual_docs")
	if err != nil {
		fmt.Printf("Failed to initialize RAG: %v\n", err)
		os.Exit(1)
	}
	defer rag.Close()

	// Add documents - the system will automatically:
	// - Split documents into semantic chunks
	// - Generate rich context for each chunk
	// - Store embeddings with contextual information
	docsPath := filepath.Join("examples", "docs")
	if err := rag.AddDocuments(context.Background(), docsPath); err != nil {
		fmt.Printf("Failed to add documents: %v\n", err)
		os.Exit(1)
	}

	// Simple search with automatic context enhancement
	query := "What are the key features of the product?"
	response, err := rag.Search(context.Background(), query)
	if err != nil {
		fmt.Printf("Failed to search: %v\n", err)
		os.Exit(1)
	}

	fmt.Printf("\nQuery: %s\nResponse: %s\n", query, response)
}

Advanced Configuration

// Create a custom configuration
config := &raggo.ContextualRAGConfig{
	Collection:   "advanced_contextual_docs",
	Model:        "text-embedding-3-small", // Embedding model
	LLMModel:     "gpt-4o-mini",           // Model for context generation
	ChunkSize:    300,                      // Larger chunks for more context
	ChunkOverlap: 75,                       // 25% overlap for better continuity
	TopK:         5,                        // Number of similar chunks to retrieve
	MinScore:     0.7,                      // Higher threshold for better relevance
}

// Initialize RAG with custom configuration
rag, err := raggo.NewContextualRAG(config)
if err != nil {
	log.Fatalf("Failed to initialize RAG: %v", err)
}
defer rag.Close()

Memory Context

For chat applications and long-term context retention:

package main

import (
    "context"
    "log"
    "github.com/teilomillet/raggo"
    "github.com/teilomillet/gollm"
)

func main() {
    // Initialize Memory Context
    memoryCtx, err := raggo.NewMemoryContext(
        os.Getenv("OPENAI_API_KEY"),
        raggo.MemoryTopK(5),
        raggo.MemoryCollection("chat"),
        raggo.MemoryStoreLastN(100),
        raggo.MemoryMinScore(0.7),
    )
    if err != nil {
        log.Fatal(err)
    }
    defer memoryCtx.Close()

    // Initialize Contextual RAG
    rag, err := raggo.NewContextualRAG(&raggo.ContextualRAGConfig{
        Collection: "docs",
        Model:     "text-embedding-3-small",
    })
    if err != nil {
        log.Fatal(err)
    }
    defer rag.Close()

    // Example chat interaction
    messages := []gollm.MemoryMessage{
        {Role: "user", Content: "How does the authentication system work?"},
    }
    
    // Store conversation
    err = memoryCtx.StoreMemory(context.Background(), messages)
    if err != nil {
        log.Fatal(err)
    }
    
    // Get enhanced response with context
    prompt := &gollm.Prompt{Messages: messages}
    enhanced, _ := memoryCtx.EnhancePrompt(context.Background(), prompt, messages)
    response, _ := rag.Search(context.Background(), enhanced.Messages[0].Content)
    
    log.Printf("Response: %s\n", response)
}

Advanced Use Cases

Full Processing Pipeline

Process large document sets with rate limiting and concurrent processing:

package main

import (
    "context"
    "log"
    "sync"
    "time"
    "github.com/teilomillet/raggo"
    "golang.org/x/time/rate"
)

const (
    GPT_RPM_LIMIT   = 5000    // Requests per minute
    GPT_TPM_LIMIT   = 4000000 // Tokens per minute
    MAX_CONCURRENT  = 10      // Max concurrent goroutines
)

func main() {
    // Initialize components
    parser := raggo.NewParser()
    chunker := raggo.NewChunker(raggo.ChunkSize(500))
    embedder := raggo.NewEmbedder(
        raggo.SetProvider("openai"),
        raggo.SetModel("text-embedding-3-small"),
    )

    // Create rate limiters
    limiter := rate.NewLimiter(rate.Limit(GPT_RPM_LIMIT/60), GPT_RPM_LIMIT)
    
    // Process documents concurrently
    var wg sync.WaitGroup
    semaphore := make(chan struct{}, MAX_CONCURRENT)

    files, _ := filepath.Glob("./documents/*.pdf")
    for _, file := range files {
        wg.Add(1)
        semaphore <- struct{}{} // Acquire semaphore
        
        go func(file string) {
            defer wg.Done()
            defer func() { <-semaphore }() // Release semaphore
            
            // Wait for rate limit
            limiter.Wait(context.Background())
            
            // Process document
            doc, _ := parser.Parse(file)
            chunks := chunker.Chunk(doc.Content)
            embeddings, _ := embedder.CreateEmbeddings(chunks)
            
            log.Printf("Processed %s: %d chunks\n", file, len(chunks))
        }(file)
    }
    
    wg.Wait()
}

Best Practices

Resource Management

• Always use defer Close()
• Monitor memory usage
• Clean up old data

Performance

• Use concurrent processing for large datasets
• Configure appropriate chunk sizes
• Enable hybrid search when needed

Context Management

• Use Memory Context for chat applications
• Configure context window size
• Clean up old memories periodically

Examples

Check /examples for more:

• Basic usage: /examples/simple/
• Context-aware: /examples/contextual/
• Chat applications: /examples/chat/
• Memory usage: /examples/memory_enhancer_example.go
• Full pipeline: /examples/full_process.go
• Benchmarks: /examples/process_embedding_benchmark.go

License

MIT License - see LICENSE file

Documentation

Overview

Package raggo provides a high-level interface for text chunking and token management, designed for use in retrieval-augmented generation (RAG) applications.

Package raggo provides utilities for concurrent document loading and processing.

Package raggo provides advanced Retrieval-Augmented Generation (RAG) capabilities with contextual awareness and memory management.

Package raggo provides advanced Retrieval-Augmented Generation (RAG) capabilities with contextual awareness and memory management.

Package raggo provides a high-level interface for text embedding and retrieval operations in RAG (Retrieval-Augmented Generation) systems. It simplifies the process of converting text into vector embeddings using various providers.

Package raggo provides a high-level interface for document loading and processing in RAG (Retrieval-Augmented Generation) systems. The loader component handles various input sources with support for concurrent operations and configurable behaviors.

Package raggo provides a high-level logging interface for the Raggo framework, built on top of the core rag package logging system. It offers:

• Multiple severity levels (Debug, Info, Warn, Error)
• Structured logging with key-value pairs
• Global log level control
• Consistent logging across the framework

Package raggo provides advanced context-aware retrieval and memory management capabilities for RAG (Retrieval-Augmented Generation) systems.

Package raggo provides a flexible and extensible document parsing system for RAG (Retrieval-Augmented Generation) applications. The system supports multiple file formats and can be extended with custom parsers.

Package raggo implements a comprehensive Retrieval-Augmented Generation (RAG) system that enhances language models with the ability to access and reason over external knowledge. The system seamlessly integrates vector similarity search with natural language processing to provide accurate and contextually relevant responses.

The package offers two main interfaces:

• RAG: A full-featured implementation with extensive configuration options
• SimpleRAG: A streamlined interface for basic use cases

The RAG system works by: 1. Processing documents into semantic chunks 2. Storing document vectors in a configurable database 3. Finding relevant context through similarity search 4. Generating responses that combine retrieved context with queries

Key Features:

• Multiple vector database support (Milvus, in-memory, Chrome)
• Intelligent document chunking and embedding
• Hybrid search capabilities
• Context-aware retrieval
• Configurable LLM integration

Example Usage:

config := raggo.DefaultRAGConfig()
config.APIKey = os.Getenv("OPENAI_API_KEY")

rag, err := raggo.NewRAG(
    raggo.SetProvider("openai"),
    raggo.SetModel("text-embedding-3-small"),
    raggo.WithMilvus("my_documents"),
)
if err != nil {
    log.Fatal(err)
}

// Add documents
err = rag.LoadDocuments(context.Background(), "path/to/docs")

// Query the system
results, err := rag.Query(context.Background(), "your question here")

Package raggo provides a comprehensive registration system for vector database implementations in RAG (Retrieval-Augmented Generation) applications. This package enables dynamic registration and management of vector databases with support for concurrent operations, configurable processing, and extensible architecture.

Package raggo implements a sophisticated document retrieval system that combines vector similarity search with optional reranking strategies. The retriever component serves as the core engine for finding and ranking relevant documents based on semantic similarity and other configurable criteria.

Key features:

• Semantic similarity search using vector embeddings
• Hybrid search combining vector and keyword matching
• Configurable reranking strategies
• Flexible result filtering and scoring
• Extensible callback system for result processing

SimpleRAG provides a minimal, easy-to-use interface for RAG operations. It simplifies the configuration and usage of the RAG system while maintaining core functionality. This implementation is ideal for:

• Quick prototyping
• Simple document retrieval needs
• Learning the RAG system

Example usage:

config := raggo.DefaultConfig()
config.APIKey = "your-api-key"

rag, err := raggo.NewSimpleRAG(config)
if err != nil {
    log.Fatal(err)
}

// Add documents
err = rag.AddDocuments(context.Background(), "path/to/docs")

// Search
response, err := rag.Search(context.Background(), "your query")

Package raggo provides a high-level abstraction over various vector database implementations. This file defines the VectorDB type, which wraps the lower-level rag.VectorDB interface with additional functionality and type safety.

Index

• Constants
• func Debug(msg string, keysAndValues ...interface{})
• func DefaultSentenceSplitter() func(string) []string
• func Error(msg string, keysAndValues ...interface{})
• func GetVectorDB(dbType string, cfg *Config) (rag.VectorDB, error)
• func Info(msg string, keysAndValues ...interface{})
• func ListRegisteredDBs() []string
• func MemoryCollection(collection string) func(*MemoryContextOptions)
• func MemoryHybridSearch(enabled bool) func(*MemoryContextOptions)
• func MemoryMinScore(score float64) func(*MemoryContextOptions)
• func MemoryScoreInclusion(include bool) func(*MemoryContextOptions)
• func MemoryStoreLastN(n int) func(*MemoryContextOptions)
• func MemoryStoreRAGInfo(store bool) func(*MemoryContextOptions)
• func MemoryTopK(k int) func(*MemoryContextOptions)
• func MemoryVectorDB(dbType, address string) func(*MemoryContextOptions)
• func Register(ctx context.Context, source string, opts ...RegisterOption) error
• func RegisterVectorDB(dbType string, factory func(cfg *Config) (rag.VectorDB, error))
• func SetFileTypeDetector(p Parser, detector func(string) string)
• func SetLogLevel(level LogLevel)
• func SmartSentenceSplitter() func(string) []string
• func StoreWithContext(ctx context.Context, source string, opts ContextualStoreOptions) error
• func Warn(msg string, keysAndValues ...interface{})
• func WithParser(p Parser, fileType string, parser Parser)
• type Chunk
• type Chunker
  • func NewChunker(options ...ChunkerOption) (Chunker, error)
• type ChunkerOption
  • func ChunkOverlap(overlap int) ChunkerOption
  • func ChunkSize(size int) ChunkerOption
  • func WithSentenceSplitter(splitter func(string) []string) ChunkerOption
  • func WithTokenCounter(counter TokenCounter) ChunkerOption
• type ConcurrentPDFLoader
  • func NewConcurrentPDFLoader(opts ...LoaderOption) ConcurrentPDFLoader
• type Config
• type ContextualRAG
  • func NewContextualRAG(config *ContextualRAGConfig) (*ContextualRAG, error)
  • func NewDefaultContextualRAG(collection string) (*ContextualRAG, error)
  • func (r *ContextualRAG) AddDocuments(ctx context.Context, source string) error
  • func (r *ContextualRAG) Close() error
  • func (r *ContextualRAG) Search(ctx context.Context, query string) (string, error)
• type ContextualRAGConfig
  • func DefaultContextualConfig() ContextualRAGConfig
• type ContextualStoreOptions
• type Document
• type EmbeddedChunk
• type Embedder
  • func NewEmbedder(opts ...EmbedderOption) (Embedder, error)
• type EmbedderOption
  • func SetEmbedderAPIKey(apiKey string) EmbedderOption
  • func SetEmbedderModel(model string) EmbedderOption
  • func SetEmbedderProvider(provider string) EmbedderOption
  • func SetOption(key string, value interface{}) EmbedderOption
• type EmbeddingService
  • func NewEmbeddingService(embedder Embedder) *EmbeddingService
  • func (s *EmbeddingService) Embed(ctx context.Context, text string) ([]float64, error)
  • func (s *EmbeddingService) EmbedChunks(ctx context.Context, chunks []rag.Chunk) ([]rag.EmbeddedChunk, error)
• type Field
• type Index
• type Loader
  • func NewLoader(opts ...LoaderOption) Loader
• type LoaderOption
  • func SetLoaderTimeout(timeout time.Duration) LoaderOption
  • func SetTempDir(dir string) LoaderOption
  • func WithHTTPClient(client *http.Client) LoaderOption
• type LogLevel
• type Logger
• type MemoryContext
  • func NewMemoryContext(apiKey string, opts ...func(*MemoryContextOptions)) (*MemoryContext, error)
  • func (m *MemoryContext) Close() error
  • func (m *MemoryContext) EnhancePrompt(ctx context.Context, prompt *gollm.Prompt, memory []gollm.MemoryMessage) (*gollm.Prompt, error)
  • func (m *MemoryContext) GetOptions() MemoryContextOptions
  • func (m *MemoryContext) GetRetriever() *Retriever
  • func (m *MemoryContext) StoreLastN(ctx context.Context, memory []gollm.MemoryMessage, n int) error
  • func (m *MemoryContext) StoreMemory(ctx context.Context, messages []gollm.MemoryMessage) error
  • func (m *MemoryContext) UpdateOptions(opts ...func(*MemoryContextOptions))
• type MemoryContextOptions
• type Option
  • func WithAddress(address string) Option
  • func WithDimension(dimension int) Option
  • func WithMaxPoolSize(size int) Option
  • func WithTimeout(timeout time.Duration) Option
  • func WithType(dbType string) Option
• type Parser
  • func NewParser(opts ...ParserOption) Parser
  • func PDFParser() Parser
  • func TextParser() Parser
• type ParserOption
  • func WithLoader(loader *rag.Loader) ParserOption
• type RAG
  • func NewRAG(opts ...RAGOption) (*RAG, error)
  • func (r *RAG) Close() error
  • func (r *RAG) LoadDocuments(ctx context.Context, source string) error
  • func (r *RAG) ProcessWithContext(ctx context.Context, source string, llmModel string) error
  • func (r *RAG) Query(ctx context.Context, query string) ([]RetrieverResult, error)
• type RAGConfig
  • func DefaultRAGConfig() *RAGConfig
• type RAGOption
  • func SetAPIKey(key string) RAGOption
  • func SetChunkOverlap(overlap int) RAGOption
  • func SetChunkSize(size int) RAGOption
  • func SetCollection(name string) RAGOption
  • func SetDBAddress(address string) RAGOption
  • func SetDebug(debug bool) RAGOption
  • func SetMinScore(score float64) RAGOption
  • func SetModel(model string) RAGOption
  • func SetProvider(provider string) RAGOption
  • func SetSearchStrategy(strategy string) RAGOption
  • func SetTimeout(timeout time.Duration) RAGOption
  • func SetTopK(k int) RAGOption
  • func WithMilvus(collection string) RAGOption
  • func WithOpenAI(apiKey string) RAGOption
• type Record
• type RegisterConfig
• type RegisterOption
  • func WithChunking(size, overlap int) RegisterOption
  • func WithCollection(name string, autoCreate bool) RegisterOption
  • func WithConcurrency(max int) RegisterOption
  • func WithEmbedding(provider, model, key string) RegisterOption
  • func WithVectorDB(dbType string, config map[string]string) RegisterOption
• type Retriever
  • func NewRetriever(opts ...RetrieverOption) (*Retriever, error)
  • func (r *Retriever) Close() error
  • func (r *Retriever) GetVectorDB() *VectorDB
  • func (r *Retriever) Retrieve(ctx context.Context, query string) ([]RetrieverResult, error)
• type RetrieverConfig
• type RetrieverOption
  • func WithColumns(columns ...string) RetrieverOption
  • func WithHybrid(enabled bool) RetrieverOption
  • func WithMinScore(score float64) RetrieverOption
  • func WithRetrieveCallbacks(onResult func(SearchResult), onError func(error)) RetrieverOption
  • func WithRetrieveCollection(name string) RetrieverOption
  • func WithRetrieveDB(dbType, address string) RetrieverOption
  • func WithRetrieveDimension(dimension int) RetrieverOption
  • func WithRetrieveEmbedding(provider, model, key string) RetrieverOption
  • func WithTopK(k int) RetrieverOption
• type RetrieverResult
• type Schema
• type SearchResult
• type SimpleRAG
  • func NewSimpleRAG(config SimpleRAGConfig) (*SimpleRAG, error)
  • func (s *SimpleRAG) AddDocuments(ctx context.Context, source string) error
  • func (s *SimpleRAG) Close() error
  • func (s *SimpleRAG) Search(ctx context.Context, query string) (string, error)
• type SimpleRAGConfig
  • func DefaultConfig() SimpleRAGConfig
• type TokenCounter
  • func NewDefaultTokenCounter() TokenCounter
  • func NewTikTokenCounter(encoding string) (TokenCounter, error)
• type Vector
• type VectorDB
  • func NewVectorDB(opts ...Option) (*VectorDB, error)
  • func (vdb *VectorDB) Address() string
  • func (vdb *VectorDB) Close() error
  • func (vdb *VectorDB) Connect(ctx context.Context) error
  • func (vdb *VectorDB) CreateCollection(ctx context.Context, name string, schema Schema) error
  • func (vdb *VectorDB) CreateIndex(ctx context.Context, collectionName, field string, index Index) error
  • func (vdb *VectorDB) Dimension() int
  • func (vdb *VectorDB) DropCollection(ctx context.Context, name string) error
  • func (vdb *VectorDB) Flush(ctx context.Context, collectionName string) error
  • func (vdb *VectorDB) HasCollection(ctx context.Context, name string) (bool, error)
  • func (vdb *VectorDB) HybridSearch(ctx context.Context, collectionName string, vectors map[string]Vector, ...) ([]SearchResult, error)
  • func (vdb *VectorDB) Insert(ctx context.Context, collectionName string, data []Record) error
  • func (vdb *VectorDB) LoadCollection(ctx context.Context, name string) error
  • func (vdb *VectorDB) Search(ctx context.Context, collectionName string, vectors map[string]Vector, ...) ([]SearchResult, error)
  • func (vdb *VectorDB) SetColumnNames(names []string)
  • func (vdb *VectorDB) Type() string

Constants

const (
	// LogLevelOff disables all logging output
	LogLevelOff = rag.LogLevelOff

	// LogLevelError enables only error messages
	// Use for conditions that prevent normal operation
	LogLevelError = rag.LogLevelError

	// LogLevelWarn enables warning and error messages
	// Use for potentially harmful situations
	LogLevelWarn = rag.LogLevelWarn

	// LogLevelInfo enables info, warning, and error messages
	// Use for general operational information
	LogLevelInfo = rag.LogLevelInfo

	// LogLevelDebug enables all message types
	// Use for detailed debugging information
	LogLevelDebug = rag.LogLevelDebug
)

Log levels define the available logging severities. Higher levels include messages from all lower levels.

Functions

func Debug

func Debug(msg string, keysAndValues ...interface{})

Debug logs a message at debug level with optional key-value pairs. Debug messages provide detailed information for troubleshooting.

Example usage:

raggo.Debug("Processing chunk",
    "id", chunkID,
    "size", chunkSize,
    "overlap", overlap)

func DefaultSentenceSplitter

func DefaultSentenceSplitter() func(string) []string

DefaultSentenceSplitter returns the basic sentence splitter function that splits text on common punctuation marks (., !, ?). Suitable for simple English text without complex formatting.

func Error

func Error(msg string, keysAndValues ...interface{})

Error logs a message at error level with optional key-value pairs. Error messages indicate serious problems that affect normal operation.

Example usage:

raggo.Error("Failed to connect to vector store",
    "error", err,
    "retries", retryCount)

func GetVectorDB

func GetVectorDB(dbType string, cfg *Config) (rag.VectorDB, error)

GetVectorDB retrieves a vector database implementation from the registry. It returns an error if the requested implementation is not found or if creation fails.

Example:

db, err := GetVectorDB("milvus", &Config{
    Address: "localhost:19530",
})

func Info

func Info(msg string, keysAndValues ...interface{})

Info logs a message at info level with optional key-value pairs. Info messages provide general operational information.

Example usage:

raggo.Info("Document added successfully",
    "collection", collectionName,
    "documentID", docID)

func ListRegisteredDBs

func ListRegisteredDBs() []string

ListRegisteredDBs returns a list of all registered vector database types. This is useful for discovering available implementations and validating configuration options.

Example:

dbs := ListRegisteredDBs()
for _, db := range dbs {
    fmt.Printf("Supported database: %s\n", db)
}

func MemoryCollection

func MemoryCollection(collection string) func(*MemoryContextOptions)

MemoryCollection specifies the vector database collection for storing memories. Different collections can be used to separate contexts for different use cases.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryCollection("support_chat"),  // Store support chat memories
)

func MemoryHybridSearch

func MemoryHybridSearch(enabled bool) func(*MemoryContextOptions)

MemoryHybridSearch enables or disables hybrid search (vector + keyword matching). Note that some vector databases like ChromemDB don't support hybrid search.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryHybridSearch(false),  // Disable hybrid search for ChromaDB
)

func MemoryMinScore

func MemoryMinScore(score float64) func(*MemoryContextOptions)

MemoryMinScore sets the similarity threshold for memory retrieval. Memories with scores below this threshold are considered irrelevant.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryMinScore(0.7),  // Only retrieve highly similar memories
)

func MemoryScoreInclusion

func MemoryScoreInclusion(include bool) func(*MemoryContextOptions)

MemoryScoreInclusion controls whether similarity scores are included in results. Useful for debugging or implementing custom relevance filtering.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryScoreInclusion(true),  // Include similarity scores
)

func MemoryStoreLastN

func MemoryStoreLastN(n int) func(*MemoryContextOptions)

MemoryStoreLastN limits memory storage to the N most recent interactions. This helps manage memory usage and maintain relevant context windows.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryStoreLastN(100),  // Keep last 100 interactions
)

func MemoryStoreRAGInfo

func MemoryStoreRAGInfo(store bool) func(*MemoryContextOptions)

MemoryStoreRAGInfo enables storage of RAG-enhanced context information. This provides richer context by storing processed and enhanced memories.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryStoreRAGInfo(true),  // Store enhanced context
)

func MemoryTopK

func MemoryTopK(k int) func(*MemoryContextOptions)

MemoryTopK configures the number of relevant memories to retrieve. Higher values provide more comprehensive context but may impact performance.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryTopK(5),  // Retrieve top 5 relevant memories
)

func MemoryVectorDB

func MemoryVectorDB(dbType, address string) func(*MemoryContextOptions)

MemoryVectorDB configures the vector database type and address. This allows choosing between different vector store implementations and their locations.

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryVectorDB("chromem", "./.chromem/chat"),  // Use Chromem store
)

func Register

func Register(ctx context.Context, source string, opts ...RegisterOption) error

Register processes documents from various sources and stores them in a vector database. It handles the entire pipeline from document loading to vector storage:

1. Document loading from files, directories, or URLs
2. Text chunking and preprocessing
3. Embedding generation
4. Vector database storage

The process is highly configurable through RegisterOptions and supports progress monitoring and error handling through callbacks.

Example:

err := Register(ctx, "docs/",
    WithVectorDB("milvus", map[string]string{
        "address": "localhost:19530",
    }),
    WithCollection("technical_docs", true),
    WithChunking(512, 64),
    WithEmbedding("openai", "text-embedding-3-small", os.Getenv("OPENAI_API_KEY")),
    WithConcurrency(4),
)

func RegisterVectorDB

func RegisterVectorDB(dbType string, factory func(cfg *Config) (rag.VectorDB, error))

RegisterVectorDB registers a new vector database implementation. It allows third-party implementations to be integrated into the Raggo ecosystem at runtime.

Example:

RegisterVectorDB("custom_db", func(cfg *Config) (rag.VectorDB, error) {
    return NewCustomDB(cfg)
})

func SetFileTypeDetector

func SetFileTypeDetector(p Parser, detector func(string) string)

SetFileTypeDetector customizes how file types are detected. This allows for sophisticated file type detection beyond simple extension matching.

Example:

SetFileTypeDetector(parser, func(path string) string {
    // Custom logic to determine file type
    if strings.HasSuffix(path, ".md") {
        return "markdown"
    }
    return "unknown"
})

func SetLogLevel

func SetLogLevel(level LogLevel)

SetLogLevel sets the global log level for the raggo package. Messages below this level will not be logged.

Example usage:

// Enable all logging including debug
raggo.SetLogLevel(raggo.LogLevelDebug)

// Only log errors
raggo.SetLogLevel(raggo.LogLevelError)

// Disable all logging
raggo.SetLogLevel(raggo.LogLevelOff)

func SmartSentenceSplitter

func SmartSentenceSplitter() func(string) []string

SmartSentenceSplitter returns an advanced sentence splitter that handles:

• Multiple punctuation marks
• Quoted sentences
• Parenthetical content
• Lists and enumerations

Recommended for complex text with varied formatting and structure.

func StoreWithContext

func StoreWithContext(ctx context.Context, source string, opts ContextualStoreOptions) error

StoreWithContext processes documents and stores them with enhanced contextual information. It uses a combination of:

• Semantic chunking for document segmentation
• Language model enrichment for context generation
• Vector embedding for efficient retrieval
• Batch processing for performance

The function automatically handles:

• Default configuration values
• Resource management
• Error handling and reporting
• Context-aware processing

Example usage:

opts := raggo.ContextualStoreOptions{
    Collection: "my_docs",
    APIKey:     os.Getenv("OPENAI_API_KEY"),
    ChunkSize:  512,
    BatchSize:  100,
}

err := raggo.StoreWithContext(ctx, "path/to/docs", opts)

func Warn

func Warn(msg string, keysAndValues ...interface{})

Warn logs a message at warning level with optional key-value pairs. Warning messages indicate potential issues that don't prevent operation.

Example usage:

raggo.Warn("High memory usage detected",
    "usedMB", memoryUsed,
    "threshold", threshold)

func WithParser

func WithParser(p Parser, fileType string, parser Parser)

WithParser adds a custom parser for a specific file type. This enables the parsing system to handle additional file formats through custom implementations.

Example:

// Add support for markdown files
WithParser(parser, "markdown", &MarkdownParser{})

Types

type Chunk

type Chunk = rag.Chunk

Chunk represents a piece of text with associated metadata including its content, token count, and position within the original document. It tracks:

• The actual text content
• Number of tokens in the chunk
• Starting and ending sentence indices

type Chunker

type Chunker interface {
	// Chunk splits the input text into a slice of Chunks according to the
	// implementation's strategy.
	Chunk(text string) []Chunk
}

Chunker defines the interface for text chunking implementations. Implementations of this interface provide strategies for splitting text into semantically meaningful chunks while preserving context.

func NewChunker

func NewChunker(options ...ChunkerOption) (Chunker, error)

NewChunker creates a new Chunker with the given options. By default, it creates a TextChunker with:

• Chunk size: 200 tokens
• Chunk overlap: 50 tokens
• Default word-based token counter
• Basic sentence splitter

Use the provided option functions to customize these settings.

type ChunkerOption

type ChunkerOption = rag.TextChunkerOption

ChunkerOption is a function type for configuring Chunker instances. It follows the functional options pattern for clean and flexible configuration.

func ChunkOverlap

func ChunkOverlap(overlap int) ChunkerOption

ChunkOverlap sets the number of tokens that should overlap between adjacent chunks. This helps maintain context across chunk boundaries and improves retrieval quality.

func ChunkSize

func ChunkSize(size int) ChunkerOption

ChunkSize sets the target size of each chunk in tokens. This determines how much text will be included in each chunk before starting a new one.

func WithSentenceSplitter

func WithSentenceSplitter(splitter func(string) []string) ChunkerOption

WithSentenceSplitter sets a custom sentence splitter function. The function should take a string and return a slice of strings, where each string is a sentence. This allows for:

• Custom sentence boundary detection
• Language-specific splitting rules
• Special handling of abbreviations or formatting

func WithTokenCounter

func WithTokenCounter(counter TokenCounter) ChunkerOption

WithTokenCounter sets a custom token counter implementation. This allows you to use different tokenization strategies, such as:

• Word-based counting (DefaultTokenCounter)
• Model-specific tokenization (TikTokenCounter)
• Custom tokenization schemes

type ConcurrentPDFLoader

type ConcurrentPDFLoader interface {
	// Embeds the basic Loader interface
	Loader

	// LoadPDFsConcurrent loads a specified number of PDF files concurrently from a source directory.
	// If the source directory contains fewer files than the requested count, it automatically
	// duplicates existing PDFs to reach the desired number.
	//
	// Parameters:
	//   - ctx: Context for cancellation and timeout
	//   - sourceDir: Directory containing source PDF files
	//   - targetDir: Directory where duplicated PDFs will be stored
	//   - count: Desired number of PDF files to load
	//
	// Returns:
	//   - []string: Paths to all successfully loaded files
	//   - error: Any error encountered during the process
	//
	// Example usage:
	//   loader := raggo.NewConcurrentPDFLoader(raggo.SetTimeout(1*time.Minute))
	//   files, err := loader.LoadPDFsConcurrent(ctx, "source", "target", 10)
	LoadPDFsConcurrent(ctx context.Context, sourceDir string, targetDir string, count int) ([]string, error)
}

ConcurrentPDFLoader extends the basic Loader interface with concurrent PDF processing capabilities. It provides efficient handling of multiple PDF files by:

• Loading files in parallel using goroutines
• Managing concurrent file operations safely
• Handling file duplication when needed
• Providing progress tracking and error handling

func NewConcurrentPDFLoader

func NewConcurrentPDFLoader(opts ...LoaderOption) ConcurrentPDFLoader

NewConcurrentPDFLoader creates a new ConcurrentPDFLoader with the given options. It supports all standard loader options plus concurrent processing capabilities.

Options can include:

• SetTimeout: Maximum time for loading operations
• SetTempDir: Directory for temporary files
• SetRetryCount: Number of retries for failed operations

Example:

loader := raggo.NewConcurrentPDFLoader(
  raggo.SetTimeout(1*time.Minute),
  raggo.SetTempDir(os.TempDir()),
)

type Config

type Config struct {
	Type        string        // Database type (e.g., "milvus", "memory")
	Address     string        // Connection address
	MaxPoolSize int           // Maximum number of connections
	Timeout     time.Duration // Operation timeout
	Dimension   int           // Vector dimension
}

Config holds configuration options for VectorDB instances. It provides a clean way to configure database connections without exposing implementation details.

type ContextualRAG

type ContextualRAG struct {
	// contains filtered or unexported fields
}

ContextualRAG provides a high-level interface for context-aware document processing and retrieval. It enhances traditional RAG systems by:

• Maintaining semantic relationships between document chunks
• Generating rich contextual metadata for improved retrieval
• Supporting customizable chunking and embedding strategies
• Providing flexible LLM integration for response generation

Example usage:

// Create with default settings
rag, err := raggo.NewDefaultContextualRAG("my_docs")
if err != nil {
    log.Fatal(err)
}
defer rag.Close()

// Add documents with automatic context generation
err = rag.AddDocuments(context.Background(), "path/to/docs")

// Perform context-aware search
response, err := rag.Search(context.Background(), "How does feature X work?")

func NewContextualRAG

func NewContextualRAG(config *ContextualRAGConfig) (*ContextualRAG, error)

NewContextualRAG creates a new ContextualRAG instance with custom configuration. It provides advanced control over:

• Document processing behavior
• Embedding generation
• Retrieval strategies
• LLM integration

The function will:

• Merge provided config with defaults
• Validate settings
• Initialize vector store
• Set up LLM integration

Example:

config := &raggo.ContextualRAGConfig{
    Collection: "my_docs",
    ChunkSize:  300,
    TopK:       5,
}
rag, err := raggo.NewContextualRAG(config)

func NewDefaultContextualRAG

func NewDefaultContextualRAG(collection string) (*ContextualRAG, error)

NewDefaultContextualRAG creates a new instance with production-ready defaults. It's ideal for quick setup while maintaining good performance.

The function:

• Uses environment variables for API keys
• Sets optimal processing parameters
• Configures reliable retrieval settings

Example:

rag, err := raggo.NewDefaultContextualRAG("my_collection")
if err != nil {
    log.Fatal(err)
}

func (*ContextualRAG) AddDocuments

func (r *ContextualRAG) AddDocuments(ctx context.Context, source string) error

AddDocuments processes and stores documents with contextual awareness. The function:

• Splits documents into semantic chunks
• Generates rich contextual metadata
• Creates and stores embeddings
• Maintains relationships between chunks

Parameters:

• ctx: Context for cancellation and timeouts
• source: Path to document or directory

Example:

err := rag.AddDocuments(ctx, "path/to/docs")

func (*ContextualRAG) Close

func (r *ContextualRAG) Close() error

Close releases all resources held by the ContextualRAG instance. Always defer Close() after creating a new instance.

func (*ContextualRAG) Search

func (r *ContextualRAG) Search(ctx context.Context, query string) (string, error)

Search performs context-aware retrieval and generates a natural language response. The process:

1. Analyzes query for context requirements
2. Retrieves relevant document chunks
3. Synthesizes information with context preservation
4. Generates a coherent response

Parameters:

• ctx: Context for cancellation and timeouts
• query: Natural language query string

Example:

response, err := rag.Search(ctx, "How does the system handle errors?")

type ContextualRAGConfig

type ContextualRAGConfig struct {
	// Collection specifies the vector database collection name
	Collection string

	// APIKey for authentication with the embedding/LLM provider
	APIKey string

	// Model specifies the embedding model for vector generation
	Model string

	// LLMModel specifies the language model for context generation
	LLMModel string

	// LLM allows using a custom LLM instance with specific configuration
	LLM gollm.LLM

	// ChunkSize controls the size of document segments (in tokens)
	// Larger values preserve more context but increase processing time
	ChunkSize int

	// ChunkOverlap determines how much text overlaps between chunks
	// Higher values help maintain context across chunk boundaries
	ChunkOverlap int

	// TopK specifies how many similar chunks to retrieve
	// Adjust based on needed context breadth
	TopK int

	// MinScore sets the minimum similarity threshold for retrieval
	// Higher values increase precision but may reduce recall
	MinScore float64
}

ContextualRAGConfig provides fine-grained control over the RAG system's behavior. It allows customization of:

• Document processing (chunk size, overlap)
• Embedding generation (model selection)
• Retrieval strategy (top-k, similarity threshold)
• LLM integration (custom instance, model selection)

Example configuration:

config := &raggo.ContextualRAGConfig{
    Collection:   "technical_docs",
    Model:        "text-embedding-3-small",
    ChunkSize:    300,     // Larger chunks for more context
    ChunkOverlap: 50,      // Overlap for context continuity
    TopK:         5,       // Number of relevant chunks
    MinScore:     0.7,     // Similarity threshold
}

func DefaultContextualConfig

func DefaultContextualConfig() ContextualRAGConfig

DefaultContextualConfig returns a balanced configuration suitable for most use cases. It provides:

• Reasonable chunk sizes for context preservation
• Modern embedding model selection
• Conservative similarity thresholds
• Efficient batch processing settings

type ContextualStoreOptions

type ContextualStoreOptions struct {
	// Collection specifies the vector database collection name
	Collection string

	// APIKey is the authentication key for the embedding provider
	APIKey string

	// ChunkSize determines the size of text chunks in tokens
	// Larger chunks preserve more context but use more memory
	ChunkSize int

	// ChunkOverlap controls how much text overlaps between chunks
	// More overlap helps maintain context across chunk boundaries
	ChunkOverlap int

	// BatchSize sets how many documents to process simultaneously
	// Higher values increase throughput but use more memory
	BatchSize int

	// ModelName specifies which language model to use for context generation
	// This model enriches chunks with additional contextual information
	ModelName string
}

ContextualStoreOptions configures how documents are processed and stored with contextual information. It provides settings for:

• Vector database collection management
• Document chunking and processing
• Embedding model configuration
• Batch processing controls

This configuration is designed to optimize the balance between processing efficiency and context preservation.

type Document

type Document = rag.Document

Document represents a parsed document with its content and metadata. The structure includes:

• Content: The extracted text from the document
• Metadata: Additional information about the document

Example:

doc := Document{
    Content: "Extracted text content...",
    Metadata: map[string]string{
        "file_type": "pdf",
        "file_path": "/path/to/doc.pdf",
    },
}

type EmbeddedChunk

type EmbeddedChunk = rag.EmbeddedChunk

EmbeddedChunk represents a chunk of text with its embeddings and metadata. It serves as the core data structure for storing and retrieving embedded content in the RAG system.

Structure:

• Text: The original text content that was embedded
• Embeddings: Vector representations from different models/providers
• Metadata: Additional context and information about the chunk

Example:

chunk := EmbeddedChunk{
    Text: "Sample text content",
    Embeddings: map[string][]float64{
        "default": []float64{0.1, 0.2, 0.3},
    },
    Metadata: map[string]interface{}{
        "source": "document1.txt",
        "timestamp": time.Now(),
    },
}

type Embedder

type Embedder = providers.Embedder

Embedder interface defines the contract for embedding implementations. This allows for different embedding providers to be used interchangeably.

func NewEmbedder

func NewEmbedder(opts ...EmbedderOption) (Embedder, error)

NewEmbedder creates a new Embedder instance based on the provided options. It handles provider selection and configuration, returning a ready-to-use embedding interface.

Returns an error if:

• No provider is specified
• The provider is not supported
• Configuration is invalid
• Authentication fails

Example:

embedder, err := NewEmbedder(
    SetEmbedderProvider("openai"),
    SetEmbedderModel("text-embedding-ada-002"),
    SetEmbedderAPIKey(os.Getenv("OPENAI_API_KEY")),
)
if err != nil {
    log.Fatal(err)
}

type EmbedderOption

type EmbedderOption = rag.EmbedderOption

EmbedderOption is a function type for configuring the Embedder. It follows the functional options pattern to provide a clean and flexible configuration API.

Common options include:

• SetEmbedderProvider: Choose the embedding service provider
• SetEmbedderModel: Select the specific embedding model
• SetEmbedderAPIKey: Configure authentication
• SetOption: Set custom provider-specific options

func SetEmbedderAPIKey

func SetEmbedderAPIKey(apiKey string) EmbedderOption

SetEmbedderAPIKey sets the authentication key for the embedding service. This is required for most cloud-based embedding providers.

Security Note: Store API keys securely and never commit them to version control. Consider using environment variables or secure key management systems.

Example:

embedder, err := NewEmbedder(
    SetEmbedderProvider("openai"),
    SetEmbedderAPIKey(os.Getenv("OPENAI_API_KEY")),
)

func SetEmbedderModel

func SetEmbedderModel(model string) EmbedderOption

SetEmbedderModel sets the specific model to use for embedding. Available models depend on the chosen provider:

• OpenAI: "text-embedding-ada-002" (recommended)
• Cohere: "embed-multilingual-v2.0"
• Local: Depends on configured models

Example:

embedder, err := NewEmbedder(
    SetEmbedderProvider("openai"),
    SetEmbedderModel("text-embedding-ada-002"),
)

func SetEmbedderProvider

func SetEmbedderProvider(provider string) EmbedderOption

SetEmbedderProvider sets the provider for the Embedder. Supported providers include:

• "openai": OpenAI's text-embedding-ada-002 and other models
• "cohere": Cohere's embedding models
• "local": Local embedding models (if configured)

Example:

embedder, err := NewEmbedder(
    SetEmbedderProvider("openai"),
    SetEmbedderModel("text-embedding-ada-002"),
)

func SetOption

func SetOption(key string, value interface{}) EmbedderOption

SetOption sets a custom option for the Embedder. This allows for provider-specific configuration that isn't covered by the standard options.

Example:

embedder, err := NewEmbedder(
    SetEmbedderProvider("openai"),
    SetOption("timeout", 30*time.Second),
    SetOption("max_retries", 3),
)

type EmbeddingService

type EmbeddingService struct {
	// contains filtered or unexported fields
}

EmbeddingService handles the embedding process for text content. It supports multiple embedders for different fields or purposes, allowing for flexible embedding strategies.

func NewEmbeddingService

func NewEmbeddingService(embedder Embedder) *EmbeddingService

NewEmbeddingService creates a new embedding service with the specified embedder as the default embedding provider.

Example:

embedder, _ := NewEmbedder(SetEmbedderProvider("openai"))
service := NewEmbeddingService(embedder)

func (*EmbeddingService) Embed

func (s *EmbeddingService) Embed(ctx context.Context, text string) ([]float64, error)

Embed generates embeddings for a single text string using the default embedder. This is a convenience method for simple embedding operations.

Example:

text := "Sample text to embed"
embedding, err := service.Embed(ctx, text)
if err != nil {
    log.Fatal(err)
}

func (*EmbeddingService) EmbedChunks

func (s *EmbeddingService) EmbedChunks(ctx context.Context, chunks []rag.Chunk) ([]rag.EmbeddedChunk, error)

EmbedChunks processes a slice of text chunks and generates embeddings for each one. It supports multiple embedding fields per chunk, using different embedders for each field if configured.

The function:

1. Processes each chunk through configured embedders
2. Combines embeddings from all fields
3. Preserves chunk metadata
4. Handles errors for individual chunks

Example:

chunks := []rag.Chunk{
    {Text: "First chunk", TokenSize: 10},
    {Text: "Second chunk", TokenSize: 12},
}
embedded, err := service.EmbedChunks(ctx, chunks)

type Field

type Field = rag.Field

type Index

type Index = rag.Index

type Loader

type Loader interface {
	// LoadURL downloads and processes a document from the given URL.
	// The function handles:
	//   - HTTP/HTTPS downloads
	//   - Timeout management
	//   - Temporary file storage
	//
	// Returns the processed content and any errors encountered.
	LoadURL(ctx context.Context, url string) (string, error)

	// LoadFile processes a local file at the given path.
	// The function:
	//   - Verifies file existence
	//   - Handles file reading
	//   - Manages temporary storage
	//
	// Returns the processed content and any errors encountered.
	LoadFile(ctx context.Context, path string) (string, error)

	// LoadDir recursively processes all files in a directory.
	// The function:
	//   - Walks the directory tree
	//   - Processes each file
	//   - Handles errors gracefully
	//
	// Returns paths to all processed files and any errors encountered.
	LoadDir(ctx context.Context, dir string) ([]string, error)
}

Loader represents the main interface for loading documents from various sources. It provides a unified API for handling:

• URLs: Download and process remote documents
• Files: Load and process local files
• Directories: Recursively process directory contents

The interface is designed to be thread-safe and supports concurrent operations with configurable timeouts and error handling.

func NewLoader

func NewLoader(opts ...LoaderOption) Loader

NewLoader creates a new Loader with the specified options. It initializes a loader with sensible defaults and applies any provided configuration options.

Default settings:

• Standard HTTP client
• 30-second timeout
• System temporary directory

Example:

loader := NewLoader(
    WithHTTPClient(customClient),
    SetLoaderTimeout(time.Minute),
    SetTempDir("/custom/temp"),
)

type LoaderOption

type LoaderOption = rag.LoaderOption

LoaderOption is a functional option for configuring a Loader. It follows the functional options pattern to provide a clean and extensible configuration API.

Common options include:

• WithHTTPClient: Custom HTTP client configuration
• SetLoaderTimeout: Operation timeout settings
• SetTempDir: Temporary storage location

func SetLoaderTimeout

func SetLoaderTimeout(timeout time.Duration) LoaderOption

SetLoaderTimeout sets a custom timeout for all loader operations. The timeout applies to:

• URL downloads
• File operations
• Directory traversal

Example:

// Set a 2-minute timeout for all operations
loader := NewLoader(SetLoaderTimeout(2 * time.Minute))

func SetTempDir

func SetTempDir(dir string) LoaderOption

SetTempDir sets the temporary directory for file operations. This directory is used for:

• Storing downloaded files
• Creating temporary copies
• Processing large documents

Example:

// Use a custom temporary directory
loader := NewLoader(SetTempDir("/path/to/temp"))

func WithHTTPClient

func WithHTTPClient(client *http.Client) LoaderOption

WithHTTPClient sets a custom HTTP client for the Loader. This enables customization of:

• Transport settings
• Proxy configuration
• Authentication mechanisms
• Connection pooling

Example:

client := &http.Client{
    Timeout: 60 * time.Second,
    Transport: &http.Transport{
        MaxIdleConns: 10,
        IdleConnTimeout: 30 * time.Second,
    },
}
loader := NewLoader(WithHTTPClient(client))

type LogLevel

type LogLevel = rag.LogLevel

LogLevel represents the severity of a log message. It is used to control which messages are output and to indicate the importance of logged information.

Available levels (from least to most severe):

• LogLevelDebug: Detailed information for debugging
• LogLevelInfo: General operational messages
• LogLevelWarn: Warning conditions
• LogLevelError: Error conditions
• LogLevelOff: Disable all logging

type Logger

type Logger = rag.Logger

Logger interface defines the logging operations available. It supports structured logging with key-value pairs for better log aggregation and analysis.

Example usage:

logger.Debug("Processing document",
    "filename", "example.pdf",
    "size", 1024,
    "chunks", 5)

type MemoryContext

type MemoryContext struct {
	// contains filtered or unexported fields
}

MemoryContext provides intelligent context management for RAG systems by:

• Storing and retrieving relevant past interactions
• Enriching queries with historical context
• Managing memory lifecycle and relevance

It integrates with vector databases for efficient similarity search and supports configurable memory management strategies.

func NewMemoryContext

func NewMemoryContext(apiKey string, opts ...func(*MemoryContextOptions)) (*MemoryContext, error)

NewMemoryContext creates a new memory context manager with the specified options. It initializes the underlying vector store and configures memory management.

The function supports multiple configuration options through functional parameters:

• TopK: Number of relevant memories to retrieve
• MinScore: Similarity threshold for relevance
• Collection: Vector store collection name
• StoreLastN: Recent memory retention limit
• StoreRAGInfo: Enhanced context storage

Example:

ctx, err := raggo.NewMemoryContext(apiKey,
    raggo.MemoryTopK(5),
    raggo.MemoryMinScore(0.7),
    raggo.MemoryCollection("chat_memory"),
    raggo.MemoryStoreLastN(100),
    raggo.MemoryStoreRAGInfo(true),
)
if err != nil {
    log.Fatal(err)
}
defer ctx.Close()

func (*MemoryContext) Close

func (m *MemoryContext) Close() error

Close releases resources held by the memory context. Always defer Close() after creating a new memory context.

func (*MemoryContext) EnhancePrompt

func (m *MemoryContext) EnhancePrompt(ctx context.Context, prompt *gollm.Prompt, memory []gollm.MemoryMessage) (*gollm.Prompt, error)

EnhancePrompt enriches a prompt with relevant context from memory. It retrieves and integrates past interactions to provide better context.

Example:

enhanced, err := ctx.EnhancePrompt(context.Background(), prompt, messages)

func (*MemoryContext) GetOptions

func (m *MemoryContext) GetOptions() MemoryContextOptions

GetOptions returns the current context options configuration. Useful for inspecting or copying the current settings.

func (*MemoryContext) GetRetriever

func (m *MemoryContext) GetRetriever() *Retriever

GetRetriever returns the underlying retriever for advanced configuration. This provides access to low-level retrieval settings and operations.

func (*MemoryContext) StoreLastN

func (m *MemoryContext) StoreLastN(ctx context.Context, memory []gollm.MemoryMessage, n int) error

StoreLastN stores only the most recent N messages from the memory. This helps maintain a sliding window of relevant context.

Example:

err := ctx.StoreLastN(context.Background(), messages, 10)  // Keep last 10 messages

func (*MemoryContext) StoreMemory

func (m *MemoryContext) StoreMemory(ctx context.Context, messages []gollm.MemoryMessage) error

StoreMemory stores and indexes conversation messages in the memory context using ChromeDB. It processes the messages by: 1. Converting messages into a formatted text string 2. Generating embeddings using the configured embedder 3. Creating a ChromeDB document with Text, Embedding, and Metadata fields 4. Storing the document in the specified collection

The stored document structure:

• Text: Formatted conversation text
• Embedding: Vector representation of the text
• Metadata:
• timestamp: Time of storage
• collection: Collection name
• type: Document type ("memory")
• roles: Comma-separated list of conversation roles

Example:

err := ctx.StoreMemory(context.Background(), []gollm.MemoryMessage{
    {Role: "user", Content: "How does feature X work?"},
    {Role: "assistant", Content: "Feature X works by..."},
})

func (*MemoryContext) UpdateOptions

func (m *MemoryContext) UpdateOptions(opts ...func(*MemoryContextOptions))

UpdateOptions allows updating context options at runtime. This enables dynamic reconfiguration of memory management behavior.

Example:

ctx.UpdateOptions(
    raggo.MemoryTopK(10),      // Increase context breadth
    raggo.MemoryMinScore(0.8), // Raise relevance threshold
)

type MemoryContextOptions

type MemoryContextOptions struct {
	// TopK determines how many relevant memories to retrieve for context
	// Higher values provide more context but may introduce noise
	TopK int

	// MinScore sets the minimum similarity threshold for memory retrieval
	// Higher values ensure more relevant but fewer memories
	MinScore float64

	// Collection specifies the vector database collection for storing memories
	Collection string

	// DBType specifies the vector database type (e.g., "chromem", "milvus")
	DBType string

	// DBAddress specifies the vector database address (e.g., "./.chromem/name" or "localhost:19530")
	DBAddress string

	// UseHybrid enables hybrid search (vector + keyword matching)
	// Note: Some vector databases like ChromaDB don't support hybrid search
	UseHybrid bool

	// IncludeScore determines whether to include relevance scores in results
	IncludeScore bool

	// StoreLastN limits memory storage to the N most recent interactions
	// Use 0 for unlimited storage
	StoreLastN int

	// StoreRAGInfo enables storage of RAG-enhanced context information
	// This provides richer context by storing processed and enhanced memories
	StoreRAGInfo bool
}

MemoryContextOptions configures the behavior of the contextual memory system. It provides fine-grained control over memory storage, retrieval, and relevance assessment through a set of configurable parameters.

type Option

type Option func(*Config)

Option is a function type for configuring VectorDB instances. It follows the functional options pattern for clean and flexible configuration.

func WithAddress

func WithAddress(address string) Option

WithAddress sets the database connection address. Examples: - Milvus: "localhost:19530" - Memory: "" (no address needed) - ChromeM: "./data/vectors.db"

func WithDimension

func WithDimension(dimension int) Option

WithDimension sets the dimension of vectors to be stored. This must match the dimension of your embedding model: - text-embedding-3-small: 1536 - text-embedding-ada-002: 1536 - Cohere embed-multilingual-v3.0: 1024

func WithMaxPoolSize

func WithMaxPoolSize(size int) Option

WithMaxPoolSize sets the maximum number of database connections. This is particularly relevant for Milvus and other client-server databases.

func WithTimeout

func WithTimeout(timeout time.Duration) Option

WithTimeout sets the operation timeout duration. This affects all database operations including: - Connection attempts - Search operations - Insert operations

func WithType

func WithType(dbType string) Option

WithType sets the database type. Supported types: - "milvus": Production-grade vector database - "memory": In-memory database for testing - "chromem": Chrome-based persistent storage

type Parser

type Parser interface {
	// Parse processes a file and returns its content and metadata.
	// Returns an error if the parsing operation fails.
	Parse(filePath string) (Document, error)
}

Parser defines the interface for document parsing implementations. Any type implementing this interface can be registered to handle specific file types. The interface is designed to be simple yet powerful enough to support various parsing strategies.

Implementations must handle:

• File access and reading
• Content extraction
• Metadata collection
• Error handling

Example:

// Create a new parser with default settings
parser := NewParser()

// Parse a document
doc, err := parser.Parse("document.pdf")
if err != nil {
    log.Fatal(err)
}
fmt.Printf("Content: %s\n", doc.Content)

func NewParser

func NewParser(opts ...ParserOption) Parser

NewParser creates a new Parser with the given options. It initializes a parserWrapper with default settings and applies any provided configuration options. The resulting parser implements both document parsing and loading capabilities.

Default configuration includes:

• Standard rag.Loader with default timeout and temp directory
• Default parser manager for handling various file types
• Built-in support for common file formats

Example:

// Create parser with default settings
parser := NewParser()

// Create parser with custom loader
parser := NewParser(
    WithLoader(customLoader),
)

// Create parser with multiple options
parser := NewParser(
    WithLoader(customLoader),
    WithCustomOption(...),
)

func PDFParser

func PDFParser() Parser

PDFParser returns a new parser for PDF documents. The PDF parser:

• Extracts text content from all pages
• Maintains text order
• Handles complex PDF structures
• Provides document metadata

Example:

parser := PDFParser()
doc, err := parser.Parse("document.pdf")

func TextParser

func TextParser() Parser

TextParser returns a new parser for plain text files. The text parser:

• Reads the entire file content
• Preserves text formatting
• Handles various encodings
• Provides basic metadata

Example:

parser := TextParser()
doc, err := parser.Parse("document.txt")

type ParserOption

type ParserOption func(*parserWrapper)

ParserOption defines functional options for configuring a Parser. This follows the functional options pattern, allowing flexible and extensible configuration of the parser without breaking existing code.

Custom options can be created by implementing this function type and modifying the parserWrapper fields as needed.

Example creating a custom option:

func WithCustomTimeout(timeout time.Duration) ParserOption {
    return func(pw *parserWrapper) {
        pw.loader = rag.NewLoader(rag.WithTimeout(timeout))
    }
}

func WithLoader

func WithLoader(loader *rag.Loader) ParserOption

WithLoader injects a custom loader into the parser. This option allows you to provide a pre-configured rag.Loader instance with custom settings for timeout, temporary directory, HTTP client, or other loader-specific configurations.

Example:

customLoader := rag.NewLoader(
    rag.WithTimeout(time.Minute),
    rag.WithTempDir("/custom/temp"),
)

parser := NewParser(
    WithLoader(customLoader),
)

type RAG

type RAG struct {
	// contains filtered or unexported fields
}

RAG provides a comprehensive interface for document processing and retrieval. It coordinates the interaction between multiple components: - Vector database for efficient similarity search - Embedding service for semantic vector generation - Document processor for text chunking and enrichment - Language model for context-aware response generation

The system is designed to be: - Thread-safe for concurrent operations - Memory-efficient when processing large documents - Extensible through custom implementations - Configurable for different use cases

func NewRAG

func NewRAG(opts ...RAGOption) (*RAG, error)

NewRAG creates a new RAG instance. It takes a variable number of RAGOption functions to configure the system.

func (*RAG) Close

func (r *RAG) Close() error

Close releases all resources held by the RAG system, including database connections and embedding service clients.

It should be called when the RAG system is no longer needed.

Example:

defer rag.Close()

func (*RAG) LoadDocuments

func (r *RAG) LoadDocuments(ctx context.Context, source string) error

LoadDocuments processes and stores documents in the vector database. It handles various document formats and automatically chunks text based on the configured chunk size and overlap.

The source parameter can be a file path or directory. When a directory is provided, all supported documents within it are processed recursively.

Example:

err := rag.LoadDocuments(ctx, "path/to/docs")

func (*RAG) ProcessWithContext

func (r *RAG) ProcessWithContext(ctx context.Context, source string, llmModel string) error

ProcessWithContext processes and stores documents with additional contextual information. It takes a context, source path, and an optional LLM model as input.

func (*RAG) Query

func (r *RAG) Query(ctx context.Context, query string) ([]RetrieverResult, error)

Query performs a retrieval operation using the configured search strategy. It returns a slice of RetrieverResult containing relevant document chunks and their similarity scores.

The query parameter should be a natural language question or statement. The system will convert it to a vector and find similar documents.

Example:

results, err := rag.Query(ctx, "How does feature X work?")

type RAGConfig

type RAGConfig struct {
	// Database settings control how documents are stored and indexed
	DBType      string // Vector database type (e.g., "milvus", "memory")
	DBAddress   string // Database connection address
	Collection  string // Name of the vector collection
	AutoCreate  bool   // Automatically create collection if it doesn't exist
	IndexType   string // Type of vector index (e.g., "HNSW", "IVF")
	IndexMetric string // Distance metric for similarity (e.g., "L2", "IP")

	// Processing settings determine how documents are handled
	ChunkSize    int // Size of text chunks in tokens
	ChunkOverlap int // Overlap between consecutive chunks
	BatchSize    int // Number of documents to process in parallel

	// Embedding settings configure vector generation
	Provider string // Embedding provider (e.g., "openai", "cohere")
	Model    string // Embedding model name
	LLMModel string // Language model for text generation
	APIKey   string // API key for the provider

	// Search settings control retrieval behavior
	TopK      int     // Number of results to retrieve
	MinScore  float64 // Minimum similarity score threshold
	UseHybrid bool    // Whether to use hybrid search

	// System settings affect operational behavior
	Timeout time.Duration // Operation timeout
	TempDir string        // Directory for temporary files
	Debug   bool          // Enable debug logging

	// Search parameters for fine-tuning
	SearchParams map[string]interface{} // Provider-specific search parameters
}

RAGConfig holds the complete configuration for a RAG system. It provides fine-grained control over all aspects of the system's operation, from database settings to search parameters. The configuration is designed to be flexible enough to accommodate various use cases while maintaining sensible defaults.

func DefaultRAGConfig

func DefaultRAGConfig() *RAGConfig

DefaultRAGConfig returns a default RAG configuration. It provides a reasonable set of default values for most use cases.

type RAGOption

type RAGOption func(*RAGConfig)

RAGOption is a function that modifies RAGConfig. It follows the functional options pattern for clean and flexible configuration.

func SetAPIKey

func SetAPIKey(key string) RAGOption

SetAPIKey configures the API key for the chosen provider. This key should have appropriate permissions for embedding and LLM operations.

Example:

rag, err := raggo.NewRAG(
    raggo.SetAPIKey(os.Getenv("OPENAI_API_KEY")),
)

func SetChunkOverlap

func SetChunkOverlap(overlap int) RAGOption

SetChunkOverlap specifies the overlap between consecutive chunks in tokens. Overlap helps maintain context across chunk boundaries.

Example:

rag, err := raggo.NewRAG(
    raggo.SetChunkOverlap(50),
)

func SetChunkSize

func SetChunkSize(size int) RAGOption

SetChunkSize configures the size of text chunks in tokens. Larger chunks provide more context but may reduce retrieval precision.

Example:

rag, err := raggo.NewRAG(
    raggo.SetChunkSize(512),
)

func SetCollection

func SetCollection(name string) RAGOption

SetCollection specifies the name of the vector collection to use. This collection will store document embeddings and metadata.

Example:

rag, err := raggo.NewRAG(
    raggo.SetCollection("my_documents"),
)

func SetDBAddress

func SetDBAddress(address string) RAGOption

SetDBAddress configures the connection address for the vector database. Format depends on the database type (e.g., "localhost:19530" for Milvus).

Example:

rag, err := raggo.NewRAG(
    raggo.SetDBAddress("localhost:19530"),
)

func SetDebug

func SetDebug(debug bool) RAGOption

SetDebug enables or disables debug logging. When enabled, the system will output detailed operation information.

Example:

rag, err := raggo.NewRAG(
    raggo.SetDebug(true),
)

func SetMinScore

func SetMinScore(score float64) RAGOption

SetMinScore sets the minimum similarity score threshold for retrieval. Documents with scores below this threshold are filtered out.

Example:

rag, err := raggo.NewRAG(
    raggo.SetMinScore(0.7),
)

func SetModel

func SetModel(model string) RAGOption

SetModel specifies the embedding model to use for vector generation. The model should be compatible with the chosen provider.

Example:

rag, err := raggo.NewRAG(
    raggo.SetModel("text-embedding-3-small"),
)

func SetProvider

func SetProvider(provider string) RAGOption

Common options SetProvider sets the embedding provider for the RAG system. Supported providers include "openai", "cohere", and others depending on implementation.

Example:

rag, err := raggo.NewRAG(
    raggo.SetProvider("openai"),
)

func SetSearchStrategy

func SetSearchStrategy(strategy string) RAGOption

SetSearchStrategy configures the search approach for document retrieval. Supported strategies include "simple" for pure vector search and "hybrid" for combined vector and keyword search.

Example:

rag, err := raggo.NewRAG(
    raggo.SetSearchStrategy("hybrid"),
)

func SetTimeout

func SetTimeout(timeout time.Duration) RAGOption

SetTimeout configures the maximum duration for operations. This affects database operations, embedding generation, and LLM calls.

Example:

rag, err := raggo.NewRAG(
    raggo.SetTimeout(30 * time.Second),
)

func SetTopK

func SetTopK(k int) RAGOption

SetTopK configures the number of similar documents to retrieve. Higher values provide more context but may introduce noise.

Example:

rag, err := raggo.NewRAG(
    raggo.SetTopK(5),
)

func WithMilvus

func WithMilvus(collection string) RAGOption

WithMilvus is a convenience function that configures the RAG system to use Milvus as the vector database with the specified collection.

Example:

rag, err := raggo.NewRAG(
    raggo.WithMilvus("my_documents"),
)

func WithOpenAI

func WithOpenAI(apiKey string) RAGOption

WithOpenAI is a convenience function that configures the RAG system to use OpenAI's embedding and language models.

Example:

rag, err := raggo.NewRAG(
    raggo.WithOpenAI(os.Getenv("OPENAI_API_KEY")),
)

type Record

type Record = rag.Record

type RegisterConfig

type RegisterConfig struct {
	// Storage settings control vector database configuration
	VectorDBType   string            // Type of vector database (e.g., "milvus", "pinecone")
	VectorDBConfig map[string]string // Database-specific configuration parameters
	CollectionName string            // Name of the collection to store vectors
	AutoCreate     bool              // Automatically create collection if missing

	// Processing settings define how documents are handled
	ChunkSize      int           // Size of text chunks for processing
	ChunkOverlap   int           // Overlap between consecutive chunks
	BatchSize      int           // Number of items to process in each batch
	TempDir        string        // Directory for temporary files
	MaxConcurrency int           // Maximum number of concurrent operations
	Timeout        time.Duration // Operation timeout duration

	// Embedding settings configure the embedding generation
	EmbeddingProvider string // Embedding service provider (e.g., "openai")
	EmbeddingModel    string // Specific model to use for embeddings
	EmbeddingKey      string // Authentication key for embedding service

	// Callbacks for monitoring and error handling
	OnProgress func(processed, total int) // Called to report progress
	OnError    func(error)                // Called when errors occur
}

RegisterConfig holds the complete configuration for document registration and vector database setup. It provides fine-grained control over all aspects of the registration process.

type RegisterOption

type RegisterOption func(*RegisterConfig)

RegisterOption is a function type for modifying RegisterConfig. It follows the functional options pattern to provide a clean and extensible way to configure the registration process.

func WithChunking

func WithChunking(size, overlap int) RegisterOption

WithChunking configures the text chunking parameters for document processing. The size parameter determines the length of each chunk, while overlap specifies how much text should be shared between consecutive chunks.

Example:

Register(ctx, "docs/",
    WithChunking(512, 64), // 512-token chunks with 64-token overlap
)

func WithCollection

func WithCollection(name string, autoCreate bool) RegisterOption

WithCollection sets the collection name and auto-creation behavior. When autoCreate is true, the collection will be created if it doesn't exist, including appropriate indexes for vector similarity search.

Example:

Register(ctx, "docs/",
    WithCollection("technical_docs", true),
)

func WithConcurrency

func WithConcurrency(max int) RegisterOption

WithConcurrency sets the maximum number of concurrent operations during document processing. This affects:

• Document loading
• Chunk processing
• Embedding generation
• Vector storage

Example:

Register(ctx, "docs/",
    WithConcurrency(8), // Process up to 8 items concurrently
)

func WithEmbedding

func WithEmbedding(provider, model, key string) RegisterOption

WithEmbedding configures the embedding generation settings. It specifies the provider, model, and authentication key for generating vector embeddings from text.

Supported providers:

• "openai": OpenAI's embedding models
• "cohere": Cohere's embedding models
• "local": Local embedding models

Example:

Register(ctx, "docs/",
    WithEmbedding("openai",
        "text-embedding-3-small",
        os.Getenv("OPENAI_API_KEY"),
    ),
)

func WithVectorDB

func WithVectorDB(dbType string, config map[string]string) RegisterOption

WithVectorDB configures the vector database settings for registration. It specifies the database type and its configuration parameters.

Supported database types include:

• "milvus": Milvus vector database
• "pinecone": Pinecone vector database
• "qdrant": Qdrant vector database

Example:

Register(ctx, "docs/",
    WithVectorDB("milvus", map[string]string{
        "address": "localhost:19530",
        "user": "default",
        "password": "password",
    }),
)

type Retriever

type Retriever struct {
	// contains filtered or unexported fields
}

Retriever handles semantic search operations with a reusable configuration. It provides a high-level interface for performing vector similarity searches and managing search results. The Retriever maintains connections to the vector database and embedding service throughout its lifecycle.

func NewRetriever

func NewRetriever(opts ...RetrieverOption) (*Retriever, error)

NewRetriever creates a new Retriever with the given options. It initializes the necessary connections and validates the configuration.

Example:

retriever, err := NewRetriever(
    WithRetrieveCollection("documents"),
    WithTopK(5),
    WithMinScore(0.7),
    WithRetrieveDB("milvus", "localhost:19530"),
    WithRetrieveEmbedding("openai", "text-embedding-3-small", os.Getenv("OPENAI_API_KEY")),
)

func (*Retriever) Close

func (r *Retriever) Close() error

func (*Retriever) GetVectorDB

func (r *Retriever) GetVectorDB() *VectorDB

GetVectorDB returns the underlying vector database instance. This provides access to lower-level database operations when needed.

func (*Retriever) Retrieve

func (r *Retriever) Retrieve(ctx context.Context, query string) ([]RetrieverResult, error)

Retrieve finds similar content for the given query using vector similarity search. It handles the complete retrieval pipeline:

1. Query embedding generation
2. Vector similarity search
3. Result filtering and processing
4. Metadata enrichment

Example:

results, err := retriever.Retrieve(ctx, "How does photosynthesis work?")
if err != nil {
    log.Fatal(err)
}
for _, result := range results {
    fmt.Printf("Score: %.2f, Content: %s\n", result.Score, result.Content)
}

type RetrieverConfig

type RetrieverConfig struct {
	// Core settings define the basic search behavior
	Collection string   // Name of the vector collection to search
	TopK       int      // Maximum number of results to return
	MinScore   float64  // Minimum similarity score threshold
	UseHybrid  bool     // Enable hybrid search (vector + keyword)
	Columns    []string // Columns to retrieve from the database

	// Vector DB settings configure the database connection
	DBType    string // Type of vector database (e.g., "milvus")
	DBAddress string // Database connection address
	Dimension int    // Embedding vector dimension

	// Embedding settings configure the embedding service
	Provider string // Embedding provider (e.g., "openai")
	Model    string // Model name for embeddings
	APIKey   string // Authentication key

	// Advanced settings provide additional control
	MetricType   string                 // Distance metric (e.g., "L2", "IP")
	Timeout      time.Duration          // Operation timeout
	SearchParams map[string]interface{} // Additional search parameters
	OnResult     func(SearchResult)     // Callback for each result
	OnError      func(error)            // Error handling callback
}

RetrieverConfig holds settings for the retrieval process. It provides fine-grained control over search behavior, database connections, and result processing.

type RetrieverOption

type RetrieverOption func(*RetrieverConfig)

RetrieverOption configures the retriever using the functional options pattern. This allows for flexible and extensible configuration while maintaining backward compatibility.

func WithColumns

func WithColumns(columns ...string) RetrieverOption

WithColumns specifies which columns to retrieve from the database. This can optimize performance by only fetching needed fields.

Example:

retriever, err := NewRetriever(
    WithColumns("Text", "Metadata", "Source"),
)

func WithHybrid

func WithHybrid(enabled bool) RetrieverOption

WithHybrid enables or disables hybrid search. Hybrid search combines vector similarity with keyword matching.

Example:

retriever, err := NewRetriever(
    WithHybrid(true), // Enable hybrid search
)

func WithMinScore

func WithMinScore(score float64) RetrieverOption

WithMinScore sets the minimum similarity score threshold. Results with scores below this threshold will be filtered out.

Example:

retriever, err := NewRetriever(
    WithMinScore(0.8), // Only return high-confidence matches
)

func WithRetrieveCallbacks

func WithRetrieveCallbacks(onResult func(SearchResult), onError func(error)) RetrieverOption

WithRetrieveCallbacks sets result and error handling callbacks. These callbacks are called during the retrieval process.

Example:

retriever, err := NewRetriever(
    WithRetrieveCallbacks(
        func(result SearchResult) {
            log.Printf("Found result: %v\n", result)
        },
        func(err error) {
            log.Printf("Error: %v\n", err)
        },
    ),
)

func WithRetrieveCollection

func WithRetrieveCollection(name string) RetrieverOption

WithRetrieveCollection sets the collection name for retrieval operations. The collection must exist in the vector database.

Example:

retriever, err := NewRetriever(
    WithRetrieveCollection("scientific_papers"),
)

func WithRetrieveDB

func WithRetrieveDB(dbType, address string) RetrieverOption

WithRetrieveDB configures the vector database connection. Supports various vector database implementations.

Example:

retriever, err := NewRetriever(
    WithRetrieveDB("milvus", "localhost:19530"),
)

func WithRetrieveDimension

func WithRetrieveDimension(dimension int) RetrieverOption

WithRetrieveDimension sets the embedding vector dimension. This must match the dimension of your chosen embedding model.

Example:

retriever, err := NewRetriever(
    WithRetrieveDimension(1536), // OpenAI embedding dimension
)

func WithRetrieveEmbedding

func WithRetrieveEmbedding(provider, model, key string) RetrieverOption

WithRetrieveEmbedding configures the embedding service. Supports multiple embedding providers and models.

Example:

retriever, err := NewRetriever(
    WithRetrieveEmbedding(
        "openai",
        "text-embedding-3-small",
        os.Getenv("OPENAI_API_KEY"),
    ),
)

func WithTopK

func WithTopK(k int) RetrieverOption

WithTopK sets the maximum number of results to return. The actual number of results may be less if MinScore filtering is applied.

Example:

retriever, err := NewRetriever(
    WithTopK(10), // Return top 10 results
)

type RetrieverResult

type RetrieverResult struct {
	Content    string                 `json:"content"`     // Retrieved text content
	Score      float64                `json:"score"`       // Similarity score
	Metadata   map[string]interface{} `json:"metadata"`    // Associated metadata
	Source     string                 `json:"source"`      // Source identifier
	ChunkIndex int                    `json:"chunk_index"` // Position in source
}

RetrieverResult represents a single retrieved result with its metadata and relevance information. It provides a structured way to access both the content and context of each search result.

type Schema

type Schema = rag.Schema

Types to match the internal rag package

type SearchResult

type SearchResult = rag.SearchResult

type SimpleRAG

type SimpleRAG struct {
	// contains filtered or unexported fields
}

SimpleRAG provides a minimal interface for RAG operations. It encapsulates the core functionality while hiding complexity.

func NewSimpleRAG

func NewSimpleRAG(config SimpleRAGConfig) (*SimpleRAG, error)

NewSimpleRAG creates a new SimpleRAG instance with minimal configuration. It performs the following setup: 1. Validates and applies configuration 2. Initializes the language model 3. Sets up the vector database connection 4. Prepares the retrieval system

Returns an error if: - API key is missing - LLM initialization fails - Vector database connection fails

func (*SimpleRAG) AddDocuments

func (s *SimpleRAG) AddDocuments(ctx context.Context, source string) error

AddDocuments processes and stores documents in the vector database. The function: 1. Validates the source path 2. Processes documents into chunks 3. Generates embeddings 4. Stores vectors in the database

The source parameter can be: - A single file path - A directory path (all documents will be processed) - A glob pattern (e.g., "docs/*.pdf")

func (*SimpleRAG) Close

func (s *SimpleRAG) Close() error

Close releases all resources held by the SimpleRAG instance. This includes: - Vector database connection - Language model resources - Any temporary files

func (*SimpleRAG) Search

func (s *SimpleRAG) Search(ctx context.Context, query string) (string, error)

Search performs a semantic search query and generates a response. The process: 1. Embeds the query into a vector 2. Finds similar documents in the vector database 3. Uses the LLM to generate a response based on retrieved context

Returns: - A natural language response incorporating retrieved information - An error if the search or response generation fails

type SimpleRAGConfig

type SimpleRAGConfig struct {
	Collection   string  // Name of the vector collection
	APIKey       string  // API key for services (e.g., OpenAI)
	Model        string  // Embedding model name
	ChunkSize    int     // Size of text chunks in tokens
	ChunkOverlap int     // Overlap between consecutive chunks
	TopK         int     // Number of results to retrieve
	MinScore     float64 // Minimum similarity score threshold
	LLMModel     string  // Language model for text generation
	DBType       string  // Type of vector database (e.g., "milvus", "chromem")
	DBAddress    string  // Address for the vector database
	Dimension    int     // Dimension of embedding vectors
}

SimpleRAGConfig holds configuration for SimpleRAG. It provides essential configuration options while using sensible defaults for other settings.

func DefaultConfig

func DefaultConfig() SimpleRAGConfig

DefaultConfig returns a default configuration for SimpleRAG. It provides reasonable defaults for all settings: - OpenAI's text-embedding-3-small for embeddings - Milvus as the vector database - Balanced chunk size and overlap - Conservative similarity threshold

type TokenCounter

type TokenCounter interface {
	// Count returns the number of tokens in the given text according to
	// the implementation's tokenization strategy.
	Count(text string) int
}

TokenCounter defines the interface for counting tokens in text. Different implementations can provide various tokenization strategies, from simple word-based counting to model-specific subword tokenization.

func NewDefaultTokenCounter

func NewDefaultTokenCounter() TokenCounter

NewDefaultTokenCounter creates a simple word-based token counter that splits text on whitespace. Suitable for basic use cases where exact token counts aren't critical.

func NewTikTokenCounter

func NewTikTokenCounter(encoding string) (TokenCounter, error)

NewTikTokenCounter creates a token counter using the tiktoken library, which implements the same tokenization used by OpenAI models. The encoding parameter specifies which tokenization model to use (e.g., "cl100k_base" for GPT-4, "p50k_base" for GPT-3).

type Vector

type Vector = rag.Vector

type VectorDB

type VectorDB struct {
	// contains filtered or unexported fields
}

VectorDB provides a high-level interface for vector database operations. It wraps the lower-level rag.VectorDB interface and adds: - Type-safe configuration - Connection management - Dimension tracking - Database type information

func NewVectorDB

func NewVectorDB(opts ...Option) (*VectorDB, error)

NewVectorDB creates a new vector database connection with the specified options. The function: 1. Applies all configuration options 2. Creates the appropriate database implementation 3. Sets up the connection (but doesn't connect yet)

Returns an error if: - The database type is unsupported - The configuration is invalid - The database implementation fails to initialize

func (*VectorDB) Address

func (vdb *VectorDB) Address() string

Address returns the address of the vector database.

func (*VectorDB) Close

func (vdb *VectorDB) Close() error

Close closes the vector database connection. This method should be called when the database is no longer needed.

func (*VectorDB) Connect

func (vdb *VectorDB) Connect(ctx context.Context) error

Connect establishes a connection to the vector database. This method must be called before any database operations.

func (*VectorDB) CreateCollection

func (vdb *VectorDB) CreateCollection(ctx context.Context, name string, schema Schema) error

CreateCollection creates a new collection in the database. The schema defines the structure of the collection.

func (*VectorDB) CreateIndex

func (vdb *VectorDB) CreateIndex(ctx context.Context, collectionName, field string, index Index) error

CreateIndex creates an index on a field in a collection. The index type defines the type of index to create.

func (*VectorDB) Dimension

func (vdb *VectorDB) Dimension() int

Dimension returns the dimension of the vectors in the database.

func (*VectorDB) DropCollection

func (vdb *VectorDB) DropCollection(ctx context.Context, name string) error

DropCollection drops a collection from the database. Returns an error if the collection does not exist.

func (*VectorDB) Flush

func (vdb *VectorDB) Flush(ctx context.Context, collectionName string) error

Flush flushes the pending operations in a collection. This method is used to ensure that all pending operations are written to disk.

func (*VectorDB) HasCollection

func (vdb *VectorDB) HasCollection(ctx context.Context, name string) (bool, error)

HasCollection checks if a collection exists in the database. Returns true if the collection exists, false otherwise.

func (*VectorDB) HybridSearch

func (vdb *VectorDB) HybridSearch(ctx context.Context, collectionName string, vectors map[string]Vector, topK int, metricType string, searchParams map[string]interface{}, reranker interface{}) ([]SearchResult, error)

HybridSearch performs a hybrid search in a collection. The search parameters define the search criteria. The reranker is used to rerank the search results. Returns a list of search results.

func (*VectorDB) Insert

func (vdb *VectorDB) Insert(ctx context.Context, collectionName string, data []Record) error

Insert inserts a batch of records into a collection. The records must match the schema of the collection.

func (*VectorDB) LoadCollection

func (vdb *VectorDB) LoadCollection(ctx context.Context, name string) error

LoadCollection loads a collection from disk. This method is used to load a collection that was previously created.

func (*VectorDB) Search

func (vdb *VectorDB) Search(ctx context.Context, collectionName string, vectors map[string]Vector, topK int, metricType string, searchParams map[string]interface{}) ([]SearchResult, error)

Search searches for vectors in a collection. The search parameters define the search criteria. Returns a list of search results.

func (*VectorDB) SetColumnNames

func (vdb *VectorDB) SetColumnNames(names []string)

SetColumnNames sets the column names for a collection. This method is used to set the column names for a collection.

func (*VectorDB) Type

func (vdb *VectorDB) Type() string

Type returns the type of the vector database.