semanticsearch

README

Semantic Search

This is a semantic search library inspired by Google's Discovery Engine (Google AI Search) and by the retrieval systems behind products like Google Search and NotebookLM.
It recursively indexes PDF, Markdown, code, and many other file types in a directory, then chunks them and stores them in a vector database using an embedding AI model.
It enables meaning-based search across your documents and works for both client-side and server-side solutions. Written in Go, it is portable and compiles easily to any platform, OS, client, or server.

Contents

• What semantic search is
• Use cases
• Supported formats
• How it works
• Architecture
• Requirements
• Install, build, test, lint
• Examples
• Usage
  • Full example
  • In-memory SQLite (single process)
  • Server-side setup with PostgreSQL and pgvector
  • Scaling up with HNSW
  • Choosing an embedder model
  • Optimizing search with tasks
  • Other search configurations
  • Custom AI client
  • Delta configurations
• Documents
• License

What semantic search is

Semantic search matches on meaning, not on shared words. A query and a result can rank as a strong match even when they have no words in common, because the search compares what they mean.

Search query	Search result
a gift for someone who loves cooking	The chef's guide to essential kitchen knives
how to feel less tired during the day	Tips for building a better sleep routine
my plant's leaves are turning yellow	Common causes of overwatering in houseplants
something fun to do with kids on a rainy day	Indoor board games for the whole family
ways to stay warm in winter	A guide to insulated jackets and wool layers

Use cases

Semantic Search works both as an embedded engine inside client apps (using the SQLite store, on disk or in memory) and as a server-side knowledge base (using PostgreSQL and pgvector).

Target	Use case
Desktop apps	Client-side RAG over a personal knowledge base: Google NotebookLM-like search built into the app, backed by the embedded SQLite database.
Mobile apps	The same personal knowledge base and meaning-based search running on-device, with no server, on the embedded SQLite database.
CLI tools	Terminal-based semantic search over local files and notes, backed by the embedded SQLite database.
Server-side	Meaning-based knowledge bases for web systems, for example integrating into webshops or product catalogs, backed by PostgreSQL and pgvector.

Supported formats

Format	Extensions	How it's chunked
Markdown	.md, .markdown, .mdown	Split by headings, with code blocks kept whole
PDF	.pdf	Headings detected from font sizes, read in natural page order
Plain text	.txt, .text, .log, .rst, .org, .adoc	Split into overlapping paragraphs
Code	.go, .js, .ts, .jsx, .tsx, .py, .php, .java, .rb, .rs, .c, .h, .cpp, .hpp, .cs, .sh, .sql	One section per function or class, titled with its full path
DOCX	.docx	Split by Word heading styles
HTML	.html, .htm, .xhtml	Text extracted from the HTML and split by <h1>-<h6> headings, with scripts, styles, and navigation dropped

How it works

1. Index: walk the tree; the strategy pool picks the strategy that claims each file.
2. Parse: decode bytes into heading/definition-structured sections.
3. Chunk: pack sections into token-budget chunks with overlap, each carrying its title path.
4. Embed: turn chunks into vectors via the embedding server.
5. Search: embed the query and rank chunks by vector distance using exact k-nearest-neighbor (kNN) search, comparing against every chunk for precise results.

Architecture

The Engine is the single entry point. It runs two flows, indexing files and searching. Both use the same building blocks. Strategies turn files into text chunks. An embedding model and an AI client turn text into vectors. Two stores keep the document metadata and the vectors.

flowchart TD
    App[Your application] --> Engine

    subgraph Engine [Semantic Search Engine Facade]
        Index[Index flow]
        Search[Search flow]
    end

    Index --> Strategies[Strategies<br/>Markdown · PDF · Code · Text · DOCX]
    Search --> Model
    Strategies --> Model[Embedding model<br/>prompt templates]
    Model --> Client[AI client<br/>OpenAI-compatible transport]
    Client --> Server[(Embedding server<br/>LM Studio · Ollama · remote)]

    Index --> Meta[(Metadata store<br/>SQLite · PostgreSQL)]
    Index --> Vectors[(Vector store<br/>sqlite-vec · pgvector)]
    Search --> Vectors
    Search --> Meta

    classDef blue fill:#E6F7FC,stroke:#10C2EB,stroke-width:2px,color:#0A5A72;
    classDef accent fill:#FFF3DC,stroke:#F5A623,stroke-width:2px,color:#8A5410;

    class App,Index,Search,Strategies blue;
    class Model,Client,Server,Meta,Vectors accent;

• Strategies claim files by type and split them into chunks; the AI client sends each chunk to the embedding server and gets back a vector.
• Indexing stores the document metadata and the vectors in their two stores.
• Searching embeds the query the same way, finds the nearest vectors, and resolves them back to their documents through the metadata store.

Requirements

• Every use case needs an OpenAI-compatible embedding server (it does not mean actual OpenAI models): on your own machine (LM Studio, Ollama, or llama.cpp), or on a remote host (Google AI Studio or any other server that speaks the standard protocol).
• For client-side apps (desktop, mobile, CLI), you also need a C compiler, because cgo builds mattn/go-sqlite3 and the sqlite-vec bindings from source:
  • macOS: xcode-select --install (Clang)
  • Debian / Ubuntu: sudo apt install build-essential
  • Fedora / RHEL: sudo dnf install gcc
  • Windows: install a MinGW-w64 gcc toolchain (e.g. via MSYS2) and add it to PATH
  • Windows (alternative): use WSL2 and follow the Debian / Ubuntu steps inside your Linux distribution
• For server-side apps: pure Go, so no C compiler is needed. You need a PostgreSQL server with the pgvector extension (test/docker/docker-compose.yml provides a working example).

Install, build, test, lint

Add the library to your module:

go get github.com/davidbelicza/semantic-search

Working on the library itself:

go build ./...   # build (cgo)
make test        # go test ./...
make lint        # golangci-lint

Examples

Runnable programs live in examples/, each a single main built around the sample files in examples/files. They need an OpenAI-compatible embedding server on http://127.0.0.1:1234 (e.g. LM Studio) serving EmbeddingGemma.

• basic: index into on-disk SQLite and run a search.
• progress: follow an index run with IndexOptions.OnProgress, printing how many of the scanned files have been processed.
• searchconfig: tune results with SearchConfig (task, minimum relevance, document and chunk limits).
• postgres: the server-side setup on PostgreSQL with pgvector.

git clone https://github.com/DavidBelicza/semantic-search.git
cd semantic-search

go run ./examples/basic
go run ./examples/progress
go run ./examples/searchconfig

The postgres example needs the bundled database running first:

docker compose -f test/docker/docker-compose.yml up -d
go run ./examples/postgres

Usage

Semantic Search is a library. Copy the following into a Go file (for example main.go) to get started: it composes an engine from an embedder, a metadata store, a vector store, and the strategies you want, then indexes a directory and searches it.

Full example

For CLI, desktop, or mobile apps, the recommended setup is an embedded SQLite database.

package main

import (
	"context"
	"fmt"

	"github.com/davidbelicza/semantic-search"
)

func main() {
	// Configure the search engine. You compose it from an embedder that turns
	// text into vectors, a metadata store, a vector store, and the strategies
	// that decide which file types are handled and how each one is parsed and
	// chunked.
	ctx := context.Background()
	store, _ := semanticsearch.NewSQLiteStorage(ctx, "index.db")
	defer store.Close()
	vectors, _ := semanticsearch.NewSQLiteVectorStorage(ctx, "vectors.db", 768)
	defer vectors.Close()
	model := semanticsearch.NewModel(semanticsearch.Gemma300mQAT)

	engine, err := semanticsearch.NewEngine(semanticsearch.Config{
		Model: model,
		Embedder: semanticsearch.NewAiEmbedder(semanticsearch.AiEmbedderConfig{
			Standard: semanticsearch.StandardOpenAI,
			BaseURL:  "http://127.0.0.1:1234",
		}, model),
		Storage:       store,
		VectorStorage: vectors,
		Strategies: []semanticsearch.StrategyFactory{
			semanticsearch.NewMarkdownStrategy(),
			semanticsearch.NewPDFStrategy(),
			semanticsearch.NewCodeStrategy(),
			semanticsearch.NewDocxStrategy(),
			semanticsearch.NewHTMLStrategy(),
			semanticsearch.NewTextStrategy(),
		},
	})
	if err != nil {
		panic(err)
	}

	// Index the directory. The engine maps the directory recursively, parses
	// every supported file, splits each one into chunks, and embeds those
	// chunks into vectors with the AI model.
	if err := engine.Index(ctx, "./docs", semanticsearch.IndexOptions{}); err != nil {
		panic(err)
	}

	// Search the indexed content. The query is embedded the same way, and
	// the engine returns the documents whose meaning is closest to it, each
	// carrying the chunks that matched inside it, so results are matched by
	// meaning rather than exact keywords.
	docs, _ := engine.Search(ctx, semanticsearch.SearchConfig{
		Query: "how do I detect security threats in logs",
	})
}

In-memory SQLite (single process)

Alternatively, give both stores an in-memory DSN to keep everything in RAM. Because the data lives only in this process, you must index and search in the same run. Only the two store lines change:

store, _ := semanticsearch.NewSQLiteStorage(ctx, "file:meta?mode=memory&cache=shared")
defer store.Close()
vectors, _ := semanticsearch.NewSQLiteVectorStorage(ctx, "file:vec?mode=memory&cache=shared", 768)
defer vectors.Close()

Server-side setup with PostgreSQL and pgvector

You can run this library server-side. In that case it is recommended to switch to a multi-process SQL database by swapping the two store constructors for their PostgreSQL equivalents. The server must have the pgvector extension; for local development, a ready-to-use database is provided:

docker compose -f test/docker/docker-compose.yml up -d

Only the two store lines change:

dsn := "postgres://semanticsearch:semanticsearch@127.0.0.1:5432/semanticsearch?sslmode=disable"
store, _ := semanticsearch.NewPostgresStorage(ctx, dsn)
defer store.Close()
vectors, _ := semanticsearch.NewPostgresVectorStorage(ctx, dsn, 768, semanticsearch.PostgresKNN)
defer vectors.Close()

The pgvector driver is pure Go, so a Postgres-only build (importing neither SQLite store) needs no cgo and no C compiler.

Scaling up with HNSW

If your vector database runs on the server side, you can reasonably scale it up. To do that, use PostgresHNSW instead of PostgresKNN: it builds an HNSW index for approximate nearest-neighbor search, which is sub-linear and much faster at scale. Only the vector store line changes:

vectors, _ := semanticsearch.NewPostgresVectorStorage(ctx, dsn, 768, semanticsearch.PostgresHNSW)

Choosing an embedder model

The model interface defines the model's name, dimension size, data structure format, and search query format. The example uses Gemma, which has 300 million parameters and 768 dimensions. It is a reasonable embedder model that can run locally. Changing models can significantly impact your application’s performance.

model := semanticsearch.NewModel(semanticsearch.Gemma300mQAT)

There are other pre-defined models available in this library:

• semanticsearch.NewModel(semanticsearch.Gemma300mQAT) loads text-embedding-embeddinggemma-300m-qat (768 dim)
• semanticsearch.NewModel(semanticsearch.Nomic768) loads text-embedding-nomic-embed-text-v1.5 (768 dim)
• semanticsearch.NewModel(semanticsearch.E5Large1024) loads text-embedding-multilingual-e5-large (1024 dim)
• semanticsearch.NewModel(semanticsearch.BGELarge1024) loads text-embedding-bge-large-en-v1.5 (1024 dim)
• semanticsearch.NewModel(semanticsearch.Qwen30_6B1024) loads text-embedding-qwen3-embedding-0.6b (1024 dim)
• semanticsearch.NewModel(semanticsearch.MxbaiLarge1024) loads text-embedding-mxbai-embed-large-v1 (1024 dim)

For any other model that needs no prompt templates, use NewGeneralModel with the model id and vector size. Switching models or dimensions is just a different argument.

model := semanticsearch.NewGeneralModel("text-embedding-nomic-embed-text-v1.5", 768)

If a model needs its own prompt templates, implement the EmbeddingModel interface and inject it.

type myModel struct{}

func (myModel) Name() string       { return "my-embedding-model" }
func (myModel) Dimensions() int    { return 1024 }
func (myModel) BuildData(chunk storage.Chunk) string { return chunk.Text }
func (myModel) BuildQuery(query, taskType string) (string, error) { return query, nil }

// semanticsearch.NewEngine(semanticsearch.Config{ Model: myModel{}, ... })

Optimizing search with tasks

Models can search differently depending on the task, and the available tasks depend on the model. The task is an optional last argument to Search; leave it out to use the model's default retrieval task. For example, Gemma searches differently based on its task:

semanticsearch.NewModel(semanticsearch.Gemma300mQAT)
...
engine.Search(ctx, semanticsearch.SearchConfig{Query: "I want a spicy tea"})

semanticsearch.NewModel(semanticsearch.Gemma300mQAT)
...
engine.Search(ctx, semanticsearch.SearchConfig{
	Query:    "I want a spicy tea",
	TaskType: semanticsearch.TaskGemma.Classification,
})

Gemma has 7 tasks. Other models instead take free text as the task. For example:

semanticsearch.NewModel(semanticsearch.Qwen30_6B1024)
...
engine.Search(ctx, semanticsearch.SearchConfig{
	Query:    "I want a spicy tea",
	TaskType: "Find the most exclusive product for this query",
})

Other search configurations

The search config also bounds the results: MinRelevance drops weak matches, MaxDocuments caps how many documents come back, and MaxChunks caps the chunks kept per document.

engine.Search(ctx, semanticsearch.SearchConfig{
	Query:        "I want a spicy tea",
	TaskType:     semanticsearch.TaskGemma.QuestionAnswering,
	MinRelevance: 0.3,
	MaxDocuments: 10,
	MaxChunks:    3,
})

Custom AI client

The built-in NewAiEmbedder returns an OpenAIClient that speaks the OpenAI-compatible protocol with an optional APIKey (sent as a Bearer token). For anything it does not cover, such as rotating OAuth tokens (e.g. production Vertex AI), request signing (e.g. AWS Bedrock), or a non-OpenAI wire format, implement the AiClient interface yourself and inject it. It is a single method:

type myClient struct {
	// your HTTP client, credentials, token cache, etc.
}

func (c myClient) Embed(ctx context.Context, texts []string) ([][]float32, error) {
	// Refresh your OAuth token / sign the request here, call your provider, and
	// return one vector per input text, in the same order.
}

// Inject it like any other client:
// semanticsearch.NewEngine(semanticsearch.Config{ Embedder: myClient{}, ... })

Delta configurations

By default, re-indexing removes documents whose files were deleted from disk, along with their chunks and vectors.

engine.Index(ctx, "path/to/files", semanticsearch.IndexOptions{})

Set KeepMissingFiles to keep those documents in the index even after their files are gone.

engine.Index(ctx, "path/to/files", semanticsearch.IndexOptions{KeepMissingFiles: true})

Documents

Reference

• docs/architecture.md: how the pipeline and strategies fit together.
• docs/chunking.md: how each format is parsed and chunked.

Research

• docs/research/vector-search-scaling.md: indexing and search performance, limits, and scaling options.
• docs/research/code-parsing-scaling.md: measurements for code parsing, compared with the book corpus.
• docs/research/sqlite-vec-migration.md: moving the vector store to sqlite-vec.
• docs/research/pdf-extraction-engine.md: PDF text extraction with PDFium.

License

Released under the MIT License.

Documentation

Overview

Package semanticsearch is the library entry point: it composes an embedder, a metadata store, a vector store, and a set of strategies into an Engine that indexes a directory and answers meaning-based queries. Every dependency is an interface, so callers can use the built-in implementations (the NewXxx constructors) or supply their own.

Index

• Constants
• Variables
• func NewAiEmbedder(config AiEmbedderConfig, model strategy.EmbeddingModel) strategy.AiClient
• func NewGeneralModel(name string, dimensions int) strategy.EmbeddingModel
• func NewModel(predefined PredefinedModel, dimensions ...int) strategy.EmbeddingModel
• func NewPostgresStorage(ctx context.Context, dsn string) (storage.Storage, error)
• func NewPostgresVectorStorage(ctx context.Context, dsn string, dimensions int, index PostgresVectorIndex) (storage.VectorStorage, error)
• func NewSQLiteStorage(ctx context.Context, path string) (storage.Storage, error)
• func NewSQLiteVectorStorage(ctx context.Context, path string, dimensions int) (storage.VectorStorage, error)
• type AiEmbedderConfig
• type Config
• type DocumentResult
• type Engine
  • func NewEngine(config Config) (*Engine, error)
  • func (e *Engine) Index(ctx context.Context, rootPath string, options IndexOptions) error
  • func (e *Engine) Search(ctx context.Context, config SearchConfig) ([]DocumentResult, error)
• type IndexOptions
• type IndexPhase
• type IndexProgress
• type PostgresVectorIndex
• type PredefinedModel
• type SearchConfig
• type SearchResult
• type Standard
• type StrategyFactory
  • func NewCodeStrategy() StrategyFactory
  • func NewDocxStrategy() StrategyFactory
  • func NewHTMLStrategy() StrategyFactory
  • func NewMarkdownStrategy() StrategyFactory
  • func NewPDFStrategy() StrategyFactory
  • func NewTextStrategy() StrategyFactory

Constants

const (
	// PhaseScanning walks the tree and registers what it finds. No total.
	PhaseScanning = pipeline.PhaseScanning
	// PhaseIndexing reads, chunks, and embeds the files whose content changed. total is the
	// files the walk found, so done finishes below it when only some of them changed.
	PhaseIndexing = pipeline.PhaseIndexing
	// PhaseCleanup removes documents whose files are gone. No total.
	PhaseCleanup = pipeline.PhaseCleanup
)

Variables

var (
	TaskGemma = model.GemmaTasks
	TaskNomic = model.NomicTasks
)

Model-specific query task types for Search. Any string is accepted; these are just the tasks each model documents.

Functions

func NewAiEmbedder

func NewAiEmbedder(config AiEmbedderConfig, model strategy.EmbeddingModel) strategy.AiClient

NewAiEmbedder builds the transport client for the given standard, configured to send the model's id and validate its vector size. It returns nil for an unknown standard or a nil model; NewEngine rejects a nil embedder.

func NewGeneralModel

func NewGeneralModel(name string, dimensions int) strategy.EmbeddingModel

NewGeneralModel builds a template-free model with the given model id and vector size, for any OpenAI-standard model that needs no prompt templates. Switching models or vector sizes is then just a different call, with no new type to implement.

func NewModel

func NewModel(predefined PredefinedModel, dimensions ...int) strategy.EmbeddingModel

NewModel builds the model knowledge (id, dimensions, prompt templates) for a predefined model. Any value that is not a predefined constant is treated as a raw model id and returned as a template-free GeneralModel with the given dimensions. The dimensions argument is optional and ignored for predefined models that have a fixed vector size; supply it only for an unlisted model id.

func NewPostgresStorage

func NewPostgresStorage(ctx context.Context, dsn string) (storage.Storage, error)

func NewPostgresVectorStorage

func NewPostgresVectorStorage(ctx context.Context, dsn string, dimensions int, index PostgresVectorIndex) (storage.VectorStorage, error)

NewPostgresVectorStorage opens a pgvector vector store at dsn, sized to the embedding dimensions, and prepares its schema. The server must have the pgvector extension available. The index selects exact (PostgresKNN) or approximate (PostgresHNSW) search. Point it at a different dsn than the metadata store to keep vectors in a separate database.

func NewSQLiteStorage

func NewSQLiteStorage(ctx context.Context, path string) (storage.Storage, error)

NewSQLiteStorage opens a SQLite metadata store at path and prepares its schema. The returned value is the injectable storage.Storage; a caller can implement that interface instead to use a different backend.

func NewSQLiteVectorStorage

func NewSQLiteVectorStorage(ctx context.Context, path string, dimensions int) (storage.VectorStorage, error)

NewSQLiteVectorStorage opens a sqlite-vec vector store at path, sized to the embedding dimensions, and prepares its schema. Point it at a different path than the metadata store to keep vectors in a separate database.

Types

type AiEmbedderConfig

type AiEmbedderConfig struct {
	Standard Standard
	BaseURL  string
	// APIKey is optional. When set it is sent as an "Authorization: Bearer <APIKey>" header
	// for hosted endpoints; leave it empty for local servers that need no authentication.
	APIKey string
	// Timeout is optional per-request timeout for embedding calls. Zero uses the default.
	Timeout time.Duration
}

AiEmbedderConfig configures the transport client: which protocol to speak and the endpoint, auth, and timeout to use. The model id and vector size come from the injected Model, not from here.

type Config

type Config struct {
	Model         strategy.EmbeddingModel
	Embedder      strategy.AiClient
	Storage       storage.Storage
	VectorStorage storage.VectorStorage
	Strategies    []StrategyFactory
	Searcher      search.Searcher
}

Config is the injected object graph for an Engine: the embedder, the two stores, and the strategies. All are required except Searcher, which defaults to the built-in document searcher. Every dependency is an interface, so a caller can supply the built-in implementations (via the NewXxx constructors) or their own.

type DocumentResult

type DocumentResult = search.DocumentResult

DocumentResult is one document match: its id, file name and path, relevance score, and the chunks that matched inside it, ranked best first. It is the type Search returns, defined in core/search and re-exported here for a single-import public API.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine is a configured index/search unit. Multiple engines with different embedders, stores, and strategies can run independently and in parallel.

func NewEngine

func NewEngine(config Config) (*Engine, error)

NewEngine validates the config and composes the engine. It errors on a missing dependency or when two strategies claim the same extension. Strategies are built per Index run (that is the only place they are used), so their resources live no longer than indexing needs them.

func (*Engine) Index

func (e *Engine) Index(ctx context.Context, rootPath string, options IndexOptions) error

Index runs the two pipelines: discover → register → fingerprint, then read → parse → chunk → embed. Re-running is incremental: unchanged files are not re-embedded. Documents whose files are gone are pruned unless IndexOptions.KeepMissingFiles is set. The strategies (and any resources they open, like the PDF extractor) are built here and released when indexing finishes.

func (*Engine) Search

func (e *Engine) Search(ctx context.Context, config SearchConfig) ([]DocumentResult, error)

Search embeds the query and returns the matching documents, most relevant first, each carrying the chunks that matched inside it. The config carries the query and its optional knobs (MinRelevance, MaxDocuments, MaxChunks, TaskType); passing a task type to a model that does not support one returns an error.

type IndexOptions

type IndexOptions struct {
	// FailFast aborts on the first per-document error instead of collecting and continuing.
	FailFast bool
	// IncludeHidden indexes hidden files and directories.
	IncludeHidden bool
	// FollowSymlinks resolves and indexes symlink targets.
	FollowSymlinks bool
	// KeepMissingFiles keeps documents whose files no longer exist on disk. By default indexing
	// removes them, along with their chunks and vectors.
	KeepMissingFiles bool
	// OnProgress, when set, is called as the run advances. It runs synchronously, so a slow
	// callback slows the run down.
	OnProgress IndexProgress
	// EmbedBatchSize is how many chunks are buffered across documents before a batch is sent to
	// the embedding server, and the cap on chunks per request. Leave it nil to use the default
	// (50); set it to a positive value to override (1 embeds one chunk per request). A non-nil
	// value of zero or less is rejected.
	EmbedBatchSize *int
}

IndexOptions configures an index run.

type IndexPhase

type IndexPhase = pipeline.Phase

IndexPhase names the stage an index run is currently in.

type IndexProgress

type IndexProgress = pipeline.Progress

IndexProgress receives an index run's counters. total is 0 when the phase has none.

type PostgresVectorIndex

type PostgresVectorIndex string

PostgresVectorIndex selects how the pgvector store searches.

const (
	// PostgresKNN is exact brute-force k-nearest-neighbor search (a sequential scan, 100%
	// recall). Best below a few hundred thousand vectors.
	PostgresKNN PostgresVectorIndex = "knn"
	// PostgresHNSW is approximate nearest-neighbor search backed by an HNSW index
	// (sub-linear, trades some recall for speed). Best at large scale.
	PostgresHNSW PostgresVectorIndex = "hnsw"
)

type PredefinedModel

type PredefinedModel string

PredefinedModel selects one of the built-in embedding models by name. Each one bundles the model's id, vector size, and the prompt templates it needs, so callers do not hand-write templates. For a model that is not listed, implement strategy.EmbeddingModel yourself and inject it.

const (
	// Gemma300mQAT is EmbeddingGemma (text-embedding-embeddinggemma-300m-qat, 768 dimensions).
	Gemma300mQAT PredefinedModel = "gemma-300m-qat"
	// Nomic768 is Nomic Embed Text v1.5 (768 dimensions).
	Nomic768 PredefinedModel = "nomic-v1.5"
	// E5Large1024 is Multilingual E5 large (1024 dimensions).
	E5Large1024 PredefinedModel = "e5-large"
	// BGELarge1024 is BGE large en v1.5 (1024 dimensions).
	BGELarge1024 PredefinedModel = "bge-large-en-v1.5"
	// Qwen30_6B1024 is Qwen3 Embedding 0.6B (1024 dimensions).
	Qwen30_6B1024 PredefinedModel = "qwen3-0.6b"
	// MxbaiLarge1024 is mxbai embed large v1 (1024 dimensions).
	MxbaiLarge1024 PredefinedModel = "mxbai-large-v1"
)

Predefined embedding models. Each constant bundles a model id, its native vector size, and the prompt templates the model requires. The dimension is encoded in the constant name so it is visible at the call site (Gemma keeps its parameter-based name).

type SearchConfig

type SearchConfig = search.SearchConfig

SearchConfig is the whole input to a search: the query and its optional knobs. It is defined in core/search and re-exported here for a single-import public API.

type SearchResult

type SearchResult = search.SearchResult

SearchResult is one chunk match: the document it belongs to, the chunk id, its title and text, and the relevance score (0 to 1, higher is closer). It is defined in core/search and re-exported here for a single-import public API.

type Standard

type Standard string

Standard identifies the wire protocol an AI embedder speaks. Most embedding servers are OpenAI-compatible; other standards can be added later without changing callers.

const StandardOpenAI Standard = "openai"

StandardOpenAI is the OpenAI-compatible /v1/embeddings protocol (LM Studio, Ollama, and most local servers).

type StrategyFactory

type StrategyFactory struct {
	Extensions []string
	Build      func(model strategy.EmbeddingModel, embedder strategy.AiClient) (strategy.Strategy, func() error, error)
}

StrategyFactory is a deferred strategy: it declares the extensions the strategy claims (so the engine can reject duplicates before indexing) and builds the strategy once the engine supplies the shared model and embedder. Build may return a cleanup the engine runs after indexing (e.g. to release the PDF extractor); the cleanup is nil when there is nothing to release.

To register a custom strategy, construct a StrategyFactory whose Build returns it.

func NewCodeStrategy

func NewCodeStrategy() StrategyFactory

NewCodeStrategy registers the source-code strategy.

func NewDocxStrategy

func NewDocxStrategy() StrategyFactory

NewDocxStrategy registers the DOCX strategy.

func NewHTMLStrategy

func NewHTMLStrategy() StrategyFactory

NewHTMLStrategy registers the HTML strategy.

func NewMarkdownStrategy

func NewMarkdownStrategy() StrategyFactory

NewMarkdownStrategy registers the Markdown strategy.

func NewPDFStrategy

func NewPDFStrategy() StrategyFactory

NewPDFStrategy registers the PDF strategy. Each engine gets its own PDFium extractor, which the engine releases after indexing.

func NewTextStrategy

func NewTextStrategy() StrategyFactory

NewTextStrategy registers the plain-text strategy (the general strategy used directly).