kb

Documentation

Overview

Package kb provides a knowledge base system for storing and searching documents.

Documents are chunked and embedded for hybrid search combining vector similarity and full-text search using Reciprocal Rank Fusion (RRF).

Index

• type Document
• type KBStore
  • func NewKBStore(db *sql.DB, embedder embeddings.Embedder, chunkSize, chunkOverlap int) *KBStore
  • func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
  • func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error
  • func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)
  • func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error
  • func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error
  • func (s *KBStore) FilesystemMirrorPath() string
  • func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)
  • func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)
  • func (s *KBStore) RebuildFTS(ctx context.Context) error
  • func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)
  • func (s *KBStore) SetSyncWorkers(workers int)
  • func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error
  • func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)
  • func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error
  • func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error
  • func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error
• type SearchResult
• type SyncStats

Types

type Document

type Document struct {
	ID        int64
	FilePath  string
	Content   string
	Metadata  map[string]interface{}
	CreatedAt time.Time
	UpdatedAt time.Time
}

Document represents a stored document in the knowledge base.

type KBStore

type KBStore struct {
	// contains filtered or unexported fields
}

KBStore manages knowledge base documents with chunking, embeddings, and hybrid search. It uses SQLite for storage with FTS5 for full-text search and in-memory vector search.

func NewKBStore

func NewKBStore(db *sql.DB, embedder embeddings.Embedder, chunkSize, chunkOverlap int) *KBStore

NewKBStore creates a new KBStore instance.

Parameters:

• db: SQLite database connection
• embedder: Embedder for generating chunk embeddings
• chunkSize: Maximum chunk size in characters (0 = use default)
• chunkOverlap: Overlap between consecutive chunks (0 = use default)

func (*KBStore) AddDocument

func (s *KBStore) AddDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

AddDocument adds a new document to the knowledge base. It chunks the content, generates embeddings, and updates the FTS index.

func (*KBStore) ConfigureFilesystemMirror

func (s *KBStore) ConfigureFilesystemMirror(dirPath string) error

ConfigureFilesystemMirror sets a base directory where KB documents are mirrored as markdown files on add/update/delete operations.

func (*KBStore) CountDocuments

func (s *KBStore) CountDocuments(ctx context.Context) (int64, error)

CountDocuments returns the total number of documents.

func (*KBStore) DeleteDocument

func (s *KBStore) DeleteDocument(ctx context.Context, filePath string) error

DeleteDocument removes a document and all its chunks from the knowledge base.

func (*KBStore) DeleteDocumentFromFilesystem

func (s *KBStore) DeleteDocumentFromFilesystem(filePath string) error

DeleteDocumentFromFilesystem removes a mirrored KB document from disk. If no mirror path is configured, this is a no-op.

func (*KBStore) FilesystemMirrorPath

func (s *KBStore) FilesystemMirrorPath() string

FilesystemMirrorPath returns the configured filesystem mirror path.

func (*KBStore) GetDocument

func (s *KBStore) GetDocument(ctx context.Context, filePath string) (*Document, error)

GetDocument retrieves a document by file path.

func (*KBStore) ListDocuments

func (s *KBStore) ListDocuments(ctx context.Context, limit, offset int) ([]Document, error)

ListDocuments returns paginated documents.

func (*KBStore) RebuildFTS

func (s *KBStore) RebuildFTS(ctx context.Context) error

RebuildFTS rebuilds the FTS5 index from kb_chunks.

func (*KBStore) SearchDocuments

func (s *KBStore) SearchDocuments(ctx context.Context, query string, limit int) ([]SearchResult, error)

SearchDocuments performs hybrid search combining vector similarity and FTS. Results are fused using Reciprocal Rank Fusion (RRF).

func (*KBStore) SetSyncWorkers

func (s *KBStore) SetSyncWorkers(workers int)

SetSyncWorkers configures how many concurrent workers are used for KB filesystem import preprocessing (file reads and in-memory preparation). SQLite writes remain serialized through a single writer goroutine.

func (*KBStore) SyncDirectory

func (s *KBStore) SyncDirectory(ctx context.Context, dirPath string) error

SyncDirectory imports or syncs all .md files from a directory. Existing documents are updated, new files are added.

func (*KBStore) SyncDirectoryWithStats

func (s *KBStore) SyncDirectoryWithStats(ctx context.Context, dirPath string, deleteMissing bool) (SyncStats, error)

SyncDirectoryWithStats imports or syncs all markdown files from a directory. It recursively scans for .md files, upserts modified documents, and optionally deletes KB documents that no longer exist on disk for that directory source.

func (*KBStore) UpdateDocument

func (s *KBStore) UpdateDocument(ctx context.Context, filePath, content string, metadata map[string]interface{}) error

UpdateDocument updates an existing document's content and metadata. It re-chunks and re-embeds the content.

func (*KBStore) WatchDirectory

func (s *KBStore) WatchDirectory(ctx context.Context, dirPath string) error

WatchDirectory monitors a KB source directory recursively and keeps KB documents in sync with .md file changes in near real-time.

func (*KBStore) WriteDocumentToFilesystem

func (s *KBStore) WriteDocumentToFilesystem(filePath, content string) error

WriteDocumentToFilesystem writes a KB document to the configured mirror path. If no mirror path is configured, this is a no-op.

type SearchResult

type SearchResult struct {
	Document     Document
	ChunkContent string
	Score        float64
	Rank         int
}

SearchResult represents a ranked search result from the knowledge base.

type SyncStats

type SyncStats struct {
	Scanned   int `json:"scanned"`
	Added     int `json:"added"`
	Updated   int `json:"updated"`
	Unchanged int `json:"unchanged"`
	Deleted   int `json:"deleted"`
}

SyncStats contains counters from a filesystem-to-KB synchronization run.