ingest

Documentation

Index

• func IsExcluded(relPath string, filters []string) bool
• func LoadExcludedPaths(vaultPath string) []string
• func RunProgress(totalFiles int, progressCh <-chan ProgressUpdate)
• type Embedder
• type ObsidianAppConfig
• type Pipeline
  • func NewPipeline(s *store.Store, e Embedder, workers int) *Pipeline
  • func (p *Pipeline) Run(ctx context.Context, vaultPath string, glob string, stdin io.Reader, ...) error
• type ProgressUpdate

Functions

func IsExcluded

func IsExcluded(relPath string, filters []string) bool

IsExcluded checks if the relative path matches any ignore filters.

func LoadExcludedPaths

func LoadExcludedPaths(vaultPath string) []string

LoadExcludedPaths reads the userIgnoreFilters and attachmentFolderPath from .obsidian/app.json. Returns nil if the file is absent or unreadable.

func RunProgress

func RunProgress(totalFiles int, progressCh <-chan ProgressUpdate)

RunProgress logs ingestion progress via structured slog events.

Types

type Embedder

type Embedder interface {
	Embed(ctx context.Context, text string) ([]float32, error)
}

Embedder abstracts vector embedding so the pipeline can be tested with mocks.

type ObsidianAppConfig

type ObsidianAppConfig struct {
	UserIgnoreFilters    []string `json:"userIgnoreFilters"`
	AttachmentFolderPath string   `json:"attachmentFolderPath"`
}

ObsidianAppConfig represents relevant fields from .obsidian/app.json.

type Pipeline

type Pipeline struct {
	MinChunkWords   int // minimum word count to keep a chunk (filters junk)
	ChunkSize       int // maximum runes per chunk fed to Parse
	ChunkOverlap    int // overlap runes between sub-chunks when a section is split
	MaxEmbedTokens  int // max tokens for embed text (model sequence length)
	RespectExclude  bool
	SkipAttachments bool
	// contains filtered or unexported fields
}

Pipeline orchestrates the ingestion of markdown files into the ChromaDB store.

func NewPipeline

func NewPipeline(s *store.Store, e Embedder, workers int) *Pipeline

NewPipeline creates an ingestion pipeline with the given number of concurrent workers. Set ChunkSize, ChunkOverlap, and MinChunkWords on the returned Pipeline before calling Run.

func (*Pipeline) Run

func (p *Pipeline) Run(ctx context.Context, vaultPath string, glob string, stdin io.Reader, stdout io.Writer) error

Run walks the vault directory, finds markdown files matching glob, and ingests them into the store with a TUI progress bar rendered to stdout.

type ProgressUpdate

type ProgressUpdate struct {
	Done    int
	Total   int
	Current string
	Final   bool
}

ProgressUpdate represents a status update during file processing.