ingest

Documentation

Index

• type Embedder
• type Pipeline
  • func NewPipeline(s *store.Store, e Embedder, workers int) *Pipeline
  • func (p *Pipeline) Run(ctx context.Context, vaultPath string, glob string, stdin io.Reader, ...) error

Types

type Embedder

type Embedder interface {
	Embed(ctx context.Context, text string) ([]float32, error)
}

Embedder abstracts vector embedding so the pipeline can be tested with mocks.

type Pipeline

type Pipeline struct {
	// contains filtered or unexported fields
}

Pipeline orchestrates the ingestion of markdown files into the ChromaDB store.

func NewPipeline

func NewPipeline(s *store.Store, e Embedder, workers int) *Pipeline

NewPipeline creates an ingestion pipeline with the given number of concurrent workers.

func (*Pipeline) Run

func (p *Pipeline) Run(ctx context.Context, vaultPath string, glob string, stdin io.Reader, stdout io.Writer) error

Run walks the vault directory, finds markdown files matching glob, and ingests them into the store with a TUI progress bar rendered to stdout.