ingest

Documentation

Overview

Package ingest turns documents on disk into indexed chunks: it walks folders, extracts text with provenance (page numbers, headings), and splits it for the index.

Index

• Constants
• func SetOCR(fn func(pngImage []byte) (string, error))
• func SkipDir(name string) bool
• func Supported(path string) bool
• type Chunk
  • func SplitSections(sections []Section) []Chunk
• type Config
  • func LoadConfig(root string) (Config, error)
  • func (c Config) Excluded(rel string) bool
  • func (c Config) Save(root string) error
• type Progress
• type Section
  • func Extract(path string) ([]Section, error)
• type Stats
  • func IndexFolder(st *index.Store, folder, tag string, emb embed.Embedder, ...) (Stats, error)
  • func IndexOne(st *index.Store, sourceID int64, path string, emb embed.Embedder) (Stats, error)

Constants

const (
	PhaseExtracting = "extracting"
	PhaseEmbedding  = "embedding"
)

Progress phases, in the order a pass runs them.

const ConfigName = ".haypile.yml"

ConfigName is the per-folder configuration file, created by `hay init` and editable by hand. The daemon watches it: saving a change reconciles the index — newly excluded files drop out, newly included ones come in.

Functions

func SetOCR

func SetOCR(fn func(pngImage []byte) (string, error))

SetOCR installs the transcriber used for pages with no extractable text (scanned PDFs). A nil fn — or never calling this — means such pages index empty, exactly as before.

func SkipDir

func SkipDir(name string) bool

SkipDir reports whether a directory name is machine-managed and never worth walking into.

func Supported

func Supported(path string) bool

Supported reports whether the file at path is a format Haypile can index.

Types

type Chunk

type Chunk struct {
	Seq  int
	Page int // 1-based page number; 0 when the format has no pages
	Text string
}

Chunk is one indexable piece of a document.

func SplitSections

func SplitSections(sections []Section) []Chunk

SplitSections chunks extracted sections for indexing. Structure decides the boundaries: chunks never cross a section (page, heading scope), and within a section paragraphs pack together up to the budget. Consecutive chunks overlap so a fact straddling a boundary is findable in both.

type Config

type Config struct {
	// Tag applies to searches scoped with --tag. A tag passed explicitly
	// to `hay add` wins over this.
	Tag string `yaml:"tag,omitempty"`
	// Exclude lists doublestar patterns (gitignore-flavored globs) matched
	// against paths relative to the folder: "drafts/**", "*.bak",
	// "**/archive/**".
	Exclude []string `yaml:"exclude,omitempty"`
}

Config is a source folder's indexing configuration.

func LoadConfig

func LoadConfig(root string) (Config, error)

LoadConfig reads root's .haypile.yml. A missing file is a valid empty config; a malformed one is an error — silently indexing everything against the user's written intent would be worse than failing.

func (Config) Excluded

func (c Config) Excluded(rel string) bool

func (Config) Save

func (c Config) Save(root string) error

Save writes the config to root's .haypile.yml.

type Progress

type Progress struct {
	Phase       string `json:"phase"`
	Path        string `json:"path,omitempty"` // file just finished (extracting)
	FilesDone   int    `json:"files_done"`
	FilesTotal  int    `json:"files_total"`
	BytesDone   int64  `json:"bytes_done"`
	BytesTotal  int64  `json:"bytes_total"`
	ChunksDone  int    `json:"chunks_done"`
	ChunksTotal int    `json:"chunks_total"`
}

Progress is a snapshot of one indexing pass. Extraction is measured in files and bytes because that is what the walk knows up front; embedding is measured in chunks because extraction only just created them. Totals are fixed for the life of a phase, so done/total is an honest fraction.

type Section

type Section struct {
	Text string
	Page int // 1-based page number; 0 when the format has no pages
	// ScanSkipped marks a page that looked scanned (an image, no text)
	// with no vision model around to transcribe it; ScanFailed marks one
	// where a model existed but errored or read nothing. Both index
	// empty and both are counted into Stats, because the two situations
	// need different advice and silence would be mistaken for a broken
	// index.
	ScanSkipped bool
	ScanFailed  bool
}

Section is a contiguous run of extracted text with its provenance. The chunker never merges across section boundaries — a section is the unit a citation points at (a PDF page, a markdown heading's scope).

func Extract

func Extract(path string) ([]Section, error)

Extract pulls the text out of a document as ordered sections.

type Stats

type Stats struct {
	Indexed  int // files (re)indexed this pass
	Skipped  int // files already indexed and unchanged
	Failed   int // files whose extraction failed (indexing continues)
	Chunks   int // chunks written this pass
	Embedded int // vectors computed this pass (cache hits excluded)
	// ScanSkipped counts pages that looked scanned (an image, no text)
	// but had no vision model to transcribe them; ScanFailed counts
	// pages where the model errored or read nothing. Both index empty.
	// Surfaced by the CLI and web UI so the silence is never mistaken
	// for a broken index.
	ScanSkipped int
	ScanFailed  int
}

Stats reports what one IndexFolder pass did.

func IndexFolder

func IndexFolder(st *index.Store, folder, tag string, emb embed.Embedder, progress func(Progress)) (Stats, error)

IndexFolder walks folder, indexes every supported file into st, and prunes records for files that vanished from disk. Unchanged files (same content hash) are skipped. If emb is non-nil, chunks are embedded for semantic search, cheapest first: the content-addressed cache is consulted before the model runs. progress, if non-nil, receives a snapshot after every file and every embedding batch.

func IndexOne

func IndexOne(st *index.Store, sourceID int64, path string, emb embed.Embedder) (Stats, error)

IndexOne brings a single changed file under an existing source up to date — the daemon's watcher calls this per file event instead of re-walking the whole folder.