chunk

Documentation

Overview

Package chunk turns ([]parse.SymbolSpan, source) into ([]types.Chunk) — the records the embedder + vector store actually persist.

Strategy:

1. Each symbol span (function/method/type/...) becomes one chunk.
2. Spans whose Text exceeds MaxInputTokens are split into sub-chunks (function bodies) or truncated at the head so the signature stays intact. Note that the mock embedder has infinite max input, so this only triggers with the real ONNX embedder.
3. A file_header chunk captures the first N lines of the file — package decl + imports + top-level const/var. This lets queries like "what package owns the metrics client" hit the right file even when nothing function-level matches.

Index

• Constants
• func BuildEmbedText(c types.Chunk) string
• func RawEmbedText(c types.Chunk) string
• type Chunker
  • func New(opts Options) *Chunker
  • func (c *Chunker) Chunk(in Input) []types.Chunk
• type Input
• type Options
• type Stats
  • func Summarize(chunks []types.Chunk) Stats

Constants

const DefaultFileHeaderLines = 50

DefaultFileHeaderLines is the number of leading lines of each file captured as a "file_header" chunk when Options.FileHeaderLines is unset. 50 is enough to cover the package decl + a typical import block + top-level consts. Per-project ckv.yaml can override this via chunking.file_header_lines.

const FileHeaderLines = DefaultFileHeaderLines

FileHeaderLines is the legacy export retained for callers that reference the constant directly. New code should set Options.FileHeaderLines and read Chunker.opts.

Functions

func BuildEmbedText

func BuildEmbedText(c types.Chunk) string

BuildEmbedText returns the embedder input for c with a rule-based contextual prefix prepended. Cheap, deterministic, no LLM call.

Format: one descriptive line + blank line + raw chunk text.

File header chunks:

"language: go. file: server.go. file header.\n\n<text>"

Doc section chunks (markdown):

"language: markdown. file: docs/x.md. section: why-sqlite-vec.\n\n<text>"

Symbol chunks:

"language: go. file: server.go. symbol: Server.Listen (Method).\n\n<text>"

Design notes:

• The prefix is regenerated on every build/reindex. It is NOT persisted in c.Text — chunk IDs hash the raw text only, so re-running with the prefix on/off does not invalidate IDs.
• The query intent is NOT prefixed. The asymmetry is intentional (Anthropic Contextual Retrieval pattern): prefixed chunks become easier to retrieve from natural-language queries because the embedding now carries location/type signal in addition to body.
• We pick natural-language phrasing over bracketed tags so the embedder (typically trained on English prose) parses it cleanly.

func RawEmbedText

func RawEmbedText(c types.Chunk) string

RawEmbedText returns c.Text unchanged. Pass it as embedTextFn to disable contextual prefixing — useful for A/B measurement and for the existing baseline test corpus.

Types

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker turns parsed spans into Chunks.

func New

func New(opts Options) *Chunker

New returns a Chunker with the given options. opts.MaxInputTokens of 0 disables truncation (appropriate for the mock embedder).

func (*Chunker) Chunk

func (c *Chunker) Chunk(in Input) []types.Chunk

Chunk produces all chunks for one file. The returned slice carries IDs and content_sha256 already computed — callers do not need to hash again before Upsert.

type Input

type Input struct {
	File       string // repo-relative path
	Language   string // "go" | "typescript" | "solidity" | "markdown"
	CommitHash string // built-time git HEAD
	Source     []byte // full file contents
	Spans      []parse.SymbolSpan
}

Input is everything the chunker needs about one file.

type Options

type Options struct {
	MaxInputTokens    int  // hard upper bound on chunk Text size; 0 → no cap
	IncludeFileHeader bool // emit a file_header chunk per file (default: true)
	// FileHeaderLines overrides DefaultFileHeaderLines (50). 0 keeps
	// the default. Sourced from project ckv.yaml.chunking.file_header_lines.
	FileHeaderLines int
	// IncludeFileFull emits an additional coarse chunk covering the whole file
	// (Phase B multi-granularity, roadmap §3.1). Off by default — opt-in for
	// measurement. Unlike file_header (first N lines) this spans the entire
	// file, giving file/module-level queries a coarse target alongside the fine
	// symbol chunks. Skipped for markdown (heading sections are already coarse).
	IncludeFileFull bool
}

Options configure chunking. Zero value uses documented defaults.

type Stats

type Stats struct {
	Total         int
	Symbol        int
	FileHeader    int
	Doc           int
	FunctionSplit int
	PRDoc         int
	Invariant     int
	Truncated     int
}

Stats summarizes the output of one Chunk() call; useful for build progress and the bootstrap report.

func Summarize

func Summarize(chunks []types.Chunk) Stats

Summarize counts chunk kinds. Cheap O(n) pass over the slice.