chunk

Documentation

Overview

Package chunk splits source files into retrieval units behind a runtime-selectable Chunker interface (see registry.go).

This package moved here from ken (ADR-034); "DESIGN.md" refers to https://github.com/townsendmerino/ken/blob/main/docs/DESIGN.md.

The Chunker interface seam supports three options per ken's DESIGN.md §2: `regex` (default; per-language regex rules, Stage 2), `treesitter` (opt-in; gotreesitter + cAST, v0.2.0 / ADR-010), and `line` (universal fallback, also used internally by the other two when they can't handle a language). Each registers itself via init() the database/sql way — chunk must not import its sub-chunker packages or there's an import cycle.

Load-bearing invariants every Chunker must satisfy:

• **Byte-fidelity (structural chunkers):** for regex and treesitter, concatenating the returned Chunk.Text fields in order reproduces the input source exactly; tests pin this per language because downstream code (snippet display, embedding, find_related's resolve-by-line lookup) trusts it. The "line" chunker is the intentional exception — it emits OVERLAPPING windows (Overlap lines shared between neighbors) for recall, so its concatenation is a superset of the source. Because ChunkFile falls back to line for unsupported languages (as do treesitter/markdown on failure), a resolve-by-line consumer must tolerate that overlap on fallback files rather than assume exact reconstruction.
• **Stamping:** the chunker leaves Chunk.File empty; ChunkFile stamps it on the way out so callers can pass any chunker and get consistent results.
• **Stateless after construction:** Chunk is safe to call across goroutines on a single Chunker instance (the registry hands out pointer instances, not factories).

Line numbers in the Chunk struct are 1-based and inclusive on both ends, matching how editors and grep report positions.

Example

Chunk a source file by name. The byte-fidelity invariant holds for every chunker: concatenating the chunks' Text reproduces the source exactly.

package main

import (
	"fmt"

	"github.com/townsendmerino/aikit/chunk"
	_ "github.com/townsendmerino/aikit/chunk/regex"
)

func main() {
	src := []byte("package main\n\nfunc main() {\n\tprintln(\"hi\")\n}\n")

	chunks, err := chunk.ChunkFile("regex", "main.go", src, 60)
	if err != nil {
		panic(err)
	}

	var joined []byte
	for _, c := range chunks {
		joined = append(joined, c.Text...)
	}
	fmt.Println("got chunks:", len(chunks) >= 1)
	fmt.Println("byte-faithful:", string(joined) == string(src))
}

Output:
got chunks: true
byte-faithful: true

Index

• Constants
• func Language(p string) string
• func Names() []string
• func Register(name string, c Chunker)
• type Chunk
  • func ChunkFile(name, file string, source []byte, chunkSize int) ([]Chunk, error)
• type Chunker
  • func Get(name string) (Chunker, error)
• type LineChunker
  • func NewLineChunker() *LineChunker
  • func (lc *LineChunker) Chunk(file string, source []byte) []Chunk

Examples

• Package

Constants

const DefaultChunkSize = 1500

DefaultChunkSize is the target chunk size in bytes (≈characters for the ASCII-heavy code ken indexes). ken's DESIGN.md "Build order" pins Stage 2 at 1500.

Functions

func Language

func Language(p string) string

Language returns the canonical language for a path, or "" if unknown. Some basenames are recognized without an extension.

func Names

func Names() []string

Names lists registered chunker names, sorted.

func Register

func Register(name string, c Chunker)

Register adds a chunker under name. Called from init() — the "line" chunker registers itself in this package; "regex" registers from chunk/regex (blank-imported by internal/search to avoid an import cycle: chunk must not import its own sub-chunkers). External mcp.Run authors call this directly to register a custom or ken-provided chunker before invoking mcp.Run (ADR-032).

Types

type Chunk

type Chunk struct {
	File      string // path relative to the index root
	StartLine int    // 1-based, inclusive
	EndLine   int    // 1-based, inclusive
	Text      string // exact source slice for [StartLine, EndLine]
	// Tombstoned marks a chunk whose source file has been deleted or
	// replaced under v0.3's incremental indexing (see
	// internal/search/watch.go). Transiently true within a single flush
	// — the mutator marks chunks as Tombstoned in-place, then
	// compactCorpus drops them before the snapshot is published.
	// Published snapshots never carry tombstones; the field matters
	// only on previously-published snapshots that an in-flight reader
	// still holds. Every read path (Search / FindRelated /
	// ResolveChunk) filters this field defensively. Wire-format callers
	// that round-trip Chunk to disk should preserve it; today the only
	// such caller is the bench harness, which never observes tombstoned
	// chunks because they never escape the search package.
	Tombstoned bool
}

Chunk is one indexed unit of a source file. Line numbers are 1-based and inclusive on both ends, matching how editors and grep report positions.

Chunk is part of the public chunker surface (ADR-032). The File / StartLine / EndLine / Text fields are the stable contract a Chunker implementation fills in. Tombstoned is a leakier case: it's an internal incremental-indexing detail (see below) exposed here only because the same struct round-trips through the watch path — external Chunker implementations should leave it false.

func ChunkFile

func ChunkFile(name, file string, source []byte, chunkSize int) ([]Chunk, error)

ChunkFile routes one file to the named chunker, falling back to the "line" chunker for languages the chosen chunker does not support, and stamps Chunk.File on the result. This is the single entry point the orchestration layer (internal/search) uses.

type Chunker

type Chunker interface {
	// Chunk partitions source into chunks. The structural chunkers (regex,
	// treesitter) are contiguous and non-overlapping, so concatenating their
	// Text in order reproduces source byte-for-byte. The built-in "line"
	// chunker is the deliberate EXCEPTION: it emits overlapping line windows
	// (a recall feature — a definition split across a window boundary still
	// appears whole in one chunk), so its concatenation is a superset of the
	// source, not an exact copy. ChunkFile routes unsupported languages
	// through it, and treesitter/markdown fall back to it, so a consumer that
	// reconstructs a file from chunks (resolve-by-line, snippet display) must
	// not assume exact reconstruction on line-chunked files — check Name() or
	// tolerate the overlap. Chunk.File is left empty for the caller to set.
	Chunk(source []byte, language string, chunkSize int) ([]Chunk, error)
	// SupportedLanguages returns the canonical language names this chunker
	// handles. An empty slice means "all languages" (the line fallback).
	SupportedLanguages() []string
	Name() string // "line" | "regex" | (future) "chroma" | "treesitter"
}

Chunker turns a source file into retrieval units. Implementations are stateless and goroutine-safe after construction.

The signature is deliberately minimal — no context.Context (regex chunking is synchronous and fast) and no filename (the caller stamps Chunk.File; a chunker only needs the language to pick rules). ken's DESIGN.md §2 originally sketched a ctx parameter; it was dropped because nothing in Option C needs it and Options B/A can adopt this same shape (see §2).

STABILITY (ADR-032): this interface — plus Register/Get/Names, ChunkFile, and the Chunk struct — is ken's PUBLIC, 1.0-committed chunker surface. External mcp.Run authors implement Chunker (or import one of ken's registered chunkers) and Register it before calling mcp.Run. The interface is small and dependency-free on purpose; it is the swap-out boundary ADR-010 designed for. The CONCRETE chunkers ken ships behind it (especially chunk/treesitter, which is backed by the pre-1.0 gotreesitter dep) are best-effort: their exact chunk boundaries may shift across versions. Depend on the interface, not on a specific chunker's byte-for-byte output.

func Get

func Get(name string) (Chunker, error)

Get returns the registered chunker, or an error listing what is available.

type LineChunker

type LineChunker struct {
	Size    int // lines per chunk (> 0)
	Overlap int // lines shared with the previous chunk (0 <= Overlap < Size)
}

LineChunker is the language-agnostic fallback: fixed-size line windows with a small overlap so a match straddling a window boundary still lands wholly inside at least one chunk. ken's DESIGN.md §1 pins the defaults at a 50-line window with 5 lines of overlap.

It is also the seam-validation stand-in until the Chunker interface arrives in Stage 2: every other chunker must produce the same Chunk shape.

func NewLineChunker

func NewLineChunker() *LineChunker

NewLineChunker returns the default 50/5 configuration from ken's DESIGN.md.

func (*LineChunker) Chunk

func (lc *LineChunker) Chunk(file string, source []byte) []Chunk

Chunk slices source into overlapping line windows. The returned Chunk.Text is the exact byte slice of source spanning [StartLine, EndLine] including the newline that terminates each interior line. An empty source yields no chunks.