regex

Documentation

Overview

Package regex is the v1-default chunker (ken's DESIGN.md §2 Option C): one generic line-walking engine driven by per-language LanguageRules. It registers itself as "regex" via chunk.Register in init(); import it for side effects (internal/search does) — chunk must not import this package (import cycle), so registration is decoupled the database/sql way.

This package moved here from ken (ADR-034); "DESIGN.md" refers to https://github.com/townsendmerino/ken/blob/main/docs/DESIGN.md.

Algorithm (ken's DESIGN.md §2 "Build path"): walk lines, mark the start of each top-level (or member-level) definition as a candidate boundary, greedily accumulate lines into a chunk, and when the next line would exceed chunkSize snap the cut back to the latest candidate boundary that still fits. If a single definition is itself larger than chunkSize there is no boundary to snap to, so it is split at line boundaries (the line-chunker fallback rule). Chunks are a contiguous, non-overlapping partition, so concatenating Text in order reproduces the source byte-for-byte.

This file holds the config types (LanguageRules and friends) and the public Chunker API (Chunk/chunkWith). The literal-prefix/first-byte prescreen precomputation lives in chunker_prescreen.go; the hand-rolled brace-depth state machine lives in chunker_scan.go.

Index

• type Chunker
  • func New() *Chunker
  • func (c *Chunker) Chunk(source []byte, language string, chunkSize int) ([]chunk.Chunk, error)
  • func (*Chunker) Name() string
  • func (c *Chunker) SupportedLanguages() []string
• type LanguageRules

Types

type Chunker

type Chunker struct {
	// contains filtered or unexported fields
}

Chunker implements chunk.Chunker.

func New

func New() *Chunker

New returns a Chunker over every registered language ruleset.

func (*Chunker) Chunk

func (c *Chunker) Chunk(source []byte, language string, chunkSize int) ([]chunk.Chunk, error)

func (*Chunker) Name

func (*Chunker) Name() string

func (*Chunker) SupportedLanguages

func (c *Chunker) SupportedLanguages() []string

type LanguageRules

type LanguageRules struct {
	// contains filtered or unexported fields
}

LanguageRules is the per-language driver for the generic engine.