chunking

Documentation

Index

• func ASTChunker(code []byte, lang *sitter.Language) []string
• func SemanticChunker(text string, embedder func(string) ([]float32, error), threshold float32) ([]string, error)

Functions

func ASTChunker

func ASTChunker(code []byte, lang *sitter.Language) []string

ASTChunker splits source code into function/method/class-level chunks using tree-sitter AST parsing. It preserves scope integrity by never splitting inside a function body.

func SemanticChunker

func SemanticChunker(text string, embedder func(string) ([]float32, error), threshold float32) ([]string, error)

SemanticChunker splits text into semantically coherent chunks using cosine similarity valley detection between adjacent sentence embeddings.

Uses neurosnap/sentences (English tokenizer with built-in training data) for production-grade sentence boundary detection that correctly handles abbreviations (Dr., Mr., U.S.), decimal numbers (3.14), URLs, and Unicode.