textsplitter

package
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2026 License: MIT Imports: 17 Imported by: 0

README

TextSplitter for Go

A robust, recursive text splitting library for Go, designed to break down large texts into semantically meaningful chunks. This package is a port of the SentenceSplitter from LlamaIndex (Python), tailored for the Go ecosystem.

It supports:

  • Recursive Splitting: Paragraphs -> Sentences -> Words -> Characters.
  • Greedy Merging: Combines small chunks up to ChunkSize with precise ChunkOverlap.
  • Pluggable Components: Swap out tokenizers (e.g., simple whitespace, OpenAI's TikToken) and sentence splitting strategies (e.g., Regex, Neurosnap).

Installation

go get github.com/aqua777/go-llamaindex/textsplitter

Usage

Basic Usage

The default configuration uses a simple whitespace tokenizer and a regex-based sentence splitter.

package main

import (
	"fmt"
	"github.com/aqua777/go-llamaindex/textsplitter"
)

func main() {
	text := "Hello world. This is a test text that we want to split into chunks."

	// Create a splitter with:
	// - ChunkSize: 20 tokens (approx)
	// - ChunkOverlap: 5 tokens
	// - Tokenizer: nil (defaults to SimpleTokenizer)
	// - Strategy: nil (defaults to RegexSplitterStrategy)
	splitter := textsplitter.NewSentenceSplitter(20, 5, nil, nil)

	chunks := splitter.SplitText(text)

	for i, chunk := range chunks {
		fmt.Printf("Chunk %d: %s\n", i+1, chunk)
	}
}
Advanced Usage: OpenAI TikToken

For LLM applications, you often want to count tokens exactly as the model does. You can use the built-in TikTokenTokenizer.

package main

import (
	"fmt"
	"log"
	"github.com/aqua777/go-llamaindex/textsplitter"
)

func main() {
	text := "Your long text here..."

	// Initialize TikToken for GPT-3.5/4
	tokenizer, err := textsplitter.NewTikTokenTokenizer("gpt-3.5-turbo")
	if err != nil {
		log.Fatal(err)
	}

	// Use the custom tokenizer
	splitter := textsplitter.NewSentenceSplitter(1024, 200, tokenizer, nil)

	chunks := splitter.SplitText(text)
	fmt.Printf("Split into %d chunks using TikToken.\n", len(chunks))
}
Advanced Usage: Neurosnap Sentence Splitting

For higher quality sentence segmentation (handling abbreviations, etc.), you can use the NeurosnapSplitterStrategy. This package embeds the English training data, so it works out of the box with zero configuration.

package main

import (
	"fmt"
	"log"
	"github.com/aqua777/go-llamaindex/textsplitter"
)

func main() {
	// Pass nil to use the embedded English training data.
	strategy, err := textsplitter.NewNeurosnapSplitterStrategy(nil)
	if err != nil {
		log.Fatal("Could not load training data:", err)
	}

	splitter := textsplitter.NewSentenceSplitter(1024, 200, nil, strategy)

	text := "Mr. Smith went to Washington. He bought a 5.5 in. display."
	chunks := splitter.SplitText(text)

	// Should correctly handle "Mr." and "5.5 in." without splitting
	fmt.Println(chunks)
}

Configuration

NewSentenceSplitter
func NewSentenceSplitter(
    chunkSize int,
    chunkOverlap int,
    tokenizer Tokenizer,
    splitterStrategy SentenceSplitterStrategy,
) *SentenceSplitter
  • chunkSize: Target size of each chunk in tokens.
  • chunkOverlap: Number of tokens to overlap between chunks to maintain context.
  • tokenizer: Implementation of Tokenizer interface. Defaults to SimpleTokenizer (whitespace) if nil.
  • splitterStrategy: Implementation of SentenceSplitterStrategy interface. Defaults to RegexSplitterStrategy if nil.

Interfaces

You can implement your own components by satisfying these interfaces defined in iface.go:

// Tokenizer encodes text into a list of string tokens (or proxy tokens for counting).
type Tokenizer interface {
	Encode(text string) []string
}

// SentenceSplitterStrategy defines how to split a large text into primary sentences.
type SentenceSplitterStrategy interface {
	Split(text string) []string
}

License

MIT

Documentation

Index

Constants

View Source 
const (
	DefaultChunkSize     = 1024
	DefaultChunkOverlap  = 200
	DefaultParagraphSep  = "\n\n\n"
	DefaultSeparator     = " "
	DefaultChunkingRegex = `[^,.;。？！]+[,.;。？！]?|[,.;。？！]`
)
View Source 
const (
	EncodingCL100kBase = "cl100k_base" // GPT-4, GPT-3.5-turbo, text-embedding-ada-002
	EncodingP50kBase   = "p50k_base"   // Codex models, text-davinci-002/003
	EncodingR50kBase   = "r50k_base"   // GPT-3 models like davinci
	EncodingO200kBase  = "o200k_base"  // GPT-4o models
)

Common encoding names

View Source 
const MinEffectiveContentChunkTokens = 50

MinEffectiveContentChunkTokens is the minimum usable chunk size (in tokenizer units) after reserving space for metadata.

Variables

This section is empty.

Functions

func EffectiveChunkSizeAfterMetadata added in v0.0.3 

func EffectiveChunkSizeAfterMetadata(chunkSize, metadataTokenCount int) (int, error)

EffectiveChunkSizeAfterMetadata returns chunkSize minus metadataTokenCount, or an error if the remainder is below MinEffectiveContentChunkTokens.

func EffectiveChunkSizeForMetadataAwareSplit added in v0.0.3 

func EffectiveChunkSizeForMetadataAwareSplit(chunkSize int, tokenizer Tokenizer, metadata string) (int, error)

EffectiveChunkSizeForMetadataAwareSplit returns the content chunk size (in tokenizer units) after reserving space for metadata. It is shared by splitters that clone themselves with a reduced ChunkSize for metadata-aware splitting.

func GetEncodingForModel 

func GetEncodingForModel(model string) string

GetEncodingForModel returns the encoding name for a given model. Returns cl100k_base as default if model is not found.

func MetadataTokenCount added in v0.0.3 

func MetadataTokenCount(tokenizer Tokenizer, metadata string) int

MetadataTokenCount returns the tokenizer token count for metadata.

func SplitByChar 

func SplitByChar() func(string) []string

SplitByChar returns a function that splits text into characters.

func SplitByRegex 

func SplitByRegex(regexStr string) func(string) []string

SplitByRegex returns a function that splits text using a regex.

func SplitBySep 

func SplitBySep(sep string) func(string) []string

SplitBySep returns a function that splits text by a separator.

func SplitTextKeepSeparator 

func SplitTextKeepSeparator(text string, separator string) []string

SplitTextKeepSeparator splits text with separator and keeps the separator at the start of each split (except the first).

func ValidateLanguageConfig added in v0.0.3 

func ValidateLanguageConfig(c LanguageConfig) error

ValidateLanguageConfig returns an error if the language or spaCy model is not supported.

Types

type CodeSplitter added in v0.0.3 

type CodeSplitter struct {
	Language          string
	ChunkLines        int
	ChunkLinesOverlap int
	MaxChars          int
}

CodeSplitter splits code using language-specific rules: Go uses the Go parser (top-level declaration boundaries); Python uses top-level def/class/async def line detection; other languages use line windows with overlap and max size.

func NewCodeSplitter added in v0.0.3 

func NewCodeSplitter(language string, chunkLines int, chunkLinesOverlap int, maxChars int) *CodeSplitter

NewCodeSplitter creates a new CodeSplitter.

Args: 

language: The programming language of the code being split.
chunkLines: The number of lines to include in each chunk.
chunkLinesOverlap: How many lines of code each chunk overlaps with.
maxChars: Maximum number of characters (Unicode code points) per chunk.

Returns: 

A pointer to the newly created CodeSplitter.

Non-positive chunkLines or maxChars are replaced with defaults (40 lines, 1500 code points). Negative chunkLinesOverlap is replaced with the default (15). Zero overlap means no overlap. Positive overlap is clamped to be less than the effective chunk line count.

func (*CodeSplitter) SplitText added in v0.0.3 

func (s *CodeSplitter) SplitText(text string) []string

SplitText splits the provided code string into chunks.

Args: 

text: The code string to split.

Returns: 

A slice of code chunks.

type LanguageConfig added in v0.0.3 

type LanguageConfig struct {
	Language   string
	SpacyModel string
}

LanguageConfig configures the language and model name for SemanticDoubleMergingSplitter (aligned with LlamaIndex Python). The Go implementation does not load spaCy; the model field is validated for API compatibility.

type MarkdownHeaderType 

type MarkdownHeaderType struct {
	Level  int
	Header string
	Data   string
}

MarkdownHeaderType represents a markdown header.

type MarkdownSplitter 

type MarkdownSplitter struct {
	// ChunkSize is the maximum size of each chunk in tokens.
	ChunkSize int
	// ChunkOverlap is the number of overlapping tokens between chunks.
	ChunkOverlap int
	// Tokenizer is used to count tokens.
	Tokenizer Tokenizer
	// HeadersToSplitOn defines which header levels trigger splits.
	// Default: ["#", "##", "###", "####", "#####", "######"]
	HeadersToSplitOn []string
	// ReturnEachLine if true, returns each line as a separate chunk.
	ReturnEachLine bool
	// StripHeaders if true, removes headers from the output.
	StripHeaders bool
}

MarkdownSplitter splits markdown text while preserving structure. It respects headers, code blocks, and other markdown elements.

func NewMarkdownSplitter 

func NewMarkdownSplitter(chunkSize, chunkOverlap int) *MarkdownSplitter

NewMarkdownSplitter creates a new MarkdownSplitter with default settings.

func (*MarkdownSplitter) SplitText 

func (s *MarkdownSplitter) SplitText(text string) []string

SplitText splits markdown text into chunks.

func (*MarkdownSplitter) SplitTextMetadataAware 

func (s *MarkdownSplitter) SplitTextMetadataAware(text string, metadata string) ([]string, error)

SplitTextMetadataAware splits text accounting for metadata token usage.

func (*MarkdownSplitter) WithHeadersToSplitOn 

func (s *MarkdownSplitter) WithHeadersToSplitOn(headers []string) *MarkdownSplitter

WithHeadersToSplitOn sets which headers to split on.

func (*MarkdownSplitter) WithStripHeaders 

func (s *MarkdownSplitter) WithStripHeaders(strip bool) *MarkdownSplitter

WithStripHeaders sets whether to strip headers from output.

func (*MarkdownSplitter) WithTokenizer 

func (s *MarkdownSplitter) WithTokenizer(tokenizer Tokenizer) *MarkdownSplitter

WithTokenizer sets a custom tokenizer.

type MetadataAwareTextSplitter added in v0.0.3 

type MetadataAwareTextSplitter interface {
	TextSplitter

	// SplitTextMetadataAware splits text into chunks, accounting for metadata length.
	SplitTextMetadataAware(text string, metadata string) ([]string, error)
}

MetadataAwareTextSplitter splits text while accounting for metadata length (e.g. reserved context window for metadata in RAG).

type NeurosnapSplitterStrategy 

type NeurosnapSplitterStrategy struct {
	// contains filtered or unexported fields
}

NeurosnapSplitterStrategy uses neurosnap/sentences for sentence splitting.

func NewNeurosnapSplitterStrategy 

func NewNeurosnapSplitterStrategy(trainingData []byte) (*NeurosnapSplitterStrategy, error)

NewNeurosnapSplitterStrategy creates a new strategy using the provided JSON training data. If trainingData is nil or empty, it defaults to the embedded english.json training data.

func NewNeurosnapSplitterStrategyFromFile 

func NewNeurosnapSplitterStrategyFromFile(path string) (*NeurosnapSplitterStrategy, error)

NewNeurosnapSplitterStrategyFromFile creates a new strategy by reading training data from a file.

func (*NeurosnapSplitterStrategy) Split 

func (s *NeurosnapSplitterStrategy) Split(text string) []string

type RegexSplitterStrategy 

type RegexSplitterStrategy struct {
	// contains filtered or unexported fields
}

RegexSplitterStrategy uses regex for sentence splitting.

func NewRegexSplitterStrategy 

func NewRegexSplitterStrategy(regexStr string) *RegexSplitterStrategy

func (*RegexSplitterStrategy) Split 

func (s *RegexSplitterStrategy) Split(text string) []string

type SemanticDoubleMergingSplitter added in v0.0.3 

type SemanticDoubleMergingSplitter struct {
	LanguageConfig     LanguageConfig
	InitialThreshold   float64
	AppendingThreshold float64
	MergingThreshold   float64
	MaxChunkSize       int
	MergingRange       int
	MergingSeparator   string
	SentenceSplitter   SentenceSplitterStrategy
}

SemanticDoubleMergingSplitter splits text using a double merging semantic algorithm.

func NewSemanticDoubleMergingSplitter added in v0.0.3 

func NewSemanticDoubleMergingSplitter(
	languageConfig LanguageConfig,
	initialThreshold float64,
	appendingThreshold float64,
	mergingThreshold float64,
	maxChunkSize int,
	mergingRange int,
	mergingSeparator string,
	sentenceSplitter SentenceSplitterStrategy,
) *SemanticDoubleMergingSplitter

NewSemanticDoubleMergingSplitter creates a new SemanticDoubleMergingSplitter.

Args: 

languageConfig: Configuration for the language and model.
initialThreshold: Sets threshold for initializing new chunk.
appendingThreshold: Sets threshold for appending new sentences to chunk.
mergingThreshold: Sets threshold for merging whole chunks.
maxChunkSize: Maximum size of chunk (in characters).
mergingRange: How many chunks ahead beyond the nearest neighbor to merge if similar (1 or 2).
mergingSeparator: The separator to use when merging chunks.
sentenceSplitter: Strategy to split text into sentences; if nil, RegexSplitterStrategy with DefaultChunkingRegex is used.

Returns: 

A pointer to the newly created SemanticDoubleMergingSplitter. mergingRange is clamped to [1, 2].

func (*SemanticDoubleMergingSplitter) SplitText added in v0.0.3 

func (s *SemanticDoubleMergingSplitter) SplitText(text string) []string

SplitText splits the text into chunks using the double merging algorithm.

Args: 

text: The text string to split.

Returns: 

Semantically grouped chunks. Empty input, no sentences after trimming, or invalid
LanguageConfig yields a non-nil empty slice.

type SemanticSplitterNodeParser added in v0.0.3 

type SemanticSplitterNodeParser struct {
	EmbedModel                    embedding.EmbeddingModel
	BufferSize                    int
	BreakpointPercentileThreshold int
	SentenceSplitter              SentenceSplitterStrategy
}

SemanticSplitterNodeParser groups semantically related sentences using embedding similarity and a percentile-based breakpoint rule (aligned with LlamaIndex Python).

func NewSemanticSplitterNodeParser added in v0.0.3 

func NewSemanticSplitterNodeParser(
	embedModel embedding.EmbeddingModel,
	bufferSize int,
	breakpointPercentileThreshold int,
	sentenceSplitter SentenceSplitterStrategy,
) *SemanticSplitterNodeParser

NewSemanticSplitterNodeParser creates a new SemanticSplitterNodeParser.

Args: 

embedModel: The embedding model to use for similarity checks.
bufferSize: Number of sentences to include before/after when forming combined text per index.
breakpointPercentileThreshold: Percentile of pairwise dissimilarities used as the split threshold (0–100).
sentenceSplitter: Strategy to split text into sentences; if nil, RegexSplitterStrategy with DefaultChunkingRegex is used.

Returns: 

A configured SemanticSplitterNodeParser. Non-positive bufferSize defaults to 1.
Threshold is clamped to [0, 100].

func (*SemanticSplitterNodeParser) SplitText added in v0.0.3 

func (s *SemanticSplitterNodeParser) SplitText(text string) []string

SplitText splits the text into semantically grouped chunks.

Args: 

text: The text string to split.

Returns: 

A slice of chunks joined from original sentences (no added spaces between sentences).
On embedding failure, invalid embeddings for similarity, or pairwise similarity failure,
returns nil. Empty input, no embed model, or no sentences after trimming yields a non-nil
empty slice.

type SentenceSplitter 

type SentenceSplitter struct {
	ChunkSize              int
	ChunkOverlap           int
	Separator              string
	ParagraphSeparator     string
	SecondaryChunkingRegex string
	Tokenizer              Tokenizer
	SplitterStrategy       SentenceSplitterStrategy
	// contains filtered or unexported fields
}

SentenceSplitter splits text with a preference for complete sentences.

func NewSentenceSplitter 

func NewSentenceSplitter(
	chunkSize int,
	chunkOverlap int,
	tokenizer Tokenizer,
	splitterStrategy SentenceSplitterStrategy,
) *SentenceSplitter

NewSentenceSplitter creates a new SentenceSplitter. Pass 0 or empty strings to use defaults. If tokenizer is nil, defaults to SimpleTokenizer. If splitterStrategy is nil, defaults to RegexSplitterStrategy with DefaultChunkingRegex.

func NewSentenceSplitterWithValidation 

func NewSentenceSplitterWithValidation(
	chunkSize int,
	chunkOverlap int,
	tokenizer Tokenizer,
	splitterStrategy SentenceSplitterStrategy,
) (*SentenceSplitter, error)

NewSentenceSplitterWithValidation creates a new SentenceSplitter with input validation. Returns an error if parameters are invalid.

func (*SentenceSplitter) SplitText 

func (s *SentenceSplitter) SplitText(text string) []string

SplitText splits the text into chunks.

func (*SentenceSplitter) SplitTextMetadataAware 

func (s *SentenceSplitter) SplitTextMetadataAware(text string, metadata string) ([]string, error)

SplitTextMetadataAware splits text into chunks, accounting for metadata length. This is useful for RAG applications where metadata consumes context window.

func (*SentenceSplitter) Validate 

func (s *SentenceSplitter) Validate() error

Validate validates the current splitter configuration.

func (*SentenceSplitter) WithOnChunkingEnd 

func (s *SentenceSplitter) WithOnChunkingEnd(fn func(chunks []string)) *SentenceSplitter

WithOnChunkingEnd sets the callback for when chunking ends.

func (*SentenceSplitter) WithOnChunkingStart 

func (s *SentenceSplitter) WithOnChunkingStart(fn func(text []string)) *SentenceSplitter

WithOnChunkingStart sets the callback for when chunking starts.

type SentenceSplitterStrategy 

type SentenceSplitterStrategy interface {
	Split(text string) []string
}

SentenceSplitterStrategy is the interface for primary sentence splitting.

type SentenceWindow 

type SentenceWindow struct {
	// Sentence is the original sentence.
	Sentence string
	// Window is the sentence with surrounding context.
	Window string
	// Index is the sentence index in the original text.
	Index int
	// StartSentence is the index of the first sentence in the window.
	StartSentence int
	// EndSentence is the index of the last sentence in the window.
	EndSentence int
}

SentenceWindow represents a sentence with its surrounding context.

type SentenceWindowNodeData 

type SentenceWindowNodeData struct {
	// Text is the sentence text (for embedding/matching).
	Text string
	// Window is the surrounding context.
	Window string
	// Metadata contains additional metadata.
	Metadata map[string]interface{}
}

SentenceWindowNodeData contains data for creating nodes with window metadata.

type SentenceWindowSplitter 

type SentenceWindowSplitter struct {
	// WindowSize is the number of sentences to include on each side.
	WindowSize int
	// Tokenizer is used to count tokens.
	Tokenizer Tokenizer
	// SentenceSplitter is used to split text into sentences.
	SentenceSplitter SentenceSplitterStrategy
	// OriginalTextMetadataKey is the metadata key for storing the original sentence.
	OriginalTextMetadataKey string
	// WindowMetadataKey is the metadata key for storing the window text.
	WindowMetadataKey string
}

SentenceWindowSplitter splits text into sentences and includes surrounding context (window) for each sentence. This is useful for retrieval where you want to match on a specific sentence but return more context.

func NewSentenceWindowSplitter 

func NewSentenceWindowSplitter(windowSize int) *SentenceWindowSplitter

NewSentenceWindowSplitter creates a new SentenceWindowSplitter.

func (*SentenceWindowSplitter) GetWindowsText 

func (s *SentenceWindowSplitter) GetWindowsText(text string) []string

GetWindowsText returns just the window texts (for embedding the context).

func (*SentenceWindowSplitter) SplitText 

func (s *SentenceWindowSplitter) SplitText(text string) []string

SplitText splits text into sentences (returns just the sentences). Use SplitTextWithWindows for full window information.

func (*SentenceWindowSplitter) SplitTextForNodes 

func (s *SentenceWindowSplitter) SplitTextForNodes(text string) []SentenceWindowNodeData

SplitTextForNodes returns data suitable for creating nodes with window metadata.

func (*SentenceWindowSplitter) SplitTextWithWindows 

func (s *SentenceWindowSplitter) SplitTextWithWindows(text string) []SentenceWindow

SplitTextWithWindows splits text and returns sentences with their windows.

func (*SentenceWindowSplitter) WithMetadataKeys 

func (s *SentenceWindowSplitter) WithMetadataKeys(originalKey, windowKey string) *SentenceWindowSplitter

WithMetadataKeys sets custom metadata keys.

func (*SentenceWindowSplitter) WithSentenceSplitter 

func (s *SentenceWindowSplitter) WithSentenceSplitter(splitter SentenceSplitterStrategy) *SentenceWindowSplitter

WithSentenceSplitter sets a custom sentence splitter.

func (*SentenceWindowSplitter) WithTokenizer 

func (s *SentenceWindowSplitter) WithTokenizer(tokenizer Tokenizer) *SentenceWindowSplitter

WithTokenizer sets a custom tokenizer.

type SimpleTokenizer 

type SimpleTokenizer struct{}

SimpleTokenizer tokenizes text by splitting on whitespace.

func NewSimpleTokenizer 

func NewSimpleTokenizer() *SimpleTokenizer

func (*SimpleTokenizer) Encode 

func (t *SimpleTokenizer) Encode(text string) []string

type TextSplitter 

type TextSplitter interface {
	SplitText(text string) []string
}

TextSplitter is the interface for splitting text.

type TikTokenTokenizer 

type TikTokenTokenizer struct {
	// contains filtered or unexported fields
}

TikTokenTokenizer tokenizes text using OpenAI's tiktoken.

func NewTikTokenTokenizer 

func NewTikTokenTokenizer(model string) (*TikTokenTokenizer, error)

func (*TikTokenTokenizer) CountTokens 

func (t *TikTokenTokenizer) CountTokens(text string) int

CountTokens counts tokens using the TikTokenTokenizer.

func (*TikTokenTokenizer) Encode 

func (t *TikTokenTokenizer) Encode(text string) []string

type TikTokenTokenizerByEncoding 

type TikTokenTokenizerByEncoding struct {
	// contains filtered or unexported fields
}

TikTokenTokenizerByEncoding creates a tokenizer using a specific encoding name.

func NewTikTokenTokenizerByEncoding 

func NewTikTokenTokenizerByEncoding(encodingName string) (*TikTokenTokenizerByEncoding, error)

NewTikTokenTokenizerByEncoding creates a tokenizer using a specific encoding.

func (*TikTokenTokenizerByEncoding) CountTokens 

func (t *TikTokenTokenizerByEncoding) CountTokens(text string) int

CountTokens returns the number of tokens in the text.

func (*TikTokenTokenizerByEncoding) Decode 

func (t *TikTokenTokenizerByEncoding) Decode(tokenIDs []int) string

Decode converts token IDs back to text.

func (*TikTokenTokenizerByEncoding) Encode 

func (t *TikTokenTokenizerByEncoding) Encode(text string) []string

Encode tokenizes text and returns token strings.

func (*TikTokenTokenizerByEncoding) EncodeToIDs 

func (t *TikTokenTokenizerByEncoding) EncodeToIDs(text string) []int

EncodeToIDs returns the raw token IDs.

func (*TikTokenTokenizerByEncoding) EncodingName 

func (t *TikTokenTokenizerByEncoding) EncodingName() string

EncodingName returns the encoding name.

type TokenCounter 

type TokenCounter interface {
	CountTokens(text string) int
}

TokenCounter is an interface for counting tokens.

type TokenTextSplitter 

type TokenTextSplitter struct {
	// ChunkSize is the maximum number of tokens per chunk.
	ChunkSize int
	// ChunkOverlap is the number of overlapping tokens between chunks.
	ChunkOverlap int
	// Tokenizer is used to count tokens. Defaults to SimpleTokenizer.
	Tokenizer Tokenizer
	// Separator is used to split text into initial segments. Defaults to " ".
	Separator string
	// KeepSeparator determines if separators are kept in the output.
	KeepSeparator bool
}

TokenTextSplitter splits text based on token count rather than character count. This is useful when working with LLMs that have token limits.

func NewTokenTextSplitter 

func NewTokenTextSplitter(chunkSize, chunkOverlap int) *TokenTextSplitter

NewTokenTextSplitter creates a new TokenTextSplitter with default settings.

func NewTokenTextSplitterWithTokenizer 

func NewTokenTextSplitterWithTokenizer(chunkSize, chunkOverlap int, tokenizer Tokenizer) *TokenTextSplitter

NewTokenTextSplitterWithTokenizer creates a TokenTextSplitter with a custom tokenizer.

func NewTokenTextSplitterWithValidation 

func NewTokenTextSplitterWithValidation(chunkSize, chunkOverlap int, tokenizer Tokenizer) (*TokenTextSplitter, error)

NewTokenTextSplitterWithValidation creates a TokenTextSplitter with input validation. Returns an error if parameters are invalid.

func (*TokenTextSplitter) SplitText 

func (s *TokenTextSplitter) SplitText(text string) []string

SplitText splits text into chunks based on token count.

func (*TokenTextSplitter) SplitTextMetadataAware 

func (s *TokenTextSplitter) SplitTextMetadataAware(text string, metadata string) ([]string, error)

SplitTextMetadataAware splits text accounting for metadata token usage.

func (*TokenTextSplitter) Validate 

func (s *TokenTextSplitter) Validate() error

Validate validates the current splitter configuration.

func (*TokenTextSplitter) WithKeepSeparator 

func (s *TokenTextSplitter) WithKeepSeparator(keep bool) *TokenTextSplitter

WithKeepSeparator sets whether to keep separators.

func (*TokenTextSplitter) WithSeparator 

func (s *TokenTextSplitter) WithSeparator(sep string) *TokenTextSplitter

WithSeparator sets a custom separator.

type Tokenizer 

type Tokenizer interface {
	Encode(text string) []string
}

Tokenizer is the interface for tokenizing text. It encodes text into a list of string tokens.

func DefaultTokenizer 

func DefaultTokenizer() (Tokenizer, error)

DefaultTokenizer returns a shared default TikToken tokenizer using cl100k_base encoding. This is safe for concurrent use.

func MustDefaultTokenizer 

func MustDefaultTokenizer() Tokenizer

MustDefaultTokenizer returns the default tokenizer or panics on error.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.