README
¶
gotoken
A smart tokenization library for Go 1.25+ with multilingual support and language detection.
gotoken implements SmartToken: it splits each word at Unicode character-class and script boundaries into segments, then emits every contiguous run of up to D segments as an overlapping sub-token, where the depth D is chosen by a pluggable policy. The whole token is always emitted, so exact-match lookups keep full recall.
Features
- Sub-token generation — Generates multiple overlapping tokens from compound words
- Language detection — Identifies language of text segments using Unicode range tables
- Configurable depth — Policy-based control over tokenization granularity, including a hard per-token output cap
- Concurrency-safe — A
Tokenizeris immutable once built and safe to share across goroutines, provided itsPolicyis too (all built-in policies are) - Minimal dependencies — A single small dependency (
gocircular) for its generic ring buffer
Installation
go get github.com/rvncerr/gotoken
Usage
package main
import (
"fmt"
"unicode"
"github.com/rvncerr/gotoken"
)
func main() {
// Create a tokenizer with Latin and Cyrillic language support.
tok := gotoken.New(
gotoken.WithLanguages(unicode.Latin, unicode.Cyrillic),
gotoken.WithPolicy(gotoken.NewCountPolicy(16)),
)
tokens := tok.Tokenize("img2pdf helloпривет")
for sub, info := range tokens {
fmt.Printf("%q: language=%d, base=%v\n", sub, info.Language, info.Base)
}
// One map entry per distinct sub-token:
// "img", "2", "pdf", "img2", "2pdf", "img2pdf",
// "hello" (Latin), "привет" (Cyrillic), "helloпривет" (Cyrillic)
// A query for "pdf" now matches a document containing "img2pdf".
}
Policies
A policy maps a token's segment count to a tokenization depth; a depth ≤ 0 means "whole token
only". If no policy is configured, NewLinearPolicy(10, 10, 18, 2) is used.
// Fixed depth for all tokens
gotoken.NewFixedPolicy(5)
// Linear interpolation based on segment count
// Short tokens (≤3 segments): depth 10
// Long tokens (≥10 segments): depth 2
// In between: linearly interpolated
gotoken.NewLinearPolicy(3, 10, 10, 2)
// Limit maximum number of subtokens (recommended)
// Algorithmically optimal: finds largest depth that stays within limit
// Priority: whole token → single segments → pairs → triples → ...
gotoken.NewCountPolicy(50)
CountPolicy is recommended for production use:
- Guarantees bounded output size (the whole token is charged against the limit)
- No partially filled depth levels
- O(1) depth calculation via closed-form solution
Language Support
Add Unicode range tables for language detection. The order of added languages determines their index (0, 1, 2, ...); the first matching table wins.
tok := gotoken.New(
gotoken.WithLanguage(unicode.Latin),
gotoken.WithLanguage(unicode.Cyrillic),
gotoken.WithLanguage(unicode.Han),
)
// Or add multiple at once
tok := gotoken.New(
gotoken.WithLanguages(unicode.Latin, unicode.Cyrillic, unicode.Han),
)
Token Metadata
Each entry in the returned map carries a TokenInfo:
type TokenInfo struct {
Language int // index into the configured languages, or -1
Base [2]int // byte offsets [start, end) of the language-relevant portion
}
Base is relative to the start of the sub-token, not the source string. The
zero value [0, 0] is a sentinel meaning "undetermined" rather than an
empty span at offset 0 — check for it before slicing:
tokens := tok.Tokenize("hello123 123...")
tokens["hello123"] // {Language: 0, Base: [0, 5]} -> "hello" is the base
tokens["123..."] // {Language: -1, Base: [0, 0]} -> no language-relevant portion
A token is undetermined when it contains no letters, or when it spans more than three segments and so has no single identifiable language portion. This holds uniformly, including under a policy that emits only whole tokens.
API Reference
Creation
| Function | Description |
|---|---|
New(opts ...Option) |
Create a new tokenizer |
WithPolicy(p Policy) |
Set the depth policy (default: NewLinearPolicy(10, 10, 18, 2)) |
WithLanguage(rt *unicode.RangeTable) |
Add a language range table |
WithLanguages(tables ...*unicode.RangeTable) |
Add multiple language range tables |
Policies
| Constructor | Behavior |
|---|---|
NewFixedPolicy(depth) |
Constant depth for every token |
NewLinearPolicy(shortLen, shortDepth, longLen, longDepth) |
Interpolates depth between two anchors by segment count |
NewCountPolicy(maxCount) |
Largest depth that keeps a token within maxCount map entries (recommended) |
Tokenization
| Method | Description |
|---|---|
Tokenize(source string) |
Returns map[string]TokenInfo, keyed by sub-token text |
Types
| Type | Description |
|---|---|
Tokenizer |
The tokenizer; immutable and safe for concurrent use |
TokenInfo |
Detected language index and base byte range (see Token Metadata) |
Policy |
Interface mapping segment count to depth |
RuneClass |
Character classification (Letter, Digit, Punct, Other) |
License
MIT License - see LICENSE file.
Documentation
¶
Overview ¶
Package gotoken implements SmartToken, a deterministic multilingual tokenization algorithm for hashed full-text search.
SmartToken splits each whitespace-delimited token at Unicode character-class and script boundaries into segments, then emits every contiguous run of at most D segments as an overlapping sub-token, where the depth D is chosen by a pluggable Policy. The whole token is always emitted, so exact-match lookups keep full recall. Each sub-token carries a TokenInfo with the detected language index and the byte range of the language-relevant portion.
Three policies are provided: FixedPolicy returns a constant depth, LinearPolicy interpolates the depth from the segment count (the default), and CountPolicy picks the largest depth whose output stays within a hard per-token budget, which is the recommended choice for capacity-planned indexes.
A Tokenizer is immutable once constructed and safe for concurrent use, provided its Policy is too; all policies in this package are.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type CountPolicy ¶
type CountPolicy struct {
// contains filtered or unexported fields
}
CountPolicy limits the total number of subtokens generated per input token. It calculates the optimal depth to stay within the limit while:
- Always including the whole token
- Adding complete levels from smallest to largest subtokens
- Never partially filling a level
The whole token is charged against maxCount, so a token never expands into more than maxCount map entries.
func NewCountPolicy ¶
func NewCountPolicy(maxCount int) *CountPolicy
NewCountPolicy creates a policy that limits total subtoken count.
The algorithm prioritizes:
- Whole token (always included)
- Single-segment subtokens (depth 1)
- Two-segment subtokens (depth 2)
- And so on, until adding another level would exceed maxCount
func (*CountPolicy) Depth ¶
func (p *CountPolicy) Depth(segments int) int
Depth calculates the maximum depth that keeps total subtokens ≤ maxCount.
For N segments at depth D, the sliding window emits D*N - D*(D-1)/2 subtokens. Tokenize always emits the whole token on top of those whenever the window is too narrow to span it, so one slot is reserved for it and the largest D satisfying D*N - D*(D-1)/2 ≤ maxCount-1 is returned.
Returns 0 if the remaining budget is too small for even single-segment subtokens, meaning only the whole token should be emitted.
type FixedPolicy ¶
type FixedPolicy struct {
// contains filtered or unexported fields
}
FixedPolicy returns a constant depth regardless of segment count.
func NewFixedPolicy ¶
func NewFixedPolicy(depth int) *FixedPolicy
NewFixedPolicy creates a policy with constant depth. A negative depth is clamped to 0 (whole token only).
func (*FixedPolicy) Depth ¶
func (p *FixedPolicy) Depth(segments int) int
Depth returns the configured constant depth.
type LinearPolicy ¶
type LinearPolicy struct {
// contains filtered or unexported fields
}
LinearPolicy interpolates depth linearly between two points. For tokens with few segments (≤ shortLen), it returns shortDepth. For tokens with many segments (≥ longLen), it returns longDepth. For tokens in between, it linearly interpolates.
func NewLinearPolicy ¶
func NewLinearPolicy(shortLen, shortDepth, longLen, longDepth int) *LinearPolicy
NewLinearPolicy creates a policy that linearly interpolates depth.
Parameters:
- shortLen: tokens with this many segments or fewer get shortDepth
- shortDepth: depth for short tokens (typically higher)
- longLen: tokens with this many segments or more get longDepth
- longDepth: depth for long tokens (typically lower)
func (*LinearPolicy) Depth ¶
func (p *LinearPolicy) Depth(segments int) int
Depth returns the interpolated depth for the given segment count.
type Option ¶
type Option func(*Tokenizer)
Option configures a Tokenizer.
func WithLanguage ¶
func WithLanguage(rt *unicode.RangeTable) Option
WithLanguage adds a Unicode range table for language detection. The order of added languages determines their index (0, 1, 2, ...). A nil range table is ignored.
func WithLanguages ¶
func WithLanguages(tables ...*unicode.RangeTable) Option
WithLanguages adds multiple Unicode range tables for language detection. Nil range tables are ignored.
func WithPolicy ¶
WithPolicy sets the tokenization depth policy.
A nil Policy selects the default. That covers a nil interface only — an interface holding a nil pointer, as in WithPolicy(p) for a nil *FixedPolicy p, is itself non-nil and will fault on the first Depth call.
type Policy ¶
type Policy interface {
// Depth returns the maximum tokenization depth for a token with given segments.
Depth(segments int) int
}
Policy determines the tokenization depth based on segment count. A segment is a contiguous run of runes sharing a character class and, for letters, a detected language. Depth controls how many levels of sub-tokens are generated.
A Tokenizer calls Depth once per token, so an implementation shared by concurrent Tokenize calls must be safe for concurrent use. The policies in this package are immutable and therefore safe.
type RuneClass ¶
type RuneClass int
RuneClass categorizes Unicode runes into character classes.
Classes are numbered from 1. Zero is reserved as "no class assigned", so a default-initialized RuneClass never reads as a real classification.
const ( // ClassLetter covers Unicode letters (unicode.IsLetter). ClassLetter RuneClass // ClassDigit covers decimal digits only (unicode.IsDigit); other // numerals fall into ClassOther. ClassDigit // ClassPunct covers Unicode punctuation (unicode.IsPunct). ClassPunct // ClassOther covers every rune outside the classes above, including // symbols, marks, and non-decimal numerics. ClassOther )
type TokenInfo ¶
type TokenInfo struct {
// Language is the detected language index, or -1 if unknown or
// undetermined.
Language int
// Base contains half-open byte offsets [start, end) of the
// language-relevant portion, relative to the start of the (sub-)token, so
// subtoken[Base[0]:Base[1]] slices it directly.
//
// The zero value [0, 0] is a sentinel meaning "undetermined": it is used
// for non-letter tokens and for sub-tokens spanning more than
// maxAnalyzedSegments segments, where no single language span is
// identified. This holds on every path, including the whole-token-only one
// taken when a policy reports a depth of zero.
Base [2]int
}
TokenInfo contains metadata about a detected token.
type Tokenizer ¶
type Tokenizer struct {
// contains filtered or unexported fields
}
Tokenizer implements the SmartToken tokenization algorithm. It generates sub-tokens with language detection for multilingual text.
A Tokenizer is immutable once constructed, and each call to Tokenize uses its own scanning state, so it is safe for concurrent use by multiple goroutines provided the configured Policy is too. All policies in this package are.
The map returned by Tokenize is not shared and carries no such restriction.
func (*Tokenizer) Tokenize ¶
Tokenize processes the input string and returns all detected tokens with their info.
The result is keyed by sub-token text, so identical sub-token strings are deduplicated and the last occurrence wins. Map keys are sub-slices of source, so the returned map keeps source's backing storage alive.
Source Files