tokenizer

package
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 5, 2026 License: MIT Imports: 21 Imported by: 0

Documentation

Overview

Package tokenizer provides token counting for LLM models.

It supports exact BPE tokenization for OpenAI models, character-based approximation for Claude and Gemini models, and SentencePiece tokenization for open-source models like Llama and Mistral.

All counting runs locally from embedded vocabularies; no network access is required or used.

Index

Examples

Constants

View Source 
const (
	EncodingO200kBase    = bpe.EncodingO200kBase
	EncodingCL100kBase   = bpe.EncodingCL100kBase
	EncodingClaudeApprox = "claude_approx"
	EncodingGeminiApprox = "gemini_approx"
	EncodingSPM          = "spm"
)

Encoding identifiers shared across the tokenizer and CLI layers.

View Source 
const (
	DefaultCharsPerToken = 4.0
	DefaultWordsPerToken = 0.75
)

Default approximation ratios applied when CounterOptions leaves them zero.

View Source 
const NameClaudeApprox = "claude_3_approx"

NameClaudeApprox is the machine-readable identifier the Claude approximator reports; consumers key accuracy labeling off it.

Variables

View Source 
var (
	// ErrModelNotFound is returned when a requested model is not in the registry.
	ErrModelNotFound = errors.New("model not found")

	// ErrEncodingNotFound is returned when a BPE encoding name is not recognized.
	ErrEncodingNotFound = errors.New("encoding not found")

	// ErrVocabFileRequired is returned when a SentencePiece model path is empty.
	ErrVocabFileRequired = errors.New("vocab file path is required")

	// ErrBinaryFile is returned when attempting to count tokens in a binary file.
	ErrBinaryFile = errors.New("file is binary")
)

Sentinel errors for common failure modes.

Functions

func IsOpenSourceModel 

func IsOpenSourceModel(modelName string) bool

IsOpenSourceModel returns true if the model is from an open-source provider (not OpenAI, Anthropic, or Google).

func ListModels 

func ListModels() []string

ListModels returns all registered model names in sorted order.

func ModelsByEncoding 

func ModelsByEncoding() map[string][]string

ModelsByEncoding returns a map of encoding name to sorted model names.

Types

type BPETokenizerWrapper 

type BPETokenizerWrapper struct {
	// contains filtered or unexported fields
}

BPETokenizerWrapper implements exact tokenization using a BPE encoding.

func (*BPETokenizerWrapper) Contract added in v0.5.1 

func (t *BPETokenizerWrapper) Contract() ContractIdentity

Contract identifies ordinary BPE encoding. Model aliases that resolve to this wrapper therefore share one reusable result.

func (*BPETokenizerWrapper) CountTokens 

func (t *BPETokenizerWrapper) CountTokens(text string) (int, error)

CountTokens counts tokens using BPE tokenization. Counting never allows special tokens, so it takes the ordinary encode path, which skips the special-token scan and produces identical counts.

func (*BPETokenizerWrapper) DisplayName 

func (t *BPETokenizerWrapper) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*BPETokenizerWrapper) IsExact 

func (t *BPETokenizerWrapper) IsExact() bool

IsExact returns true for BPE tokenizers.

func (*BPETokenizerWrapper) Name 

func (t *BPETokenizerWrapper) Name() string

Name returns the machine-readable tokenizer identifier.

type ClaudeApproximator 

type ClaudeApproximator struct{}

ClaudeApproximator provides approximation for Claude models.

func (*ClaudeApproximator) Contract added in v0.5.1 

func (c *ClaudeApproximator) Contract() ContractIdentity

Contract identifies the cached character/word approximation family. The ratio is intentionally not part of this identity: cached character and word totals are reducible primitives from which current ratios are derived.

func (*ClaudeApproximator) CountTokens 

func (c *ClaudeApproximator) CountTokens(text string) (int, error)

CountTokens approximates token count for Claude.

func (*ClaudeApproximator) DisplayName 

func (c *ClaudeApproximator) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*ClaudeApproximator) IsExact 

func (c *ClaudeApproximator) IsExact() bool

IsExact returns false for approximations.

func (*ClaudeApproximator) Name 

func (c *ClaudeApproximator) Name() string

Name returns the machine-readable tokenizer identifier.

type ContractIdentity added in v0.5.1 

type ContractIdentity struct {
	Method              string
	Encoding            string
	Implementation      string
	VocabularyDigest    [sha256.Size]byte
	NormalizationPolicy string
	SpecialTokenPolicy  string
}

ContractIdentity is the stable compatibility identity for a tokenizer's count semantics. Model names, display names, and context windows are deliberately excluded because they describe presentation rather than the bytes produced by the tokenizer.

func ContractOf added in v0.5.1 

func ContractOf(tokenizer Tokenizer) (ContractIdentity, bool)

ContractOf returns a tokenizer's stable cache identity when it exposes one.

func (ContractIdentity) Valid added in v0.5.1 

func (identity ContractIdentity) Valid() bool

Valid reports whether the identity has the fields required for safe cache reuse. A zero vocabulary digest is valid for embedded tokenizers; external vocabulary owners must populate it from their bytes.

type ContractIdentityProvider added in v0.5.1 

type ContractIdentityProvider interface {
	Contract() ContractIdentity
}

ContractIdentityProvider marks tokenizers whose results may participate in cache reuse. Custom Tokenizer implementations that do not provide this identity remain countable but are not cache-compatible by default.

type CountFilesOptions added in v0.5.1 

type CountFilesOptions struct {
	Model      string
	All        bool
	OnProgress ProgressFunc
}

CountFilesOptions configures a multi-file count, including optional progress.

type CountResult 

type CountResult struct {
	FilePath    string         `json:"file_path"`
	IsDirectory bool           `json:"is_directory,omitempty"`
	FileCount   int            `json:"file_count,omitempty"`
	FileSize    int            `json:"file_size"`
	Characters  int            `json:"characters"`
	Words       int            `json:"words"`
	Lines       int            `json:"lines"`
	Methods     []MethodResult `json:"methods"`
}

CountResult represents the result of token counting.

type Counter 

type Counter struct {
	// contains filtered or unexported fields
}

Counter handles token counting.

func NewCounter 

func NewCounter(opts CounterOptions) (*Counter, error)

NewCounter creates a new token counter. Returns an error if the BPE tokenizers fail to initialize.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.Count(ctx, "Hello, world!", "gpt-4o", false)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, m := range result.Methods {
		if m.IsExact {
			fmt.Printf("Tokens: %d (exact)\n", m.Tokens)
		}
	}
}

Output:
Tokens: 4 (exact)

func (*Counter) Count 

func (c *Counter) Count(ctx context.Context, text string, model string, all bool) (*CountResult, error)

Count performs token counting using specified methods.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.Count(ctx, "The quick brown fox jumps over the lazy dog.", "", true)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Printf("Characters: %d\n", result.Characters)
	fmt.Printf("Words: %d\n", result.Words)
	fmt.Printf("Methods: %d\n", len(result.Methods))
}

Output:
Characters: 44
Words: 9
Methods: 7

func (*Counter) CountDirectory 

func (c *Counter) CountDirectory(ctx context.Context, path string, model string, all bool) (*CountResult, error)

CountDirectory counts tokens across all text files in a directory. It walks the directory respecting .gitignore rules and skipping binary files, then counts each file individually via CountFiles, so peak memory tracks the largest file rather than the whole tree.

func (*Counter) CountFile 

func (c *Counter) CountFile(ctx context.Context, path string, model string, all bool) (*CountResult, error)

CountFile counts tokens in a single file. It checks for context cancellation, rejects binary files, reads the file content, and delegates to Count. The result includes FilePath and FileSize.

Example 
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	f, err := os.CreateTemp("", "tcount-example-*.txt")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	defer func() { _ = os.Remove(f.Name()) }()

	if _, err := f.WriteString("Hello, world!"); err != nil {
		fmt.Println("error:", err)
		return
	}
	if err := f.Close(); err != nil {
		fmt.Println("error:", err)
		return
	}

	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.CountFile(ctx, f.Name(), "gpt-4o", false)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, m := range result.Methods {
		if m.IsExact {
			fmt.Printf("Tokens: %d\n", m.Tokens)
		}
	}
}

Output:
Tokens: 4
Example (Error) 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	_, err = counter.CountFile(ctx, "nonexistent.txt", "gpt-4o", false)
	if err != nil {
		fmt.Println("File not found (expected)")
	}
}

Output:
File not found (expected)

func (*Counter) CountFiles added in v0.5.0 

func (c *Counter) CountFiles(ctx context.Context, files []string, model string, all bool) (*CountResult, error)

CountFiles counts tokens across the given text files. Each file is read exactly once, counted, and released, so peak memory tracks the largest files in flight rather than the combined corpus. Token counts and word/line statistics are computed per file and summed: tokens never merge across file boundaries, and word counts stay correct when a file lacks a trailing newline. Files are processed on a bounded worker pool; sums are order-independent so results are deterministic.

CountFiles is equivalent to CountFilesWithOptions with a nil OnProgress.

func (*Counter) CountFilesWithCache added in v0.5.1 

func (c *Counter) CountFilesWithCache(ctx context.Context, root string, files []string, model string, all bool, store cache.Store, mode cache.ValidationMode) (*CountResult, error)

CountFilesWithCache counts the current directory membership while reusing valid per-file values from store. The files argument must come from the caller's successful current walk; the manifest never supplies membership. The cache is deliberately an explicit caller choice; CountFiles remains the cold-path oracle.

CountFilesWithCache is equivalent to CountFilesWithCacheOptions with a nil OnProgress.

func (*Counter) CountFilesWithCacheOptions added in v0.5.1 

func (c *Counter) CountFilesWithCacheOptions(ctx context.Context, root string, files []string, opts CountFilesOptions, store cache.Store, mode cache.ValidationMode) (*CountResult, error)

CountFilesWithCacheOptions is CountFilesWithCache with optional progress reporting. Cache hits are reported as completed files before cold misses are counted so the progress total advances through a warm tree without stalling.

func (*Counter) CountFilesWithOptions added in v0.5.1 

func (c *Counter) CountFilesWithOptions(ctx context.Context, files []string, opts CountFilesOptions) (*CountResult, error)

CountFilesWithOptions is CountFiles with optional progress reporting.

type CounterOptions 

type CounterOptions struct {
	CharsPerToken float64
	WordsPerToken float64
	VocabFile     string
	Provider      Provider
	Stats         *Stats
}

CounterOptions configures the counter.

type GeminiApproximator added in v0.4.0 

type GeminiApproximator struct{}

GeminiApproximator provides approximation for Google Gemini models. Gemini uses its own SentencePiece tokenizer; for exact counts supply the vocab file via --vocab-file. Without it, this character-based estimate applies.

func (*GeminiApproximator) Contract added in v0.5.1 

func (g *GeminiApproximator) Contract() ContractIdentity

Contract identifies the cached character/word approximation family.

func (*GeminiApproximator) CountTokens added in v0.4.0 

func (g *GeminiApproximator) CountTokens(text string) (int, error)

CountTokens approximates token count for Gemini.

func (*GeminiApproximator) DisplayName added in v0.4.0 

func (g *GeminiApproximator) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*GeminiApproximator) IsExact added in v0.4.0 

func (g *GeminiApproximator) IsExact() bool

IsExact returns false for approximations.

func (*GeminiApproximator) Name added in v0.4.0 

func (g *GeminiApproximator) Name() string

Name returns the machine-readable tokenizer identifier.

type MethodResult 

type MethodResult struct {
	Name          string `json:"name"`
	DisplayName   string `json:"display_name"`
	Tokens        int    `json:"tokens"`
	IsExact       bool   `json:"is_exact"`
	ContextWindow int    `json:"context_window,omitempty"`
}

MethodResult represents token count for a specific method.

type ModelMetadata 

type ModelMetadata struct {
	Name          string   // Model identifier (e.g., "gpt-4o", "claude-sonnet-4.6")
	Provider      Provider // Provider who created the model
	Encoding      string   // BPE encoding name (e.g., "o200k_base", "cl100k_base")
	ContextWindow int      // Maximum context window size in tokens
}

ModelMetadata contains comprehensive information about an LLM model.

func GetModelMetadata deprecated 

func GetModelMetadata(modelName string) *ModelMetadata

GetModelMetadata retrieves metadata for a given model name.

Deprecated: use LookupModel.

Example 
package main

import (
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	meta := tokenizer.GetModelMetadata("gpt-4o")
	if meta != nil {
		fmt.Printf("Model: %s\n", meta.Name)
		fmt.Printf("Provider: %s\n", meta.Provider)
		fmt.Printf("Encoding: %s\n", meta.Encoding)
		fmt.Printf("Context: %d\n", meta.ContextWindow)
	}
}

Output:
Model: gpt-4o
Provider: openai
Encoding: o200k_base
Context: 128000

func ListModelsByProvider 

func ListModelsByProvider(provider Provider) []ModelMetadata

ListModelsByProvider returns all models from a specific provider, sorted by name.

func LookupModel added in v0.5.0 

func LookupModel(modelName string) *ModelMetadata

LookupModel retrieves metadata for a given model name. Returns nil if model is not found in the registry.

type ProgressFunc added in v0.5.1 

type ProgressFunc func(ProgressUpdate)

ProgressFunc receives per-file progress facts. Nil means no reporting.

type ProgressUpdate added in v0.5.1 

type ProgressUpdate struct {
	FilesTotal   int
	FilesDone    int
	LastPath     string
	Bytes        int64
	Characters   int
	Words        int
	Lines        int
	MethodTokens []int // running sums aligned with planned methods for this run
}

ProgressUpdate is a pure fact snapshot after one or more files complete. The CLI owns elapsed time, spinner, labels, and paint rate.

type Provider 

type Provider string

Provider represents an LLM provider. 

const (
	ProviderOpenAI    Provider = "openai"    // OpenAI (GPT, o-series)
	ProviderAnthropic Provider = "anthropic" // Anthropic (Claude)
	ProviderMeta      Provider = "meta"      // Meta (Llama)
	ProviderDeepSeek  Provider = "deepseek"  // DeepSeek
	ProviderAlibaba   Provider = "alibaba"   // Alibaba (Qwen)
	ProviderMicrosoft Provider = "microsoft" // Microsoft (Phi)
	ProviderGoogle    Provider = "google"    // Google (Gemini)
)

func GetProviderForModel deprecated 

func GetProviderForModel(modelName string) Provider

GetProviderForModel returns the provider for a given model name.

Deprecated: use ProviderForModel.

func ProviderForModel added in v0.5.0 

func ProviderForModel(modelName string) Provider

ProviderForModel returns the provider for a given model name. Returns empty string if model is not registered.

type SPMTokenizerWrapper 

type SPMTokenizerWrapper struct {
	// contains filtered or unexported fields
}

SPMTokenizerWrapper uses a .model vocab file for exact tokenization.

func (*SPMTokenizerWrapper) Contract added in v0.5.1 

func (t *SPMTokenizerWrapper) Contract() ContractIdentity

Contract includes the vocabulary content digest rather than its path or filesystem metadata. Replacing bytes at the same path therefore invalidates only the affected SentencePiece results.

func (*SPMTokenizerWrapper) CountTokens 

func (t *SPMTokenizerWrapper) CountTokens(text string) (int, error)

CountTokens returns the token count using the SentencePiece model.

func (*SPMTokenizerWrapper) DisplayName 

func (t *SPMTokenizerWrapper) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*SPMTokenizerWrapper) IsExact 

func (t *SPMTokenizerWrapper) IsExact() bool

IsExact returns true because SentencePiece provides exact token counts.

func (*SPMTokenizerWrapper) Name 

func (t *SPMTokenizerWrapper) Name() string

Name returns the machine-readable tokenizer identifier.

type Stats added in v0.5.1 

type Stats struct {
	// contains filtered or unexported fields
}

Stats collects optional, benchmark-oriented measurements for a count run. A nil *Stats disables instrumentation and preserves the normal counting path.

func NewStats added in v0.5.1 

func NewStats() *Stats

NewStats creates an enabled instrumentation collector.

func (*Stats) ObserveMemory added in v0.5.1 

func (s *Stats) ObserveMemory()

func (*Stats) RecordAggregationDuration added in v0.5.1 

func (s *Stats) RecordAggregationDuration(duration time.Duration)

func (*Stats) RecordBinarySniffBytes added in v0.5.1 

func (s *Stats) RecordBinarySniffBytes(bytes int64)

func (*Stats) RecordBinarySniffOpen added in v0.5.1 

func (s *Stats) RecordBinarySniffOpen()

func (*Stats) RecordCacheBytesReused added in v0.5.1 

func (s *Stats) RecordCacheBytesReused(bytes int64)

RecordCacheBytesReused records the source bytes represented by reusable cached file results. In verified mode those bytes may also be read for digest validation; the separate full-read counter reports that I/O.

func (*Stats) RecordCacheHit added in v0.5.1 

func (s *Stats) RecordCacheHit(reason string, methods int)

func (*Stats) RecordCacheMiss added in v0.5.1 

func (s *Stats) RecordCacheMiss(reason string)

func (*Stats) RecordCachePartialHit added in v0.5.1 

func (s *Stats) RecordCachePartialHit(reason string, reusableMethods, missingMethods int)

func (*Stats) RecordCacheWarning added in v0.5.1 

func (s *Stats) RecordCacheWarning()

func (*Stats) RecordEligibleFile added in v0.5.1 

func (s *Stats) RecordEligibleFile()

func (*Stats) RecordEntryVisited added in v0.5.1 

func (s *Stats) RecordEntryVisited()

func (*Stats) RecordFullFileBytes added in v0.5.1 

func (s *Stats) RecordFullFileBytes(bytes int64)

func (*Stats) RecordFullFileOpen added in v0.5.1 

func (s *Stats) RecordFullFileOpen()

func (*Stats) RecordPersistenceReadyDuration added in v0.5.1 

func (s *Stats) RecordPersistenceReadyDuration(duration time.Duration)

func (*Stats) RecordTokenizationDuration added in v0.5.1 

func (s *Stats) RecordTokenizationDuration(duration time.Duration)

func (*Stats) RecordTokenizedFile added in v0.5.1 

func (s *Stats) RecordTokenizedFile(method string)

func (*Stats) RecordValidationReadDuration added in v0.5.1 

func (s *Stats) RecordValidationReadDuration(duration time.Duration)

func (*Stats) RecordWalkDuration added in v0.5.1 

func (s *Stats) RecordWalkDuration(duration time.Duration)

func (*Stats) Snapshot added in v0.5.1 

func (s *Stats) Snapshot() StatsSnapshot

Snapshot returns a race-free copy of the measurements collected so far.

type StatsSnapshot added in v0.5.1 

type StatsSnapshot struct {
	EntriesVisited           int64
	EligibleFiles            int64
	BinarySniffOpens         int64
	BinarySniffBytes         int64
	FullFileOpens            int64
	FullFileBytes            int64
	FilesTokenizedByMethod   map[string]int64
	WalkDuration             time.Duration
	ValidationReadDuration   time.Duration
	TokenizationDuration     time.Duration
	AggregationDuration      time.Duration
	PersistenceReadyDuration time.Duration
	PeakHeapAllocBytes       uint64
	CacheHits                int64
	CachePartialHits         int64
	CacheMisses              int64
	CacheMethodsAvoided      int64
	CacheBytesReused         int64
	CacheWarnings            int64
	CacheReasons             map[string]int64
}

StatsSnapshot is the immutable view of one Stats collector.

type Tokenizer 

type Tokenizer interface {
	// CountTokens returns the token count for the given text.
	CountTokens(text string) (int, error)

	// Name returns the tokenizer's machine-readable identifier.
	Name() string

	// DisplayName returns the tokenizer's human-readable name.
	DisplayName() string

	// IsExact returns true if this tokenizer produces exact counts
	// (as opposed to approximations).
	IsExact() bool
}

Tokenizer counts tokens in text using a specific tokenization method.

func NewBPETokenizer 

func NewBPETokenizer(model string) (Tokenizer, error)

NewBPETokenizer creates an exact tokenizer for the given model name. Supports OpenAI models (gpt-4o, gpt-5, o3, o4-mini, etc.) and open-source models that use BPE-compatible encodings.

Example 
package main

import (
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	tok, err := tokenizer.NewBPETokenizer("gpt-4o")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	count, err := tok.CountTokens("Hello, world!")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Printf("Tokens: %d\n", count)
	fmt.Printf("Exact: %v\n", tok.IsExact())
}

Output:
Tokens: 4
Exact: true

func NewBPETokenizerByEncoding 

func NewBPETokenizerByEncoding(encodingName string) (Tokenizer, error)

NewBPETokenizerByEncoding creates a tokenizer for a specific BPE encoding. Supported encodings: o200k_base, cl100k_base, p50k_base, r50k_base.

func NewClaudeApproximator 

func NewClaudeApproximator() Tokenizer

NewClaudeApproximator creates a character-based approximator tuned for Claude models. Uses a 3.8 characters per token ratio.

func NewGeminiApproximator added in v0.4.0 

func NewGeminiApproximator() Tokenizer

NewGeminiApproximator creates a character-based approximator tuned for Gemini models. Uses a 4.0 characters per token ratio.

func NewSPMTokenizer 

func NewSPMTokenizer(modelPath string) (Tokenizer, error)

NewSPMTokenizer creates a SentencePiece tokenizer from a .model vocab file. Supports Llama, Mistral, Gemma, and other SPM-based models.

Source Files

View all Source files

Directories

Path Synopsis
Package bpe implements Byte Pair Encoding tokenization.
 Package bpe implements Byte Pair Encoding tokenization.
Package fileops provides file system operations for token counting, including directory traversal with .gitignore support and binary detection.
 Package fileops provides file system operations for token counting, including directory traversal with .gitignore support and binary detection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.