tokenizer

package
v0.5.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 12, 2026 License: MIT Imports: 14 Imported by: 0

Documentation

Overview

Package tokenizer provides token counting for LLM models.

It supports exact BPE tokenization for OpenAI models, character-based approximation for Claude and Gemini models, and SentencePiece tokenization for open-source models like Llama and Mistral.

Index

Examples

Constants

View Source 
const (
	EncodingO200kBase    = bpe.EncodingO200kBase
	EncodingCL100kBase   = bpe.EncodingCL100kBase
	EncodingClaudeApprox = "claude_approx"
	EncodingGeminiApprox = "gemini_approx"
	EncodingSPM          = "spm"
)

Encoding identifiers shared across the tokenizer and CLI layers.

View Source 
const (
	DefaultCharsPerToken = 4.0
	DefaultWordsPerToken = 0.75
)

Default approximation ratios applied when CounterOptions leaves them zero.

View Source 
const NameClaudeApprox = "claude_3_approx"

NameClaudeApprox is the machine-readable identifier the Claude approximator reports; consumers key accuracy labeling off it.

Variables

View Source 
var (
	// ErrModelNotFound is returned when a requested model is not in the registry.
	ErrModelNotFound = errors.New("model not found")

	// ErrEncodingNotFound is returned when a BPE encoding name is not recognized.
	ErrEncodingNotFound = errors.New("encoding not found")

	// ErrVocabFileRequired is returned when a SentencePiece model path is empty.
	ErrVocabFileRequired = errors.New("vocab file path is required")

	// ErrBinaryFile is returned when attempting to count tokens in a binary file.
	ErrBinaryFile = errors.New("file is binary")
)

Sentinel errors for common failure modes.

Functions

func IsOpenSourceModel 

func IsOpenSourceModel(modelName string) bool

IsOpenSourceModel returns true if the model is from an open-source provider (not OpenAI, Anthropic, or Google).

func ListModels 

func ListModels() []string

ListModels returns all registered model names in sorted order.

func ModelsByEncoding 

func ModelsByEncoding() map[string][]string

ModelsByEncoding returns a map of encoding name to sorted model names.

Types

type BPETokenizerWrapper 

type BPETokenizerWrapper struct {
	// contains filtered or unexported fields
}

BPETokenizerWrapper implements exact tokenization using a BPE encoding.

func (*BPETokenizerWrapper) CountTokens 

func (t *BPETokenizerWrapper) CountTokens(text string) (int, error)

CountTokens counts tokens using BPE tokenization. Counting never allows special tokens, so it takes the ordinary encode path, which skips the special-token scan and produces identical counts.

func (*BPETokenizerWrapper) DisplayName 

func (t *BPETokenizerWrapper) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*BPETokenizerWrapper) IsExact 

func (t *BPETokenizerWrapper) IsExact() bool

IsExact returns true for BPE tokenizers.

func (*BPETokenizerWrapper) Name 

func (t *BPETokenizerWrapper) Name() string

Name returns the machine-readable tokenizer identifier.

type ClaudeApproximator 

type ClaudeApproximator struct{}

ClaudeApproximator provides approximation for Claude models.

func (*ClaudeApproximator) CountTokens 

func (c *ClaudeApproximator) CountTokens(text string) (int, error)

CountTokens approximates token count for Claude.

func (*ClaudeApproximator) DisplayName 

func (c *ClaudeApproximator) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*ClaudeApproximator) IsExact 

func (c *ClaudeApproximator) IsExact() bool

IsExact returns false for approximations.

func (*ClaudeApproximator) Name 

func (c *ClaudeApproximator) Name() string

Name returns the machine-readable tokenizer identifier.

type CountResult 

type CountResult struct {
	FilePath    string         `json:"file_path"`
	IsDirectory bool           `json:"is_directory,omitempty"`
	FileCount   int            `json:"file_count,omitempty"`
	FileSize    int            `json:"file_size"`
	Characters  int            `json:"characters"`
	Words       int            `json:"words"`
	Lines       int            `json:"lines"`
	Methods     []MethodResult `json:"methods"`
}

CountResult represents the result of token counting.

type Counter 

type Counter struct {
	// contains filtered or unexported fields
}

Counter handles token counting.

func NewCounter 

func NewCounter(opts CounterOptions) (*Counter, error)

NewCounter creates a new token counter. Returns an error if the BPE tokenizers fail to initialize.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.Count(ctx, "Hello, world!", "gpt-4o", false)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, m := range result.Methods {
		if m.IsExact {
			fmt.Printf("Tokens: %d (exact)\n", m.Tokens)
		}
	}
}

Output:
Tokens: 4 (exact)

func (*Counter) Count 

func (c *Counter) Count(ctx context.Context, text string, model string, all bool) (*CountResult, error)

Count performs token counting using specified methods.

Example 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.Count(ctx, "The quick brown fox jumps over the lazy dog.", "", true)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Printf("Characters: %d\n", result.Characters)
	fmt.Printf("Words: %d\n", result.Words)
	fmt.Printf("Methods: %d\n", len(result.Methods))
}

Output:
Characters: 44
Words: 9
Methods: 7

func (*Counter) CountDirectory 

func (c *Counter) CountDirectory(ctx context.Context, path string, model string, all bool) (*CountResult, error)

CountDirectory counts tokens across all text files in a directory. It walks the directory respecting .gitignore rules and skipping binary files, then counts each file individually via CountFiles, so peak memory tracks the largest file rather than the whole tree.

func (*Counter) CountFile 

func (c *Counter) CountFile(ctx context.Context, path string, model string, all bool) (*CountResult, error)

CountFile counts tokens in a single file. It checks for context cancellation, rejects binary files, reads the file content, and delegates to Count. The result includes FilePath and FileSize.

Example 
package main

import (
	"context"
	"fmt"
	"os"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	f, err := os.CreateTemp("", "tcount-example-*.txt")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	defer func() { _ = os.Remove(f.Name()) }()

	if _, err := f.WriteString("Hello, world!"); err != nil {
		fmt.Println("error:", err)
		return
	}
	if err := f.Close(); err != nil {
		fmt.Println("error:", err)
		return
	}

	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	result, err := counter.CountFile(ctx, f.Name(), "gpt-4o", false)
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	for _, m := range result.Methods {
		if m.IsExact {
			fmt.Printf("Tokens: %d\n", m.Tokens)
		}
	}
}

Output:
Tokens: 4
Example (Error) 
package main

import (
	"context"
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	ctx := context.Background()
	_, err = counter.CountFile(ctx, "nonexistent.txt", "gpt-4o", false)
	if err != nil {
		fmt.Println("File not found (expected)")
	}
}

Output:
File not found (expected)

func (*Counter) CountFiles added in v0.5.0 

func (c *Counter) CountFiles(ctx context.Context, files []string, model string, all bool) (*CountResult, error)

CountFiles counts tokens across the given text files. Each file is read exactly once, counted, and released, so peak memory tracks the largest files in flight rather than the combined corpus. Token counts and word/line statistics are computed per file and summed: tokens never merge across file boundaries, and word counts stay correct when a file lacks a trailing newline. Files are processed on a bounded worker pool; sums are order-independent so results are deterministic.

type CounterOptions 

type CounterOptions struct {
	CharsPerToken float64
	WordsPerToken float64
	VocabFile     string
	Provider      Provider
}

CounterOptions configures the counter.

type GeminiApproximator added in v0.4.0 

type GeminiApproximator struct{}

GeminiApproximator provides approximation for Google Gemini models. Gemini uses its own SentencePiece tokenizer; for exact counts supply the vocab file via --vocab-file. Without it, this character-based estimate applies.

func (*GeminiApproximator) CountTokens added in v0.4.0 

func (g *GeminiApproximator) CountTokens(text string) (int, error)

CountTokens approximates token count for Gemini.

func (*GeminiApproximator) DisplayName added in v0.4.0 

func (g *GeminiApproximator) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*GeminiApproximator) IsExact added in v0.4.0 

func (g *GeminiApproximator) IsExact() bool

IsExact returns false for approximations.

func (*GeminiApproximator) Name added in v0.4.0 

func (g *GeminiApproximator) Name() string

Name returns the machine-readable tokenizer identifier.

type MethodResult 

type MethodResult struct {
	Name          string `json:"name"`
	DisplayName   string `json:"display_name"`
	Tokens        int    `json:"tokens"`
	IsExact       bool   `json:"is_exact"`
	ContextWindow int    `json:"context_window,omitempty"`
}

MethodResult represents token count for a specific method.

type ModelMetadata 

type ModelMetadata struct {
	Name          string   // Model identifier (e.g., "gpt-4o", "claude-sonnet-4.6")
	Provider      Provider // Provider who created the model
	Encoding      string   // BPE encoding name (e.g., "o200k_base", "cl100k_base")
	ContextWindow int      // Maximum context window size in tokens
}

ModelMetadata contains comprehensive information about an LLM model.

func GetModelMetadata deprecated 

func GetModelMetadata(modelName string) *ModelMetadata

GetModelMetadata retrieves metadata for a given model name.

Deprecated: use LookupModel.

Example 
package main

import (
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	meta := tokenizer.GetModelMetadata("gpt-4o")
	if meta != nil {
		fmt.Printf("Model: %s\n", meta.Name)
		fmt.Printf("Provider: %s\n", meta.Provider)
		fmt.Printf("Encoding: %s\n", meta.Encoding)
		fmt.Printf("Context: %d\n", meta.ContextWindow)
	}
}

Output:
Model: gpt-4o
Provider: openai
Encoding: o200k_base
Context: 128000

func ListModelsByProvider 

func ListModelsByProvider(provider Provider) []ModelMetadata

ListModelsByProvider returns all models from a specific provider, sorted by name.

func LookupModel added in v0.5.0 

func LookupModel(modelName string) *ModelMetadata

LookupModel retrieves metadata for a given model name. Returns nil if model is not found in the registry.

type Provider 

type Provider string

Provider represents an LLM provider. 

const (
	ProviderOpenAI    Provider = "openai"    // OpenAI (GPT, o-series)
	ProviderAnthropic Provider = "anthropic" // Anthropic (Claude)
	ProviderMeta      Provider = "meta"      // Meta (Llama)
	ProviderDeepSeek  Provider = "deepseek"  // DeepSeek
	ProviderAlibaba   Provider = "alibaba"   // Alibaba (Qwen)
	ProviderMicrosoft Provider = "microsoft" // Microsoft (Phi)
	ProviderGoogle    Provider = "google"    // Google (Gemini)
)

func GetProviderForModel deprecated 

func GetProviderForModel(modelName string) Provider

GetProviderForModel returns the provider for a given model name.

Deprecated: use ProviderForModel.

func ProviderForModel added in v0.5.0 

func ProviderForModel(modelName string) Provider

ProviderForModel returns the provider for a given model name. Returns empty string if model is not registered.

type SPMTokenizerWrapper 

type SPMTokenizerWrapper struct {
	// contains filtered or unexported fields
}

SPMTokenizerWrapper uses a .model vocab file for exact tokenization.

func (*SPMTokenizerWrapper) CountTokens 

func (t *SPMTokenizerWrapper) CountTokens(text string) (int, error)

CountTokens returns the token count using the SentencePiece model.

func (*SPMTokenizerWrapper) DisplayName 

func (t *SPMTokenizerWrapper) DisplayName() string

DisplayName returns the human-readable tokenizer name.

func (*SPMTokenizerWrapper) IsExact 

func (t *SPMTokenizerWrapper) IsExact() bool

IsExact returns true because SentencePiece provides exact token counts.

func (*SPMTokenizerWrapper) Name 

func (t *SPMTokenizerWrapper) Name() string

Name returns the machine-readable tokenizer identifier.

type Tokenizer 

type Tokenizer interface {
	// CountTokens returns the token count for the given text.
	CountTokens(text string) (int, error)

	// Name returns the tokenizer's machine-readable identifier.
	Name() string

	// DisplayName returns the tokenizer's human-readable name.
	DisplayName() string

	// IsExact returns true if this tokenizer produces exact counts
	// (as opposed to approximations).
	IsExact() bool
}

Tokenizer counts tokens in text using a specific tokenization method.

func NewBPETokenizer 

func NewBPETokenizer(model string) (Tokenizer, error)

NewBPETokenizer creates an exact tokenizer for the given model name. Supports OpenAI models (gpt-4o, gpt-5, o3, o4-mini, etc.) and open-source models that use BPE-compatible encodings.

Example 
package main

import (
	"fmt"

	"github.com/lancekrogers/tcount/tokenizer"
)

func main() {
	tok, err := tokenizer.NewBPETokenizer("gpt-4o")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	count, err := tok.CountTokens("Hello, world!")
	if err != nil {
		fmt.Println("error:", err)
		return
	}
	fmt.Printf("Tokens: %d\n", count)
	fmt.Printf("Exact: %v\n", tok.IsExact())
}

Output:
Tokens: 4
Exact: true

func NewBPETokenizerByEncoding 

func NewBPETokenizerByEncoding(encodingName string) (Tokenizer, error)

NewBPETokenizerByEncoding creates a tokenizer for a specific BPE encoding. Supported encodings: o200k_base, cl100k_base, p50k_base, r50k_base.

func NewClaudeApproximator 

func NewClaudeApproximator() Tokenizer

NewClaudeApproximator creates a character-based approximator tuned for Claude models. Uses a 3.8 characters per token ratio.

func NewGeminiApproximator added in v0.4.0 

func NewGeminiApproximator() Tokenizer

NewGeminiApproximator creates a character-based approximator tuned for Gemini models. Uses a 4.0 characters per token ratio.

func NewSPMTokenizer 

func NewSPMTokenizer(modelPath string) (Tokenizer, error)

NewSPMTokenizer creates a SentencePiece tokenizer from a .model vocab file. Supports Llama, Mistral, Gemma, and other SPM-based models.

Source Files

View all Source files

Directories

Path Synopsis
Package bpe implements Byte Pair Encoding tokenization.
 Package bpe implements Byte Pair Encoding tokenization.
Package fileops provides file system operations for token counting, including directory traversal with .gitignore support and binary detection.
 Package fileops provides file system operations for token counting, including directory traversal with .gitignore support and binary detection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.