tcount

README

tcount

A fast token counter for LLM workflows that runs entirely on your machine: no API keys, no network calls, nothing leaves your disk. Count with exact OpenAI tokenizers, Claude and Gemini approximations, SentencePiece vocabularies, and generic estimation, from a single CLI or as a Go library.

Single file — method table, --model context usage:

Directory — live progress while a large tree is counted, then the final report:

Features

• 100% local — counting never touches the network, so it is safe for private codebases, unreleased prompts, and air-gapped machines
• CLI and Go library — the same engine ships as a command and as an importable package (Library Usage)
• Exact BPE tokenization — embedded vocabularies for GPT-5, GPT-4.1, GPT-4o, o-series, and legacy GPT-4/3.5
• Claude approximation calibrated for Anthropic models
• Gemini approximation for Google Gemini models
• SentencePiece exact tokenization for Llama and other open-source models (bring your own .model file)
• Context window usage — see what percentage of a model's context you're consuming (shown when you pass --model)
• Provider filtering — compare models from a specific provider
• Directory scanning — .gitignore-aware, skips binaries, counts files in parallel with memory bounded by your largest file
• Live counting progress — on a TTY, recursive counts show a spinner and rising file/char totals while work runs (disable with --no-progress; never shown for --json)
• JSON output for scripting and pipelines
• Experimental directory cache — opt-in reuse for repeated recursive counts, with status, clear, verification, and bypass controls

Install

npm / pnpm / bun (macOS & Linux)

npm install -g @obedience-corp/tcount
# or
pnpm add -g @obedience-corp/tcount
# or
bun add -g @obedience-corp/tcount

The npm package downloads the official release binary for your platform (with checksum verification) on first install.

Homebrew (macOS & Linux)

brew install lancekrogers/tap/tcount

Go

go install github.com/lancekrogers/tcount/cmd/tcount@latest

From source

git clone https://github.com/lancekrogers/tcount.git
cd tcount
go build -o bin/tcount ./cmd/tcount

Binary releases

Pre-built binaries for macOS, Linux, and Windows are available on the releases page.

Quick Start

# Count tokens in a file
tcount myfile.txt

# Specific model (shows context-window usage)
tcount --model gpt-5 prompt.md

# All counting methods
tcount --all prompt.md

# Filter by provider
tcount --provider openai prompt.md

# Count an entire directory tree (TTY shows live progress while counting)
tcount -d ./src

# Same count without the progress frame
tcount -d --no-progress ./src

# JSON output for scripting
tcount --json document.md

# Opt-in cache for repeated directory counts
tcount -d --cache ./src

Or import the same engine in Go:

import "github.com/lancekrogers/tcount/tokenizer"

counter, _ := tokenizer.NewCounter(tokenizer.CounterOptions{})
result, _ := counter.CountFile(ctx, "prompt.md", "gpt-5", false)

Full API in Library Usage.

Supported Models

OpenAI

Model	Encoding	Context
gpt-5, gpt-5-mini, gpt-5-nano	o200k_base	400K
gpt-5.1, gpt-5.2	o200k_base	400K
gpt-4.1, gpt-4.1-mini, gpt-4.1-nano	o200k_base	1M
gpt-4o, gpt-4o-mini	o200k_base	128K
o3, o3-mini, o4-mini	o200k_base	200K
gpt-4, gpt-4-turbo	cl100k_base	8K–128K
gpt-3.5-turbo	cl100k_base	16K

Anthropic

Model	Method	Context
claude-opus-4.6, claude-opus-4.5	Approximation	1M
claude-opus-4.1, claude-opus-4	Approximation	200K
claude-sonnet-4.6, claude-sonnet-4.5	Approximation	1M
claude-sonnet-4	Approximation	200K
claude-haiku-4.5, claude-haiku-3.5, claude-haiku-3	Approximation	200K
claude-opus-3 (deprecated)	Approximation	200K

Google

Model	Method	Context
gemini-2.5-pro, gemini-2.5-flash, gemini-2.5-flash-lite	Approximation	1M

Gemini uses its own SentencePiece tokenizer. Without a --vocab-file, tcount approximates at ~4 characters per token.

Meta (Llama)

Model	Method	Context
llama-4-scout	tiktoken approx / SentencePiece	10M
llama-4-maverick	tiktoken approx / SentencePiece	1M
llama-3.1-8b, llama-3.1-70b, llama-3.1-405b	tiktoken approx / SentencePiece	128K

DeepSeek

Model	Method	Context
deepseek-v2, deepseek-v3, deepseek-coder-v2	tiktoken approx	128K

Alibaba (Qwen)

Model	Method	Context
qwen-2.5-7b, qwen-2.5-14b, qwen-2.5-72b	tiktoken approx	128K
qwen-3-72b	tiktoken approx	128K

Microsoft (Phi)

Model	Method	Context
phi-3-mini, phi-3-small, phi-3-medium	tiktoken approx	128K

Tokenization Methods

Method	Accuracy	When Used
tiktoken (o200k_base)	Exact	GPT-5.x, GPT-4.1, GPT-4o, o3, o4-mini
tiktoken (cl100k_base)	Exact	GPT-4, GPT-3.5
Claude approximation	Estimated	All Claude models (÷3.8 char ratio)
Gemini approximation	Approximate	All Gemini models (÷4.0 char ratio)
SentencePiece	Exact	Llama with --vocab-file
tiktoken approximation	Approximate	Llama, DeepSeek, Qwen, Phi (no vocab file)
Character-based	Approximate	Any (chars ÷ configurable ratio, default 4.0)
Word-based	Approximate	Any (words × configurable multiplier, default 1.33)
Whitespace split	Approximate	Any (raw word count as lower bound)

Usage

tcount [file|directory] [flags]

Flags

Flag	Short	Description
--model		Specific model tokenizer (adds a context-usage column)
--models	-m	Show encoding-to-model lookup table
--provider		Filter by provider: openai, anthropic, google, meta, deepseek, alibaba, microsoft, all
--vocab-file		Path to SentencePiece .model file for exact Llama tokenization
--all		Show all counting methods
--json		JSON output
--recursive	-r	Recursively count files in a directory
--directory	-d	Alias for --recursive
--cache		Enable experimental persistent caching for recursive directory counts
--no-cache		Force a cold count without reading or writing cache state
--cache-verify		Hash file contents before reusing cache entries; requires --cache
--chars-per-token		Character/token ratio for approximation (default: 4.0)
--words-per-token		Words/token ratio for approximation (default: 0.75)
--verbose		Show additional details
--no-color		Disable color output

Examples

Single model

Passing --model shows the exact (or approximate) count for that model plus how much of its context window the text uses:

$ tcount --model gpt-5 tokenizer.go

Token Count Report for: tokenizer.go

Basic Statistics
  Characters: 8,050
  Words: 978
  Lines: 259

Token Counts by Method
╭────────────────────┬────────┬──────────┬───────────────╮
│       Method       │ Tokens │ Accuracy │ Context Usage │
├────────────────────┼────────┼──────────┼───────────────┤
│ o200k_base (gpt-5) │  1,976 │ Exact    │ 0.49% of 400K │
╰────────────────────┴────────┴──────────┴───────────────╯

Without --model, tcount shows every counting method side by side (see the demo above), and --all forces the full table even when a model is set.

SentencePiece for exact Llama tokenization

# Download tokenizer.model from HuggingFace (requires auth):
# https://huggingface.co/meta-llama/Llama-3.1-8B/blob/main/original/tokenizer.model

tcount --model llama-3.1-8b --vocab-file /path/to/tokenizer.model document.md

Without --vocab-file, Llama models use a tiktoken-based approximation.

Directory scanning

Point tcount at a project directory to count every text file in one pass. On a TTY, large trees show live progress (spinner, files done/total, rising stats) on stderr, then the usual report on stdout:

tcount -d ./src

# same count without the progress frame
tcount -d --no-progress ./src

# optional scan/cache diagnostics on stderr
tcount -d --verbose ./src

When scanning directories, tcount respects .gitignore rules, skips binary files and .git directories, counts each file individually on a bounded worker pool, and sums the results. Counting per file keeps memory proportional to the largest file rather than the whole tree, and tokens never merge across file boundaries (the sum matches counting each file on its own). Progress is omitted for --json, non-TTY stderr, and --no-progress.

Experimental directory cache

The directory cache is off by default. Opt in explicitly with --cache:

# First run counts and stores reusable per-file results.
tcount -d --cache ./src

# A later run reuses unchanged files. Use content verification when exactness
# must survive a timestamp-preserving rewrite.
tcount -d --cache --cache-verify ./src

# Inspect or remove state without running a count.
tcount cache status ./src
tcount cache status --json ./src
tcount cache clear ./src
tcount cache clear --all

# Force a cold run; this does not construct or access the cache store.
tcount -d --no-cache ./src

Cache files live under the platform user-cache directory in the namespaced tcount/v1/roots tree. Set TCOUNT_CACHE_DIR to choose a different parent; the same tcount/v1 namespace is appended there. The manifest stores the canonical root, relative file paths, file metadata, content digests for populated entries, aggregate text statistics, and tokenizer counts. The verified mode recomputes and compares those digests before reuse. It does not store source file contents, but paths and counts can still be sensitive.

Plain --cache uses metadata validation (relative path, size, and nanosecond modification time), so a rewrite that preserves all three can be a false hit. Use --cache-verify for SHA-256 content validation. Cache load, validation, or persistence failures do not replace a successful cold count; --verbose reports them on stderr. Explicit cache status and cache clear operations return failures to the caller. The feature is experimental and its on-disk format may change before it becomes a default.

See the directory-cache operations guide for the storage, privacy, exactness, and rollout details.

JSON output

$ tcount --json --model gpt-5 tokenizer.go
{
  "file_path": "tokenizer.go",
  "file_size": 8050,
  "characters": 8050,
  "words": 978,
  "lines": 259,
  "methods": [
    {
      "name": "bpe_gpt_5",
      "display_name": "o200k_base (gpt-5)",
      "tokens": 1976,
      "is_exact": true,
      "context_window": 400000
    }
  ]
}

# Extract a specific count
tcount --json myfile.txt | jq '.methods[] | select(.name == "bpe_gpt_5") | .tokens'

# Batch count all markdown files
for f in docs/*.md; do tcount --json "$f"; done | jq -s '.'

Library Usage

tcount can be used as a Go library in your own projects.

Installation

go get github.com/lancekrogers/tcount/tokenizer

Basic Token Counting

package main

import (
    "context"
    "fmt"
    "log"

    "github.com/lancekrogers/tcount/tokenizer"
)

func main() {
    counter, err := tokenizer.NewCounter(tokenizer.CounterOptions{})
    if err != nil {
        log.Fatal(err)
    }

    ctx := context.Background()
    result, err := counter.Count(ctx, "Hello, world!", "gpt-4o", false)
    if err != nil {
        log.Fatal(err)
    }

    for _, m := range result.Methods {
        if m.IsExact {
            fmt.Printf("Tokens: %d (exact, %s)\n", m.Tokens, m.DisplayName)
        }
    }
}

File and Directory Counting

ctx := context.Background()

// Count tokens in a single file
result, err := counter.CountFile(ctx, "document.md", "gpt-4o", false)

// Count tokens across a directory (respects .gitignore, skips binaries,
// counts files in parallel and sums the results)
result, err := counter.CountDirectory(ctx, "./src", "", true)
fmt.Printf("Files: %d, Tokens: %d\n", result.FileCount, result.Methods[0].Tokens)

// Count an explicit list of files with the same per-file summing
result, err := counter.CountFiles(ctx, []string{"a.md", "b.md"}, "", true)

Direct BPE Tokenizer Access

tok, err := tokenizer.NewBPETokenizer("gpt-4o")
if err != nil {
    log.Fatal(err)
}

count, _ := tok.CountTokens("Hello, world!")
fmt.Printf("Tokens: %d, Exact: %v\n", count, tok.IsExact())

Model Discovery

// Look up metadata for a specific model
meta := tokenizer.LookupModel("gpt-4o")
fmt.Printf("Encoding: %s, Context: %d\n", meta.Encoding, meta.ContextWindow)

// List all registered models
models := tokenizer.ListModels()

// List models by provider
openaiModels := tokenizer.ListModelsByProvider(tokenizer.ProviderOpenAI)

Development

Requires just for the build system.

just                       # List all recipes
just build                 # Build (with fmt + vet)
just test all              # Run all tests
just test unit             # Unit tests only
just test integration      # Integration tests only
just test coverage         # Coverage report
just test bench            # Benchmarks
just release all            # Cross-compile for all platforms

License

MIT License. See LICENSE for details.