tokencount

Documentation

Overview

Package tokencount provides token estimation for LLM text.

Index

• Constants
• func EstimateTokens(text string) int

Constants

const CharsPerToken = 4

CharsPerToken is the approximate number of characters per token for GPT-style tokenizers.

Functions

func EstimateTokens

func EstimateTokens(text string) int

EstimateTokens estimates the number of LLM tokens in the given text. Uses a simple heuristic of approximately 4 characters per token, which is a common approximation for English text with GPT-style tokenizers.