token

Documentation

Overview

Package token provides token counting utilities using tiktoken encoding.

Index

• type Counter
  • func NewCounter() (*Counter, error)
  • func (c *Counter) CountMessages(system string, messages []MessageContent) (int, error)
  • func (c *Counter) CountTokens(text string) (int, error)
• type MessageContent

Types

type Counter

type Counter struct {
	// contains filtered or unexported fields
}

Counter handles token counting for text and message arrays.

func NewCounter

func NewCounter() (*Counter, error)

NewCounter creates a new token counter with cl100k_base encoding.

func (*Counter) CountMessages

func (c *Counter) CountMessages(system string, messages []MessageContent) (int, error)

CountMessages counts tokens in a message array. Estimates tokens for system prompt + messages with formatting overhead.

func (*Counter) CountTokens

func (c *Counter) CountTokens(text string) (int, error)

CountTokens counts tokens in a string.

type MessageContent

type MessageContent struct {
	Role        string
	Content     string
	ExtraTokens int
}

MessageContent represents a single message in a conversation.