tokens

Documentation

Overview

Package tokens provides JSONL-based token usage parsing and aggregation for Claude Code sessions.

Privacy guarantee: ParseResult and all derived types carry only token counts, tool names (e.g. "Bash", "mcp__datadog__search_logs"), skill names (short /command strings), and aggregated statistics. The actual text of user prompts, assistant responses, file contents, or command outputs is NEVER stored.

Architecture:

• Parser.ParseFile reads a Claude JSONL transcript file line-by-line using a 10MB bufio.Scanner buffer. Each line is a JSON object; malformed lines are skipped without returning an error.
• TokenStore caches parsed results keyed by file path, invalidating on modtime change. A background walker pre-parses all JSONL files on startup; fsnotify callbacks keep the cache fresh for active sessions.
• PricingTable maps normalized model family names to USD-per-MTok rates and computes estimated cost from a ParseResult.
• Associator links a ParseResult to a stapler-squad session by conversation UUID, project path prefix, or timestamp proximity.

Index

• func NormalizeModelFamily(modelID string) string
• type Associator
  • func NewAssociator(storage SessionStorage) *Associator
  • func (a *Associator) Associate(result *ParseResult) (sessionID string, isOrphan bool)
• type ModelPricing
• type ParseResult
• type Parser
  • func NewParser() *Parser
  • func (p *Parser) ParseFile(filePath string) (*ParseResult, error)
  • func (p *Parser) ParseReader(r io.Reader) (*ParseResult, error)
• type PricingTable
  • func DefaultPricingTable() *PricingTable
  • func LoadPricingOverride(configPath string) (*PricingTable, error)
  • func (pt *PricingTable) EstimateCost(r *ParseResult) float64
  • func (pt *PricingTable) IsStale() bool
  • func (pt *PricingTable) LookupByModel(modelID string) (ModelPricing, bool)
  • func (pt *PricingTable) ModelFamilyCost(r *ParseResult) map[string]float64
• type SessionRecord
• type SessionStorage
• type SkillActivation
• type TokenStore
  • func NewTokenStore(historyDir string) *TokenStore
  • func (ts *TokenStore) GetAll() []*ParseResult
  • func (ts *TokenStore) GetByUUID(uuid string) *ParseResult
  • func (ts *TokenStore) IsLoading() bool
  • func (ts *TokenStore) OnHistoryFileChanged(filePath string)
  • func (ts *TokenStore) Start(ctx context.Context)
  • func (ts *TokenStore) Stop()
  • func (ts *TokenStore) Subscribe() <-chan struct{}
  • func (ts *TokenStore) Unsubscribe(ch <-chan struct{})
• type TokenStoreReader
• type ToolTokenStats
• type TurnStats

Functions

func NormalizeModelFamily

func NormalizeModelFamily(modelID string) string

NormalizeModelFamily strips date suffixes and normalizes a raw model ID to a pricing-table key.

Examples:

"claude-sonnet-4-6-20250514" → "claude-sonnet-4"
"claude-sonnet-4-6"          → "claude-sonnet-4"
"claude-opus-4-7"            → "claude-opus-4"
"claude-3-opus-20240229"     → "claude-opus-3"
"claude-haiku-4"             → "claude-haiku-4"
"unknown-model-xyz"          → "unknown-model-xyz"

Types

type Associator

type Associator struct {
	// contains filtered or unexported fields
}

Associator links ParseResult values to stapler-squad sessions.

func NewAssociator

func NewAssociator(storage SessionStorage) *Associator

NewAssociator creates a new Associator backed by the given storage.

func (*Associator) Associate

func (a *Associator) Associate(result *ParseResult) (sessionID string, isOrphan bool)

Associate returns the stapler-squad session ID that best matches the given ParseResult, and whether the result is an orphan (no match found).

Lookup priority:

1. Exact conversation UUID match (ParseResult.SessionUUID == session.ConversationID)
2. Project path prefix match (ParseResult.ProjectPath is a prefix of session.Path)
3. Timestamp proximity (file mod time within ±5 minutes of session.CreatedAt)

type ModelPricing

type ModelPricing struct {
	ModelFamily        string  // normalized key, e.g. "claude-sonnet-4"
	InputPricePerMTok  float64 // USD per 1M input tokens
	OutputPricePerMTok float64 // USD per 1M output tokens
	CacheWritePerMTok  float64 // USD per 1M cache-write tokens
	CacheReadPerMTok   float64 // USD per 1M cache-read tokens
	EffectiveDate      string  // ISO date of last price update
}

ModelPricing holds per-model token prices in USD per million tokens.

type ParseResult

type ParseResult struct {
	SessionUUID  string
	ProjectPath  string   // decoded from project dir name (best-effort)
	PrimaryModel string   // most-used model in this session
	Models       []string // all distinct models observed

	TotalInput    int64
	TotalOutput   int64
	CacheCreation int64
	CacheRead     int64
	MessageCount  int

	TurnTimeline     []TurnStats // per-assistant-message stats for burn rate chart
	ToolUsage        map[string]ToolTokenStats
	SkillActivations []SkillActivation

	ParsedAt    time.Time
	FileModTime time.Time // used for cache invalidation
}

ParseResult holds aggregated token data extracted from one JSONL file. Privacy: only tool names, skill names (short strings), and token counts. Message content is never stored.

type Parser

type Parser struct{}

Parser parses Claude Code JSONL transcript files into ParseResult values.

func NewParser

func NewParser() *Parser

NewParser creates a new Parser.

func (*Parser) ParseFile

func (p *Parser) ParseFile(filePath string) (*ParseResult, error)

ParseFile reads a JSONL transcript file and returns an aggregated ParseResult. Malformed or truncated lines are skipped without returning an error. The caller must not retain message content — ParseResult only holds aggregates.

func (*Parser) ParseReader

func (p *Parser) ParseReader(r io.Reader) (*ParseResult, error)

ParseReader parses JSONL from an io.Reader. Suitable for tests that pass in strings via strings.NewReader.

type PricingTable

type PricingTable struct {
	Prices     map[string]ModelPricing
	LoadedAt   time.Time
	ConfigPath string // empty = hardcoded only
}

PricingTable maps normalized model family names to pricing. Hardcoded defaults; overridable via config JSON.

func DefaultPricingTable

func DefaultPricingTable() *PricingTable

DefaultPricingTable returns a PricingTable with hardcoded defaults as of 2026-05-15. Prices are in USD per million tokens.

func LoadPricingOverride

func LoadPricingOverride(configPath string) (*PricingTable, error)

LoadPricingOverride loads pricing from a JSON file and merges it over the hardcoded defaults. Unknown fields are ignored. The file must be a JSON object mapping model family names to ModelPricing objects.

func (*PricingTable) EstimateCost

func (pt *PricingTable) EstimateCost(r *ParseResult) float64

EstimateCost computes USD cost for a ParseResult using the PricingTable. Returns 0.0 if the model is not found in the table.

func (*PricingTable) IsStale

func (pt *PricingTable) IsStale() bool

IsStale returns true when any entry in the table has an EffectiveDate older than 30 days, indicating the pricing data may be outdated.

func (*PricingTable) LookupByModel

func (pt *PricingTable) LookupByModel(modelID string) (ModelPricing, bool)

LookupByModel returns the ModelPricing for a raw model ID (normalizes first). Returns zero-value ModelPricing and false if not found.

func (*PricingTable) ModelFamilyCost

func (pt *PricingTable) ModelFamilyCost(r *ParseResult) map[string]float64

ModelFamilyCost returns a breakdown of estimated cost per model family.

type SessionRecord

type SessionRecord struct {
	SessionID      string
	ConversationID string // matches ParseResult.SessionUUID
	Path           string // working directory
	CreatedAt      time.Time
}

SessionRecord is a minimal snapshot of a stapler-squad session used for matching against ParseResult values. This avoids importing the full session package and prevents circular dependencies.

type SessionStorage

type SessionStorage interface {
	// ListSessionRecords returns a snapshot of all sessions for association.
	ListSessionRecords() []SessionRecord
}

SessionStorage is the interface Associator uses to look up sessions. Implemented by session.Storage (or a test stub).

type SkillActivation

type SkillActivation struct {
	Name      string // e.g. "code-review", "/plan:feature"
	TurnIndex int    // which human turn triggered it
	IsCommand bool   // true for /command, false for skill name
}

SkillActivation records a detected skill or command invocation.

type TokenStore

type TokenStore struct {
	// contains filtered or unexported fields
}

TokenStore caches parsed JSONL results keyed by file path. It pre-parses all JSONL files in a directory on startup and keeps the cache fresh via fsnotify callbacks.

func NewTokenStore

func NewTokenStore(historyDir string) *TokenStore

NewTokenStore creates a TokenStore that will pre-parse all JSONL files in historyDir on startup.

func (*TokenStore) GetAll

func (ts *TokenStore) GetAll() []*ParseResult

GetAll returns a snapshot of all cached ParseResult values under read lock.

func (*TokenStore) GetByUUID

func (ts *TokenStore) GetByUUID(uuid string) *ParseResult

GetByUUID returns the ParseResult for a given conversation UUID, or nil.

func (*TokenStore) IsLoading

func (ts *TokenStore) IsLoading() bool

IsLoading returns true while the background walk is still in progress.

func (*TokenStore) OnHistoryFileChanged

func (ts *TokenStore) OnHistoryFileChanged(filePath string)

OnHistoryFileChanged is called by the HistoryFileWatcher callback when a file is created or modified. It enqueues the file for re-parsing.

func (*TokenStore) Start

func (ts *TokenStore) Start(ctx context.Context)

Start launches background workers and the initial directory walker. It stops when ctx is cancelled. Call this once after creating the store.

func (*TokenStore) Stop

func (ts *TokenStore) Stop()

Stop cancels the background context, stopping all goroutines.

func (*TokenStore) Subscribe

func (ts *TokenStore) Subscribe() <-chan struct{}

Subscribe returns a channel that receives a struct{} whenever the store is updated. The caller should drain the channel promptly to avoid blocking notifications.

func (*TokenStore) Unsubscribe

func (ts *TokenStore) Unsubscribe(ch <-chan struct{})

Unsubscribe removes a subscriber channel.

type TokenStoreReader

type TokenStoreReader interface {
	GetAll() []*ParseResult
	GetByUUID(uuid string) *ParseResult
	IsLoading() bool
	Subscribe() <-chan struct{}
	Unsubscribe(ch <-chan struct{})
}

TokenStoreReader is the read-only interface InsightsService needs from a TokenStore. Defined as an interface so test fakes can be injected without constructing a real store.

type ToolTokenStats

type ToolTokenStats struct {
	ToolName  string
	CallCount int
	// MCPServer is non-empty when tool follows mcp__<server>__<tool> pattern.
	MCPServer string
}

ToolTokenStats aggregates attribution for one tool name. Token attribution is message-level (not per-tool-call); CallCount is exact.

type TurnStats

type TurnStats struct {
	Timestamp     time.Time
	Model         string
	Input         int64
	Output        int64
	CacheCreation int64
	CacheRead     int64
	ToolNames     []string // tool_use block names in this message
}

TurnStats is per-assistant-message token data (for timeline/burn-rate chart).