tokens

package
v1.46.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: AGPL-3.0 Imports: 13 Imported by: 0

Documentation

Overview

Package tokens provides JSONL-based token usage parsing and aggregation for Claude Code sessions.

Privacy guarantee: ParseResult and all derived types carry only token counts, tool names (e.g. "Bash", "mcp__datadog__search_logs"), skill names (short /command strings), and aggregated statistics. The actual text of user prompts, assistant responses, file contents, or command outputs is NEVER stored.

Architecture:

  • Parser.ParseFile reads a Claude JSONL transcript file line-by-line using a 10MB bufio.Scanner buffer. Each line is a JSON object; malformed lines are skipped without returning an error.
  • TokenStore caches parsed results keyed by file path, invalidating on modtime change. A background walker pre-parses all JSONL files on startup; fsnotify callbacks keep the cache fresh for active sessions.
  • PricingTable maps normalized model family names to USD-per-MTok rates and computes estimated cost from a ParseResult.
  • Associator links a ParseResult to a stapler-squad session by conversation UUID, project path prefix, or timestamp proximity.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func NormalizeModelFamily 

func NormalizeModelFamily(modelID string) string

NormalizeModelFamily strips date suffixes and normalizes a raw model ID to a pricing-table key.

Examples: 

"claude-sonnet-4-6-20250514" → "claude-sonnet-4"
"claude-sonnet-4-6"          → "claude-sonnet-4"
"claude-opus-4-7"            → "claude-opus-4"
"claude-3-opus-20240229"     → "claude-opus-3"
"claude-haiku-4"             → "claude-haiku-4"
"unknown-model-xyz"          → "unknown-model-xyz"

Types

type Associator 

type Associator struct {
	// contains filtered or unexported fields
}

Associator links ParseResult values to stapler-squad sessions.

func NewAssociator 

func NewAssociator(storage SessionStorage) *Associator

NewAssociator creates a new Associator backed by the given storage.

func (*Associator) Associate 

func (a *Associator) Associate(result *ParseResult) (sessionID string, isOrphan bool)

Associate returns the stapler-squad session ID that best matches the given ParseResult, and whether the result is an orphan (no match found).

Lookup priority:

  1. Exact conversation UUID match (ParseResult.SessionUUID == session.ConversationID)
  2. Project path prefix match (ParseResult.ProjectPath is a prefix of session.Path)
  3. Timestamp proximity (file mod time within ±5 minutes of session.CreatedAt)

func (*Associator) AssociateWithSnapshot added in v1.44.0 

func (a *Associator) AssociateWithSnapshot(result *ParseResult, sessions []SessionRecord) (sessionID string, isOrphan bool)

AssociateWithSnapshot is Associate against a pre-fetched session snapshot (see Snapshot), instead of re-querying storage on every call.

func (*Associator) Snapshot added in v1.44.0 

func (a *Associator) Snapshot() []SessionRecord

Snapshot fetches the current session records once, for callers that need to call AssociateWithSnapshot for many results without re-querying storage per call. Both ListInstancesFiltered-style loops in InsightsService previously called Associate per result, each paying a fresh ListSessionRecords() -> ListInstanceData() full-repository scan.

type ModelPricing 

type ModelPricing struct {
	ModelFamily        string  // normalized key, e.g. "claude-sonnet-4"
	InputPricePerMTok  float64 // USD per 1M input tokens
	OutputPricePerMTok float64 // USD per 1M output tokens
	CacheWritePerMTok  float64 // USD per 1M cache-write tokens
	CacheReadPerMTok   float64 // USD per 1M cache-read tokens
	EffectiveDate      string  // ISO date of last price update
}

ModelPricing holds per-model token prices in USD per million tokens.

type ParseResult 

type ParseResult struct {
	SessionUUID  string
	ProjectPath  string   // decoded from project dir name (best-effort)
	PrimaryModel string   // most-used model in this session
	Models       []string // all distinct models observed

	TotalInput    int64
	TotalOutput   int64
	CacheCreation int64
	CacheRead     int64
	MessageCount  int

	TurnTimeline     []TurnStats // per-assistant-message stats for burn rate chart
	ToolUsage        map[string]ToolTokenStats
	SkillActivations []SkillActivation

	ParsedAt    time.Time
	FileModTime time.Time // used for cache invalidation
}

ParseResult holds aggregated token data extracted from one JSONL file. Privacy: only tool names, skill names (short strings), and token counts. Message content is never stored.

type Parser 

type Parser struct{}

Parser parses Claude Code JSONL transcript files into ParseResult values.

func NewParser 

func NewParser() *Parser

NewParser creates a new Parser.

func (*Parser) ParseFile 

func (p *Parser) ParseFile(filePath string) (*ParseResult, error)

ParseFile reads a JSONL transcript file and returns an aggregated ParseResult. Malformed or truncated lines are skipped without returning an error. The caller must not retain message content — ParseResult only holds aggregates.

func (*Parser) ParseReader 

func (p *Parser) ParseReader(r io.Reader) (*ParseResult, error)

ParseReader parses JSONL from an io.Reader. Suitable for tests that pass in strings via strings.NewReader.

type PricingTable 

type PricingTable struct {
	Prices     map[string]ModelPricing
	LoadedAt   time.Time
	ConfigPath string // empty = hardcoded only
}

PricingTable maps normalized model family names to pricing. Hardcoded defaults; overridable via config JSON.

func DefaultPricingTable 

func DefaultPricingTable() *PricingTable

DefaultPricingTable returns a PricingTable with hardcoded defaults as of 2026-07-27. Prices are in USD per million tokens.

func LoadPricingOverride 

func LoadPricingOverride(configPath string) (*PricingTable, error)

LoadPricingOverride loads pricing from a JSON file and merges it over the hardcoded defaults. Unknown fields are ignored. The file must be a JSON object mapping model family names to ModelPricing objects.

func (*PricingTable) EstimateCost 

func (pt *PricingTable) EstimateCost(r *ParseResult) (cost float64, unpriced []string)

EstimateCost computes USD cost for a ParseResult using the PricingTable. Returns 0.0 for the cost of any model family not found in the table, and reports those skipped families in unpriced (sorted) so the caller can distinguish "genuinely zero usage" from "usage present but unpriced" — see ADR-001-unpriced-signal-return-shape.md. A family with zero usage across every counter (e.g. Claude Code's internal "<synthetic>" turns, which the parser already filters out of TurnTimeline in production — see parser.go's syntheticModelSentinel) is never flagged unpriced, since there is nothing to price and nothing to warn about.

func (*PricingTable) IsStale 

func (pt *PricingTable) IsStale() bool

IsStale returns true when any entry in the table has an EffectiveDate older than 30 days, indicating the pricing data may be outdated.

func (*PricingTable) LookupByModel 

func (pt *PricingTable) LookupByModel(modelID string) (ModelPricing, bool)

LookupByModel returns the ModelPricing for a raw model ID (normalizes first). Returns zero-value ModelPricing and false if not found.

func (*PricingTable) ModelFamilyCost 

func (pt *PricingTable) ModelFamilyCost(r *ParseResult) (costs map[string]float64, unpriced map[string]bool)

ModelFamilyCost returns a breakdown of estimated cost per model family, plus the set of families that had usage but no PricingTable entry (unpriced). A family with zero usage across every counter is never flagged unpriced — see EstimateCost's doc comment and ADR-001-unpriced-signal-return-shape.md for why.

type SessionRecord 

type SessionRecord struct {
	SessionID      string
	ConversationID string // matches ParseResult.SessionUUID
	Path           string // working directory
	CreatedAt      time.Time
}

SessionRecord is a minimal snapshot of a stapler-squad session used for matching against ParseResult values. This avoids importing the full session package and prevents circular dependencies.

type SessionStorage 

type SessionStorage interface {
	// ListSessionRecords returns a snapshot of all sessions for association.
	ListSessionRecords() []SessionRecord
}

SessionStorage is the interface Associator uses to look up sessions. Implemented by session.Storage (or a test stub).

type SkillActivation 

type SkillActivation struct {
	Name      string // e.g. "code-review", "/plan:feature"
	TurnIndex int    // which human turn triggered it
	IsCommand bool   // true for /command, false for skill name
}

SkillActivation records a detected skill or command invocation.

type TokenStore 

type TokenStore struct {
	// contains filtered or unexported fields
}

TokenStore caches parsed JSONL results keyed by file path. It pre-parses all JSONL files in a directory on startup and keeps the cache fresh via fsnotify callbacks.

func NewTokenStore 

func NewTokenStore(historyDir string) *TokenStore

NewTokenStore creates a TokenStore that will pre-parse all JSONL files in historyDir on startup.

func (*TokenStore) GetAll 

func (ts *TokenStore) GetAll() []*ParseResult

GetAll returns a snapshot of all cached ParseResult values under read lock.

func (*TokenStore) GetByUUID 

func (ts *TokenStore) GetByUUID(uuid string) *ParseResult

GetByUUID returns the ParseResult for a given conversation UUID, or nil.

func (*TokenStore) IsLoading 

func (ts *TokenStore) IsLoading() bool

IsLoading returns true while the background walk is still in progress.

func (*TokenStore) OnHistoryFileChanged 

func (ts *TokenStore) OnHistoryFileChanged(filePath string)

OnHistoryFileChanged is called by the HistoryFileWatcher callback when a file is created or modified. It enqueues the file for re-parsing.

func (*TokenStore) Start 

func (ts *TokenStore) Start(ctx context.Context)

Start launches background workers and the initial directory walker. It stops when ctx is cancelled. Call this once after creating the store.

func (*TokenStore) Stop 

func (ts *TokenStore) Stop()

Stop cancels the background context, stopping all goroutines.

func (*TokenStore) Subscribe 

func (ts *TokenStore) Subscribe() <-chan struct{}

Subscribe returns a channel that receives a struct{} whenever the store is updated. The caller should drain the channel promptly to avoid blocking notifications.

func (*TokenStore) Unsubscribe 

func (ts *TokenStore) Unsubscribe(ch <-chan struct{})

Unsubscribe removes a subscriber channel.

type TokenStoreReader 

type TokenStoreReader interface {
	GetAll() []*ParseResult
	GetByUUID(uuid string) *ParseResult
	IsLoading() bool
	Subscribe() <-chan struct{}
	Unsubscribe(ch <-chan struct{})
}

TokenStoreReader is the read-only interface InsightsService needs from a TokenStore. Defined as an interface so test fakes can be injected without constructing a real store.

type ToolTokenStats 

type ToolTokenStats struct {
	ToolName  string
	CallCount int
	// MCPServer is non-empty when tool follows mcp__<server>__<tool> pattern.
	MCPServer string
}

ToolTokenStats aggregates attribution for one tool name. Token attribution is message-level (not per-tool-call); CallCount is exact.

type TurnStats 

type TurnStats struct {
	Timestamp     time.Time
	Model         string
	Input         int64
	Output        int64
	CacheCreation int64
	CacheRead     int64
	ToolNames     []string // tool_use block names in this message
}

TurnStats is per-assistant-message token data (for timeline/burn-rate chart).

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.