tokens

package
v1.24.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Apr 13, 2026 License: AGPL-3.0 Imports: 12 Imported by: 0

Documentation

Index

Constants

View Source 
const DefaultMetadataURL = "https://raw.githubusercontent.com/BerriAI/litellm/main/model_prices_and_context_window.json"

DefaultMetadataURL is the default URL for litellm model metadata.

Variables

View Source 
var KnownLimits = map[string]int{

	"claude-sonnet-4-20250514":  200_000,
	"claude-opus-4-20250514":    200_000,
	"claude-haiku-3-5-20241022": 200_000,
	"claude-sonnet-4-6-latest":  200_000,
	"claude-opus-4-6-latest":    200_000,

	"gpt-4o":      128_000,
	"gpt-4o-mini": 128_000,
	"gpt-4-turbo": 128_000,
	"gpt-4":       8_192,
	"o1":          200_000,
	"o1-mini":     128_000,
	"o3":          200_000,
	"o3-mini":     200_000,
	"o4-mini":     200_000,

	"gemini-2.5-pro":   1_048_576,
	"gemini-2.5-flash": 1_048_576,
	"gemini-2.0-flash": 1_048_576,
	"gemini-1.5-pro":   2_097_152,
	"gemini-1.5-flash": 1_048_576,
}

KnownLimits maps model identifiers to their context window size in tokens.

Functions

func DiscoverModels added in v1.3.0 

func DiscoverModels(baseURL, apiKey string) ([]string, error)

DiscoverModels queries an OpenAI-compatible /models endpoint to list available models. It tries {baseURL}/models first; if that returns 404 and baseURL doesn't already end with /v1, it retries {baseURL}/v1/models. Returns sorted model IDs on success.

func EnrichWithMetadata added in v1.3.0 

func EnrichWithMetadata(modelIDs []string, store *MetadataStore) []config.ModelSummary

EnrichWithMetadata cross-references discovered model IDs against the litellm MetadataStore to attach capability information. Models without a litellm match are included with zero-value capabilities.

func FetchMetadata 

func FetchMetadata(url string, timeout time.Duration) ([]byte, error)

FetchMetadata fetches the litellm model metadata JSON from the given URL. It returns the raw JSON bytes on success.

func FormatCost added in v1.10.0 

func FormatCost(usd float64) string

FormatCost formats a USD amount for display.

func FormatTokenCount 

func FormatTokenCount(n int) string

FormatTokenCount formats a token count for display. e.g., 1234 → "1.2K", 1234567 → "1.2M", 500 → "500"

func LimitForModel 

func LimitForModel(model string, configOverride int) int

LimitForModel returns the context window limit for a model.

  • If configOverride > 0, it takes precedence.
  • Otherwise, the built-in KnownLimits is consulted.
  • If the model is unknown and no override is set, returns 0 (unlimited).

func LoadCachedMetadata 

func LoadCachedMetadata(path string) ([]byte, time.Time, error)

LoadCachedMetadata reads cached metadata JSON from disk. Returns the raw bytes, the file modification time, and any error.

func ParseMetadata 

func ParseMetadata(data []byte) (map[string]ModelInfo, error)

ParseMetadata parses raw litellm JSON into a map of model name → ModelInfo. Unknown fields in each entry are silently ignored.

func SaveCachedMetadata 

func SaveCachedMetadata(path string, data []byte) error

SaveCachedMetadata writes raw JSON to the cache path with 0600 permissions. Parent directories are created if needed.

Types

type CostFunc added in v1.10.0 

type CostFunc func(model, providerType string, inputTokens, outputTokens int) float64

CostFunc computes the estimated USD cost for given token counts. model is the model name, providerType is the provider type string.

type MetadataStore 

type MetadataStore struct {
	// contains filtered or unexported fields
}

MetadataStore wraps parsed litellm model metadata and provides lookup methods.

func NewMetadataStore 

func NewMetadataStore(url string, cachePath string, cacheTTL time.Duration) (*MetadataStore, error)

NewMetadataStore creates a MetadataStore by orchestrating the fetch/cache/fallback flow:

  1. Check cache mtime vs TTL — if fresh, use cache
  2. If stale or missing, fetch from URL
  3. On fetch success, parse and save cache
  4. On fetch failure, fall back to stale cache
  5. If no cache available, fall back to an empty map

Errors are logged as warnings but never returned — the store always succeeds.

func (*MetadataStore) CapabilitiesForModel 

func (s *MetadataStore) CapabilitiesForModel(model string, providerType string) ModelInfo

CapabilitiesForModel returns the parsed capabilities for a model. If the model is not found, a zero-value ModelInfo is returned.

func (*MetadataStore) CostForTokens added in v1.10.0 

func (s *MetadataStore) CostForTokens(model string, providerType string, inputTokens, outputTokens int) float64

CostForTokens returns the estimated cost in USD for the given input/output token counts. Returns 0 if pricing data is unavailable.

func (*MetadataStore) LimitForModel 

func (s *MetadataStore) LimitForModel(model string, providerType string, configOverride int) int

LimitForModel returns the context window limit for a model using three-tier fallback:

  1. Config override (if > 0)
  2. Litellm max_input_tokens from metadata
  3. Hardcoded KnownLimits
  4. 0 (unlimited)

func (*MetadataStore) Lookup 

func (s *MetadataStore) Lookup(model string, providerType string) (ModelInfo, bool)

Lookup searches for a model in the metadata store using multi-strategy matching:

  1. Exact match on the key
  2. Try "{providerType}/{model}"
  3. Scan keys ending with "/{model}"

func (*MetadataStore) ModelsForProvider added in v1.1.0 

func (s *MetadataStore) ModelsForProvider(providerType string) []config.ModelSummary

ModelsForProvider returns a sorted list of ModelSummary entries for the given provider type. Models are found by matching slash-prefixed keys and bare keys against the provider's matcher patterns. The list is sorted with priority models first, then alphabetically.

func (*MetadataStore) SupportsToolCalling added in v1.13.1 

func (s *MetadataStore) SupportsToolCalling(model string, providerType string) bool

SupportsToolCalling returns true if the model is known to support structured tool/function calling according to litellm metadata. Returns true for unknown models (optimistic default).

type ModelInfo 

type ModelInfo struct {
	MaxInputTokens          int     `json:"max_input_tokens"`
	MaxOutputTokens         int     `json:"max_output_tokens"`
	InputCostPerToken       float64 `json:"input_cost_per_token"`
	OutputCostPerToken      float64 `json:"output_cost_per_token"`
	SupportsFunctionCalling bool    `json:"supports_function_calling"`
	SupportsVision          bool    `json:"supports_vision"`
	SupportsReasoning       bool    `json:"supports_reasoning"`
	SupportsResponseSchema  bool    `json:"supports_response_schema"`
}

ModelInfo holds parsed metadata for a single model from the litellm database.

type ProviderUsage 

type ProviderUsage struct {
	ProviderID      string
	Model           string
	InputTokens     int     // accumulated across all turns (for display)
	OutputTokens    int     // accumulated across all turns (for display)
	LastInputTokens int     // most recent request's input tokens (for context % calculation)
	Limit           int     // 0 means unlimited
	Cost            float64 // accumulated estimated cost in USD (0 if pricing unavailable)
}

ProviderUsage holds the accumulated usage and limit for one provider.

func (ProviderUsage) Percent 

func (u ProviderUsage) Percent() float64

Percent returns the context window usage as a percentage of the limit, based on the most recent request's input tokens (not accumulated total). Returns 0 if the limit is unlimited (0).

type TokenTracker 

type TokenTracker struct {
	// contains filtered or unexported fields
}

TokenTracker accumulates token usage per provider across a session.

func NewTracker 

func NewTracker(providerModels map[string]string, providerLimits map[string]int) *TokenTracker

NewTracker creates a TokenTracker. Pass provider ID → model name and provider ID → resolved context limit.

func (*TokenTracker) Add 

func (t *TokenTracker) Add(providerID string, u Usage)

Add records token usage for a provider from a single API call.

func (*TokenTracker) Get 

func (t *TokenTracker) Get(providerID string) ProviderUsage

Get returns the accumulated usage for a single provider.

func (*TokenTracker) Reset added in v1.23.0 

func (t *TokenTracker) Reset()

Reset clears all accumulated token usage and cost for every provider, preserving the provider list, models, limits, and cost function.

func (*TokenTracker) SetCostFunc added in v1.10.0 

func (t *TokenTracker) SetCostFunc(fn CostFunc)

SetCostFunc sets an optional cost estimation function.

func (*TokenTracker) SetProviderType added in v1.10.0 

func (t *TokenTracker) SetProviderType(providerID, providerType string)

SetProviderType records the provider type for a provider ID.

func (*TokenTracker) Summary 

func (t *TokenTracker) Summary() []ProviderUsage

Summary returns a snapshot of all providers' usage.

func (*TokenTracker) TotalCost added in v1.10.0 

func (t *TokenTracker) TotalCost() float64

TotalCost returns the accumulated cost across all providers.

func (*TokenTracker) WouldExceedLimit 

func (t *TokenTracker) WouldExceedLimit(providerID string) bool

WouldExceedLimit returns true if the provider's current input tokens are at or above its context limit. Returns false if limit is 0 (unlimited).

type Usage 

type Usage struct {
	InputTokens  int
	OutputTokens int
}

Usage holds token counts for a single API call.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.