Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrBudgetExceeded indicates the request exceeds available budget ErrBudgetExceeded = errors.New("budget exceeded") // ErrNoProviders indicates no providers are available ErrNoProviders = errors.New("no providers available") // ErrAllProvidersFailed indicates all providers in the chain failed ErrAllProvidersFailed = errors.New("all providers failed") )
Functions ¶
func ResolveModelKey ¶
ResolveModelKey returns the single model key that both the budget ceiling and the charge must be accounted against for this request.
Preference order: what the provider says it will actually send, then the caller's explicit ModelKey. The result is empty only when neither is known, and a configured tracker refuses an empty key rather than admitting spend it cannot price.
It deliberately never substitutes a placeholder such as "default": a configured price table has no entry for that, so the request would be refused while the provider bills a real model.
Types ¶
type Estimator ¶
type Estimator struct {
// contains filtered or unexported fields
}
Estimator counts tokens for budget checks, caching results per (model, input).
Entries are keyed by a digest of the input: the cache must not keep prompt text alive, because prompts carry repository content. Eviction is arbitrary, not LRU - the cache exists to avoid recomputation, not to guarantee hits.
func (*Estimator) Estimate ¶
Estimate returns the provider's own token count for input, falling back to the character heuristic when the provider cannot count.
func (*Estimator) EstimateTokens ¶
EstimateTokens returns a model-independent token estimate for input.
type ModelResolver ¶
ModelResolver is implemented by providers that can name, before the call is made, the concrete model they will bill against for a given set of options.
This exists because Options.ModelKey is routinely empty - DefaultOptions never sets it - while every provider here substitutes a model of its own ("gpt-4", "llama3", a configured LiteLLM model). Without a pre-call answer the ceiling is checked against a key no price table contains while the charge lands on a different one, so the ceiling reads a ledger that never moves.
type Options ¶
type Options struct {
System string `json:"system,omitempty"`
Temperature float64 `json:"temperature,omitempty"`
MaxTokens int `json:"max_tokens,omitempty"`
ModelKey string `json:"model_key,omitempty"`
}
Options represents LLM request options. Every field here must be carried by a provider; a field no provider sends is a silent no-op for the caller.
func DefaultOptions ¶
func DefaultOptions() Options
DefaultOptions returns sensible defaults for LLM options.
Temperature is left at its zero value, deliberately (AUR-460): a fixed 0.3 default meant every request carried an explicit sampling parameter, and a growing family of gateway-served models (measured: gpt-5.6-luna, -sol, -terra) 400s on ANY explicit "temperature" value, including 0 and including that model's own advertised default. A caller that wants a specific temperature still sets Options.Temperature itself; the field stays on the struct for that. What no longer happens is this package picking a non-zero value nobody asked for and every provider dutifully forwarding it upstream.
type Orchestrator ¶
type Orchestrator struct {
// contains filtered or unexported fields
}
Orchestrator manages LLM provider chains with fallback and budget enforcement
func NewOrchestrator ¶
func NewOrchestrator(primary Provider, fallbacks []Provider, tracker *cost.Tracker) *Orchestrator
NewOrchestrator creates a new orchestrator with a primary provider and optional fallbacks
func (*Orchestrator) Complete ¶
Complete executes a completion request with fallback chain and budget enforcement
func (*Orchestrator) GetProviderChain ¶
func (o *Orchestrator) GetProviderChain() []string
GetProviderChain returns the current provider chain (primary + fallbacks)
func (*Orchestrator) RemainingBudget ¶
func (o *Orchestrator) RemainingBudget() (float64, float64)
RemainingBudget returns the remaining budget (perRun, daily)
func (*Orchestrator) ResetPerRunBudget ¶
func (o *Orchestrator) ResetPerRunBudget()
ResetPerRunBudget resets the per-run budget counter
func (*Orchestrator) UntrackedSpends ¶
func (o *Orchestrator) UntrackedSpends() int
UntrackedSpends returns how many completions were paid for but charged to no budget, because the model key has no price entry.
type Provider ¶
type Provider interface {
Complete(prompt string, opts Options) (Response, error)
Tokens(input string) (int, error)
Name() string
}
Provider defines the interface for LLM providers.
Complete takes no context: cancellation cannot reach the provider's transport, so the orchestrator can only abandon a slow call, not stop it.
Source Files
Directories