llm

README

pi-llm-go

Minimal, provider-agnostic LLM client for Go. Streaming completions, tool calling, prompt caching, token counting, cost projection, and retry middleware for Anthropic Claude (Messages API), OpenAI (Chat Completions + Responses API, including GPT-5 family), Google Gemini (text / image / video), and any OpenAI-compatible endpoint (Azure OpenAI, Groq, Together, vLLM, OpenRouter, Ollama). Idiomatic Go: iter.Seq2 streaming, sealed sum types, errors.Is sentinels, context.Context cancellation.

Status: v1.0.0 — stable. The public API follows semver: no breaking changes without a major (module-path) bump. See CHANGELOG.md. Dogfooded in production at Noumenal.

Install

go get github.com/amit-timalsina/pi-llm-go

Requires Go 1.23+ (for iter.Seq2).

Capability matrix

Capability	Anthropic	OpenAI Chat	OpenAI Responses	Gemini
Streaming text	✅	✅	✅	✅
Tool calling	✅	✅	✅	✅
Forced tool choice (Request.ToolChoice)	✅	✅	✅	✅
Strict / grammar-constrained tool input (Tool.Strict)	✅	✅	✅	✅
Image input	✅	✅	✅	✅
Video input	reject at wire	reject at wire	reject at wire	✅ native
Extended thinking	✅	—	✅ (reasoning summaries)	✅
Prompt caching (5m + 1h tier)	✅	automatic	automatic	single-TTL
Per-TTL cache-write Usage breakdown	✅	—	—	—
CountTokens (no-spend pre-flight)	✅	— (no API)	— (no API)	✅
Cost projection helper	✅	✅	✅	✅
Retry middleware (Options.Retry)	✅	✅	✅	✅
Files API helper (>20 MB inputs)	—	—	—	✅

Quickstart — one-shot Claude

package main

import (
    "context"
    "fmt"
    "os"

    llm "github.com/amit-timalsina/pi-llm-go"
    "github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
    p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

    msg, err := llm.Complete(context.Background(), p, llm.Request{
        Model:     anthropic.ClaudeSonnet4_6,
        MaxTokens: 1024,
        Messages: []llm.Message{
            {Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "Reply with one word: hello"}}},
        },
    })
    if err != nil { panic(err) }

    for _, block := range msg.Content {
        if tb, ok := block.(llm.TextBlock); ok {
            fmt.Println(tb.Text)
        }
    }
}

Quickstart — streaming with iter.Seq2

for event, err := range p.Stream(ctx, req) {
    if err != nil { /* handle */ break }
    if d, ok := event.(llm.EventTextDelta); ok {
        fmt.Print(d.Delta)
    }
}

Stream returns iter.Seq2[llm.StreamEvent, error]. Complete is a synchronous helper that drains the stream and returns the final assistant Message.

Quickstart — tool calling

req := llm.Request{
    Model:     anthropic.ClaudeSonnet4_6,
    MaxTokens: 1024,
    Tools: []llm.Tool{{
        Name:        "get_weather",
        Description: "Get the weather for a city",
        InputSchema: json.RawMessage(`{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}`),
    }},
    Messages: []llm.Message{
        {Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "What's the weather in Tokyo?"}}},
    },
}
msg, _ := llm.Complete(ctx, p, req)
for _, b := range msg.Content {
    if tc, ok := b.(llm.ToolCallBlock); ok {
        fmt.Printf("tool=%s args=%s\n", tc.Name, tc.Arguments)
        // Execute the tool yourself, then send a ToolResultBlock on the next turn.
        // For a built-in execution loop, use https://github.com/amit-timalsina/pi-agent-go
    }
}

Force a tool call with Request.ToolChoice — Auto (default), Any (must call some tool), Tool (must call a named tool), or None (text only):

req.ToolChoice = llm.ToolChoice{Type: llm.ToolChoiceTool, Name: "get_weather"}

Constrain arguments to the schema with Tool.Strict: true. The provider then samples tool arguments under a grammar derived from InputSchema, so enum fields and required keys come back exactly as declared instead of being silently dropped — no caller-side validate-and-retry loop. Wired across all four providers (Anthropic, OpenAI Chat, OpenAI Responses, Gemini):

Tools: []llm.Tool{{Name: "set_valve", InputSchema: schema, Strict: true}}

See examples/strict_tool_use for the end-to-end pattern.

When to pick pi-llm-go

You want	Pick
One Go interface across Anthropic / OpenAI / Gemini, streaming-first	pi-llm-go
Only OpenAI, no streaming abstraction needed	sashabaranov/go-openai
Only Anthropic, vendor SDK guarantees	anthropics/anthropic-sdk-go
A heavyweight framework with chains, agents, memory, embeddings, vector stores	tmc/langchaingo
Built-in agent loop on top of a provider-agnostic LLM client	pi-llm-go + pi-agent-go

pi-llm-go is the streaming-completions layer. ~1.5kLoC of plain Go: one interface, four providers, no model registries, no plugin systems, no abstractions beyond what the LLM wire format demands.

Features

• Streaming-first. Stream() returns iter.Seq2[StreamEvent, error] — Go 1.23 iterators, no callbacks, no goroutine leaks. Complete() is the synchronous helper for one-shot use.
• Sealed sum types. Block and StreamEvent are interfaces with package-private marker methods. Type-switch exhaustively; the compiler tells you if you miss a case.
• Tool calling. Declare tools on Request.Tools; receive ToolCallBlocks on the response; send ToolResultBlocks back. Pi-llm-go does not execute tools — that's pi-agent-go's job.
• Extended thinking. Set Request.Thinking and read ThinkingBlock content back. Two shapes on Anthropic: adaptive ThinkingConfig{Effort: llm.EffortMedium} (Opus 4.6+, required on 4.7+) or legacy ThinkingConfig{BudgetTokens: N} (Sonnet/Haiku); Effort wins when both are set. Add Display: llm.DisplaySummarized to the adaptive shape to stream summarized thinking text — on Opus 4.7+ the default is signature-only blocks with no text. OpenAI Responses exposes reasoning via its own Options.ReasoningEffort; Gemini thinks natively.
• Open-closed providers. Implement LLM.Stream to add custom providers; no plugin registry needed.
• Errors that branch cleanly. errors.Is(err, llm.ErrRateLimit) works through *APIError wraps; errors.As(err, &apiErr) gives you status + body.
• Cancellation = context.Context. No bespoke abort signal types.

Providers

Anthropic

import "github.com/amit-timalsina/pi-llm-go/providers/anthropic"

p, _ := anthropic.New(anthropic.Options{
    APIKey: os.Getenv("ANTHROPIC_API_KEY"),
    // BaseURL: defaults to https://api.anthropic.com
    // Version: defaults to "2023-06-01"
    // Beta:    optional anthropic-beta header values
})

Honors Request.Thinking. Surfaces all content-block types (text, thinking, tool_use).

OpenAI-compatible

import "github.com/amit-timalsina/pi-llm-go/providers/openai"

p, _ := openai.New(openai.Options{
    APIKey:  os.Getenv("OPENAI_API_KEY"),
    BaseURL: "https://api.openai.com/v1",  // or "https://api.groq.com/openai/v1", etc.
})

Talks the Chat Completions wire format, so the same provider works against any compatible host. Request.Thinking is ignored at v1 — reasoning-effort dialects vary too much across compatible hosts to map portably.

Gemini

import "github.com/amit-timalsina/pi-llm-go/providers/gemini"

p, _ := gemini.New(gemini.Options{APIKey: os.Getenv("GEMINI_API_KEY")})

Native support for the Gemini 2.5 / 3 / Robotics ER 1.6 families. Same LLM interface as the other providers, plus llm.VideoBlock for native video input (Gemini is the only provider that accepts video natively; Anthropic and OpenAI reject VideoBlock at the wire boundary with a clear pointer to the frame-extraction workaround). YouTube URLs work directly:

llm.VideoBlock{URI: "https://www.youtube.com/watch?v=..."}

For files larger than ~20 MB, the sibling providers/gemini/files sub-package handles the multipart upload + ACTIVE-state polling:

import "github.com/amit-timalsina/pi-llm-go/providers/gemini/files"

fc, _ := files.New(files.Options{APIKey: os.Getenv("GEMINI_API_KEY")})
ref, _ := fc.Upload(ctx, mp4Reader, "video/mp4", files.UploadOptions{DisplayName: "demo.mp4"})
ref, _ = fc.Wait(ctx, ref, files.WaitOptions{}) // polls until ACTIVE
defer fc.Delete(context.Background(), ref.Name) // ~48h server-side TTL if you forget

// Now use the URI in a generateContent call:
content := []llm.Block{llm.TextBlock{Text: "describe"}, llm.VideoBlock{URI: ref.URI}}

Vertex AI (gs:// URIs + OAuth) is a planned future addition; the Gemini provider currently targets the Google AI direct endpoint.

Error handling

Non-2xx HTTP responses surface as *llm.APIError wrapping one of the typed sentinels — errors.Is works through the wrapping so consumers branch on category, not status code:

ErrAuth             // 401, 403
ErrRateLimit        // 429
ErrInvalidRequest   // other 4xx (parent of the next two)
├─ ErrContextLength // prompt / max_tokens exceeds the model's window
└─ ErrPolicyViolation // input rejected by content / safety policy
ErrProvider         // generic provider problem (parent of the next two)
├─ ErrServerError   // 5xx (excluding 529)
└─ ErrOverloaded    // 529 (Anthropic infra overload)

ErrServerError and ErrOverloaded both wrap ErrProvider; ErrContextLength and ErrPolicyViolation both wrap ErrInvalidRequest. Legacy errors.Is(err, ErrProvider) and errors.Is(err, ErrInvalidRequest) callers keep matching the full subtree.

The finer 4xx sentinels are derived by inspecting the response body — provider error schemas don't carry a canonical machine-readable category for "context too long" vs "policy violation," so pi-llm-go pattern-matches the message text. Patterns cover Anthropic / OpenAI / Gemini current shapes. For structured per-provider decoding, branch on apiErr.Body directly.

Sugar helpers and the parsed Retry-After:

for ev, err := range provider.Stream(ctx, req) {
    if err == nil { /* consume ev */ continue }

    var apiErr *llm.APIError
    if errors.As(err, &apiErr) {
        switch {
        case llm.IsRateLimited(err):     // 429 → respect apiErr.RetryAfter
        case llm.IsOverloaded(err):      // 529 → short backoff, consider failover
        case llm.IsServerError(err):     // 5xx → retry, escalate if sustained
        case errors.Is(err, llm.ErrAuth):
        }
    }
    return err
}

APIError.RetryAfter is populated by all four built-in providers when the response carries a Retry-After (RFC 7231 delta-seconds or HTTP-date) or retry-after-ms (OpenAI's millisecond form, which wins when both are present).

Retry middleware

Set Options.Retry on any provider to retry retriable errors (429 / 529 / 5xx / transient network failures) with exponential backoff + full jitter:

p, _ := anthropic.New(anthropic.Options{
    APIKey: key,
    Retry:  &llm.RetryPolicy{MaxAttempts: 4, BaseDelay: time.Second, MaxDelay: 30 * time.Second},
})
// or use the sane defaults:
p2, _ := anthropic.New(anthropic.Options{APIKey: key, Retry: ptr(llm.DefaultRetryPolicy())})

Server-supplied Retry-After hints dominate the exponential schedule (capped at MaxDelay). ErrAuth, ErrInvalidRequest, and context.Canceled/DeadlineExceeded are NOT retried — caller intent always wins.

Scope: retry covers only the initial HTTP attempt. Once a 200 OK lands and streaming begins, the run is committed — a mid-stream connection break terminates the iterator rather than retrying, because resuming would replay events the consumer already saw. Callers needing at-least-once streaming should wrap pi-llm-go in their own idempotent replay layer.

llm.RunWithRetry[T] is exported so callers can build cross-provider fallback / circuit-breaker logic on top.

Telemetry: every retry emits a slog.DebugContext record so silent backoff windows are visible to ops:

DEBUG llm.retry.attempt   attempt=1 max_attempts=4 delay_ms=820 cause=overloaded retry_after_ms=500 error="..."
DEBUG llm.retry.exhausted max_attempts=4 last_cause=overloaded error="..."

Default slog level is INFO so these are silent unless you configure DEBUG. Field names are part of the v1 contract — write a custom slog.Handler to route them to Prometheus counters, dashboards, or wherever else. Otel users can bridge via otelslog.

Cost telemetry

Every completion surfaces a typed Usage value on the final EventMessageEnd (and on the *llm.Message returned by Complete / Accumulate):

msg, _ := llm.Complete(ctx, provider, req)
fmt.Printf("in=%d out=%d cache_write=%d (5m=%d, 1h=%d) cache_read=%d total=%d\n",
    msg.Usage.InputTokens, msg.Usage.OutputTokens,
    msg.Usage.CacheWriteTokens, msg.Usage.CacheWrite5mTokens, msg.Usage.CacheWrite1hTokens,
    msg.Usage.CacheReadTokens, msg.Usage.TotalTokens)

CacheWrite5mTokens and CacheWrite1hTokens are Anthropic-specific — they break CacheWriteTokens down by TTL tier so consumers can detect silent 5min fallback when CacheRetention=long was requested but the model didn't honor the extended TTL:

if req.CacheRetention == llm.CacheRetentionLong &&
    msg.Usage.CacheWrite5mTokens > 0 && msg.Usage.CacheWrite1hTokens == 0 {
    // Model downgraded the cache hold to 5min. Cost projections that
    // assumed a 1h-cached prefix across iterations need to adjust.
}

OpenAI and Gemini leave the TTL-breakdown fields at zero (their cache surfaces are opaque or single-TTL).

llm.ComputeCost applies a built-in pricing table to a Usage value and returns a dollar breakdown:

cost, err := llm.ComputeCost(msg.Usage, req.Model)
if err == nil {
    fmt.Printf("$%.4f total (in=$%.4f out=$%.4f cache_read=$%.4f cache_write_1h=$%.4f)\n",
        cost.Total(), cost.Input, cost.Output, cost.CacheRead, cost.CacheWrite1h)
}

The seed table covers the Claude 4, GPT-5, and Gemini 2.5/3.1 families (verified 2026-05-13). For any model not in the table — older snapshots, deprecated IDs, regional/batch tiers — call llm.RegisterPricing(modelID, llm.Pricing{...}) at startup with rates pulled from your own source. Registered entries override the seed.

The TTL-breakdown fields from Usage carry through automatically: when CacheWrite5mTokens and CacheWrite1hTokens are populated, they price against their respective tiers independently — so silent 5min fallback is reflected in the cost projection without any caller-side branching.

Token counting

Anthropic and Gemini expose a count-tokens endpoint that returns the input-token count without spending an inference call. Providers implement llm.TokenCounter when they support this:

if c, ok := provider.(llm.TokenCounter); ok {
    n, err := c.CountTokens(ctx, req)
    if err == nil {
        fmt.Printf("request would consume %d input tokens\n", n)
    }
}

OpenAI's Chat Completions and Responses providers do NOT implement TokenCounter — OpenAI's tokenization is local-only via tiktoken, which pi-llm-go does not bundle. Callers needing a pre-flight count for an OpenAI-hosted model should run tiktoken themselves.

Examples

Runnable examples in examples/:

• examples/streaming — basic streaming of a text response.
• examples/tool_calling — hand-rolled tool-call loop against get_current_time.
• examples/strict_tool_use — Tool.Strict + Request.ToolChoice forcing schema-valid arguments.
• examples/multimodal — image input on Anthropic + OpenAI.
• examples/multimodal_gemini — text / image / video against Gemini; --video-upload PATH exercises the Files API end-to-end.
• examples/prompt_caching — CacheRetention knob driving Anthropic prompt-cache hits.
• examples/thinking — extended thinking (Effort / BudgetTokens) with ThinkingBlock output.
• examples/openai_responses — OpenAI Responses API with reasoning summaries.
• examples/azure_openai — OpenAI-compatible provider against an Azure OpenAI deployment.
• examples/multi_turn — multi-turn conversation accumulating assistant turns.

Run them with go run ./examples/streaming (set ANTHROPIC_API_KEY first).

Versioning

As of v1.0.0 this package follows semver strictly: the exported API is stable, and no breaking change ships without a major-version (module-path) bump per Go's major-version policy. New providers, new optional Request/Options fields, and new sealed Block/StreamEvent variants are minor releases. See CHANGELOG.md for each release.

License

MIT. See LICENSE.

Acknowledgements

Designed and named after pi-ai by Mario Zechner (TypeScript, MIT). The wire-level vocabulary and event types follow the upstream's lead; the Go-native API surface (interface-based providers, iterator streaming, sealed sum types) is a from-scratch redesign for Go idioms.

Built and maintained by Amit Timalsina with Claude Code assistance — all design decisions and release calls are human-owned.

Documentation

Overview

Package llm is a minimal, provider-agnostic Go interface for streaming LLM completions with tool calling. Built-in providers cover the Anthropic Messages API and OpenAI-compatible Chat Completions APIs (OpenAI, Groq, Together, vLLM, OpenRouter, Ollama, and similar).

Two patterns of use:

• Streaming events: range over LLM.Stream to react to deltas in real time.
• One-shot completion: call Complete to drain the stream and receive the final assistant message.

Cancellation propagates through context.Context. Provider errors surface through the iterator's error return as *APIError wrapping a sentinel (ErrAuth, ErrRateLimit, ErrInvalidRequest, ErrProvider).

pi-llm-go does not execute tools. It declares tool schemas on requests and surfaces ToolCallBlocks on responses. The companion pi-agent-go module adds the execution loop.

Example

Example demonstrates the smallest useful pi-llm-go program: a one-shot Anthropic Claude completion. Replace anthropic.New / ClaudeSonnet4_6 with any other provider to switch backends.

package main

import (
	"context"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, err := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})
	if err != nil {
		panic(err)
	}

	msg, err := llm.Complete(context.Background(), p, llm.Request{
		Model:     anthropic.ClaudeSonnet4_6,
		MaxTokens: 1024,
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "Reply with one word: ready"}}},
		},
	})
	if err != nil {
		panic(err)
	}

	for _, block := range msg.Content {
		if tb, ok := block.(llm.TextBlock); ok {
			fmt.Println(tb.Text)
		}
	}
}

Index

• Variables
• func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]
• func ClassifyInvalidRequest(body []byte) error
• func IsContextLength(err error) bool
• func IsOverloaded(err error) bool
• func IsPolicyViolation(err error) bool
• func IsRateLimited(err error) bool
• func IsRetriable(err error) bool
• func IsServerError(err error) bool
• func ParseRetryAfter(headers http.Header) time.Duration
• func RegisterPricing(model string, p Pricing)
• func RunWithRetry[T any](ctx context.Context, policy RetryPolicy, do func() (T, error)) (T, error)
• func SentinelFor(status int, body []byte) error
• func SentinelForStatus(status int) error
• type APIError
  • func (e *APIError) Error() string
  • func (e *APIError) Unwrap() error
• type Block
• type CacheRetention
• type Cost
  • func ApplyPricing(usage Usage, p Pricing) Cost
  • func ComputeCost(usage Usage, model string) (Cost, error)
  • func (c Cost) Total() float64
• type Effort
• type EventMessageEnd
• type EventMessageStart
• type EventTextDelta
• type EventTextEnd
• type EventTextStart
• type EventThinkingDelta
• type EventThinkingEnd
• type EventThinkingStart
• type EventToolCallDelta
• type EventToolCallEnd
• type EventToolCallStart
• type ImageBlock
  • func (i ImageBlock) Validate() error
• type LLM
• type Message
  • func Complete(ctx context.Context, l LLM, req Request) (*Message, error)
• type Pricing
  • func PricingFor(model string) (Pricing, bool)
• type Request
• type RetryPolicy
  • func DefaultRetryPolicy() RetryPolicy
• type Role
• type StopReason
• type StreamEvent
• type TextBlock
• type ThinkingBlock
• type ThinkingConfig
• type ThinkingDisplay
• type TokenCounter
• type Tool
• type ToolCallBlock
• type ToolChoice
• type ToolChoiceType
• type ToolResultBlock
• type Usage
• type VideoBlock
  • func (v VideoBlock) Validate() error

Examples

• Package
• APIError
• CacheRetention
• Complete
• ComputeCost
• Request (ToolCalling)
• RetryPolicy
• RunWithRetry
• ThinkingConfig
• TokenCounter

Variables

var (
	ErrAuth           = errors.New("llm: authentication failed")
	ErrRateLimit      = errors.New("llm: rate limited")
	ErrInvalidRequest = errors.New("llm: invalid request")
	ErrProvider       = errors.New("llm: provider error")
	// ErrServerError signals a generic 5xx response (excluding 529).
	// Recommended consumer policy: retry with backoff; if sustained
	// past a threshold, surface for engineer escalation.
	ErrServerError = fmt.Errorf("%w: server error (5xx)", ErrProvider)
	// ErrOverloaded signals an Anthropic-style 529 "overloaded"
	// response. Recommended consumer policy: short backoff (~60s)
	// then retry; consider provider fallback if sustained.
	ErrOverloaded = fmt.Errorf("%w: overloaded (529)", ErrProvider)
	// ErrContextLength signals that the prompt + tools + thinking
	// budget exceeded the model's context window (or that max_tokens
	// requested more output than the model can produce in the
	// remaining window). Returns as a 400 in practice. Recommended
	// consumer policy: do NOT retry as-is; truncate, summarize, or
	// route to a longer-context model.
	ErrContextLength = fmt.Errorf("%w: context length exceeded", ErrInvalidRequest)
	// ErrPolicyViolation signals that the request was rejected by the
	// provider's content / safety policy. Returns as a 400 in
	// practice. Recommended consumer policy: do NOT retry; surface
	// to the user or route through a moderation step.
	ErrPolicyViolation = fmt.Errorf("%w: content policy violation", ErrInvalidRequest)
)

Sentinel errors. Wrap via APIError or return directly. Use errors.Is to branch on them in caller retry / fallback logic.

Hierarchy:

ErrProvider           // generic "something provider-side broke"
├─ ErrServerError     // HTTP 5xx (excluding 529)
└─ ErrOverloaded      // HTTP 529 (Anthropic infra overload)

ErrInvalidRequest     // HTTP 4xx (other than 401/403/429)
├─ ErrContextLength   // prompt/output exceeds context window
└─ ErrPolicyViolation // input flagged by provider safety policy

Child sentinels wrap their parents via fmt.Errorf("%w"), so existing callers using errors.Is(err, ErrInvalidRequest) or errors.Is(err, ErrProvider) continue to match the full subtree (backward compatible). The child sentinels add specificity for consumers that need distinct retry / escalation policies.

var ErrUnknownModel = errors.New("llm: unknown model pricing")

ErrUnknownModel is returned by ComputeCost when the model is not in the built-in pricing table and has no caller-registered pricing. Callers can register pricing via RegisterPricing.

Functions

func Accumulate

func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]

Accumulate folds a stream of events into a stream of progressively-built messages. Each yield emits a snapshot of the assistant message as it stands after the most recent event. The final yielded value (when err is nil) is the fully-assembled assistant message.

Callers that only want the final message should prefer Complete. Use Accumulate when intermediate snapshots are useful (e.g. driving a UI that renders each delta against a complete message tree rather than against individual deltas).

Each snapshot is an independent value — internal slices and strings are copied on emit so callers can retain previous snapshots without aliasing.

func ClassifyInvalidRequest

func ClassifyInvalidRequest(body []byte) error

ClassifyInvalidRequest inspects an error response body for known substrings that identify the request as ErrContextLength or ErrPolicyViolation, returning the more-specific sentinel when one applies and ErrInvalidRequest otherwise.

Used by providers when constructing APIError on a 4xx response — the status code maps to ErrInvalidRequest, but the body usually contains a free-form message describing the specific cause. Substring matching is a pragmatic last-resort: provider response schemas don't carry a canonical machine-readable category for "context too long" vs "policy violation" vs generic 4xx, so we pattern-match the message text.

Pattern coverage (lowercased substring matches on body bytes — see contextLengthPatterns / policyViolationPatterns for the live list):

• Context length: phrases like "context length", "context window", "prompt is too long", "too many tokens", "maximum allowed number of output tokens" (Anthropic), "maximum context length is" (OpenAI).
• Policy: "content policy" / "content_policy", "policy violation", "violates", "safety", "moderation", "blocked".

Patterns are intentionally SPECIFIC — e.g. the request field name "max_tokens" on its own is NOT a context-length signal because a generic 400 like "max_tokens must be a positive integer" would otherwise be misclassified. The patterns target phrases that only appear in genuine context-length / policy responses.

Provider-specific schemas (Anthropic's {"error":{"type":"...","message":"..."}}, OpenAI's {"error":{"code":"context_length_exceeded","message":"..."}}, Gemini's Google-style error envelope) all surface human-readable messages that hit these patterns. Callers needing structured per-provider decoding should branch on apiErr.Body themselves rather than rely on this classifier.

func IsContextLength

func IsContextLength(err error) bool

IsContextLength reports whether err is a context-length-exceeded error. Sugar for errors.Is(err, ErrContextLength).

func IsOverloaded

func IsOverloaded(err error) bool

IsOverloaded reports whether err is an Anthropic-style 529 overloaded response.

func IsPolicyViolation

func IsPolicyViolation(err error) bool

IsPolicyViolation reports whether err is a content-policy rejection. Sugar for errors.Is(err, ErrPolicyViolation).

func IsRateLimited

func IsRateLimited(err error) bool

IsRateLimited reports whether err (or anything in its Unwrap chain) is a 429 rate-limit error. Equivalent to errors.Is(err, ErrRateLimit) — sugar for the common caller-side branch.

func IsRetriable

func IsRetriable(err error) bool

IsRetriable reports whether err is one of the retriable categories. Used internally by RunWithRetry; exported so callers can build their own retry layer on top.

Note: callers who reimplement the retry loop on top of IsRetriable will NOT inherit RunWithRetry's slog telemetry — the `llm.retry.attempt` / `llm.retry.exhausted` records are emitted by RunWithRetry itself, not by IsRetriable.

Returns true for:

• ErrRateLimit (429)
• ErrOverloaded (529)
• ErrServerError (other 5xx)
• net.Error and *net.OpError (DNS, connect refused, TLS handshake)

Returns false for:

• nil
• context.Canceled / context.DeadlineExceeded
• ErrAuth, ErrInvalidRequest, ErrContextLength, ErrPolicyViolation
• any other error (including unwrapped *APIError with non-retriable Inner)

func IsServerError

func IsServerError(err error) bool

IsServerError reports whether err is a generic 5xx response (excluding 529; check IsOverloaded for that case).

func ParseRetryAfter

func ParseRetryAfter(headers http.Header) time.Duration

ParseRetryAfter extracts a wait hint from a provider response's Retry-After / retry-after-ms headers. Returns 0 if no header is present or none of them parse.

Header precedence (`retry-after-ms` wins when both are present because it carries sub-second precision):

• retry-after-ms: integer milliseconds (OpenAI convention).
• Retry-After: integer seconds (RFC 7231 delta-seconds).
• Retry-After: HTTP-date (RFC 7231; computed as now() delta).

Negative deltas (HTTP-date already in the past) are clamped to 0. Callers should still apply their own minimum bound — a 0-second server hint usually means "as soon as you can" but pummeling the API immediately is rarely productive.

func RegisterPricing

func RegisterPricing(model string, p Pricing)

RegisterPricing registers (or overrides) pricing for a model ID. Safe for concurrent use. Registered entries take precedence over the built-in seed table.

Production callers maintaining their own pricing source should call RegisterPricing once at startup for every model they bill against, rather than relying on the built-in table.

func RunWithRetry

func RunWithRetry[T any](ctx context.Context, policy RetryPolicy, do func() (T, error)) (T, error)

RunWithRetry calls do() up to policy.MaxAttempts times, sleeping policy.nextDelay between attempts when do() returns a retriable error. Returns the first successful result, the first non-retriable error, or the last result if all attempts fail.

Honors ctx: sleeping is interruptible via ctx.Done(), and a canceled ctx short-circuits before the next attempt.

do MUST construct any single-use resources (request body readers, fresh http.Request values) on each call — RunWithRetry invokes do from scratch each attempt.

Used internally by provider implementations to wrap their HTTP attempt loops; exported so callers building higher-level retry (e.g. cross-provider fallback) can compose on top.

Telemetry: every retry fires a structured slog DEBUG record via slog.Default(). The default slog handler is silent at DEBUG so consumers see nothing by default; configure DEBUG-level handling to surface them. Routing: filter on Message prefix "llm.retry." in your slog.Handler to send pi-llm-go retry records elsewhere without affecting your app default. Otel users can bridge via the otelslog package.

"llm.retry.attempt"   — emitted for each retriable failure that
                        will be retried. Fields:
                          attempt          (int, 1-indexed)
                          max_attempts     (int)
                          delay_ms         (int64, planned sleep)
                          cause            (string, see causeOf)
                          retry_after_ms   (int64, when server hint present)
                          error            (string, truncated to ~256 chars)
"llm.retry.exhausted" — emitted once when all attempts surrender.
                        Fields: max_attempts, last_cause, error.

Field names are part of pi-llm-go's v1 public contract — consumers writing custom slog.Handlers (Prometheus counter, dashboard enrichment, etc.) can rely on this set not churning post-v1.

No telemetry record is emitted for a successful first attempt (zero noise on the happy path) or for non-retriable errors (the caller sees the error immediately; the surface needs no narration).

Example

ExampleRunWithRetry shows the exported retry primitive used by every provider internally. Useful for building cross-provider fallback or circuit-breaker logic on top.

package main

import (
	"context"
	"time"

	llm "github.com/amit-timalsina/pi-llm-go"
)

func main() {
	policy := llm.RetryPolicy{MaxAttempts: 3, BaseDelay: time.Second, MaxDelay: 10 * time.Second}

	result, err := llm.RunWithRetry(context.Background(), policy, func() (string, error) {
		// Your operation here. Return a retriable error
		// (errors.Is(err, llm.ErrRateLimit) etc.) to trigger retry;
		// return any other error to abort immediately.
		return "ok", nil
	})
	_ = result
	_ = err
}

func SentinelFor

func SentinelFor(status int, body []byte) error

SentinelFor returns the most specific sentinel for an HTTP error response, combining status-code mapping with body-pattern inspection for 400 responses. Equivalent to SentinelForStatus(status) for non-400 responses; for 400s, upgrades to ErrContextLength / ErrPolicyViolation when the body matches a known pattern (see ClassifyInvalidRequest).

Prefer this over SentinelForStatus when constructing APIError values — the finer sentinels are part of pi-llm-go's public error surface.

func SentinelForStatus

func SentinelForStatus(status int) error

SentinelForStatus maps an HTTP status code to the matching sentinel error. Used by provider implementations when constructing APIError.

Status to sentinel:

401, 403       → ErrAuth
429            → ErrRateLimit
other 4xx      → ErrInvalidRequest
529            → ErrOverloaded (Anthropic-style)
other 5xx      → ErrServerError
otherwise      → ErrProvider

ErrServerError and ErrOverloaded both wrap ErrProvider, so legacy errors.Is(err, ErrProvider) keeps working for 5xx / 529 responses.

Types

type APIError

type APIError struct {
	Provider string
	Status   int
	Body     []byte
	Inner    error
	// RetryAfter, when > 0, is the server's hint for how long the
	// caller should wait before retrying. Parsed from the response's
	// Retry-After header (RFC 7231 seconds-or-HTTP-date) or, for
	// OpenAI-family providers, the `retry-after-ms` header. Zero
	// means "no header present" — callers fall back to their own
	// backoff schedule.
	RetryAfter time.Duration
}

APIError wraps a non-2xx HTTP response from a provider. The Inner field is one of the sentinel errors above so that errors.Is works through the wrapping. Status and Body let callers inspect the raw failure (e.g. to parse a structured provider error payload). RetryAfter, when non-zero, is the parsed value of the response's Retry-After or retry-after-ms header — populated by providers for 429 / 529 responses to support caller-side rate-limit / overload backoff.

Example

ExampleAPIError shows error-classification with errors.Is for typed retry / escalation policies. Sugar helpers (IsRateLimited, IsOverloaded, IsServerError, IsContextLength, IsPolicyViolation, IsRetriable) avoid string-matching on err.Error().

package main

import (
	"context"
	"errors"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})
	req := llm.Request{Model: anthropic.ClaudeSonnet4_6, MaxTokens: 1024}

	for ev, err := range p.Stream(context.Background(), req) {
		_ = ev
		if err == nil {
			continue
		}

		var apiErr *llm.APIError
		if errors.As(err, &apiErr) {
			switch {
			case llm.IsRateLimited(err):
				fmt.Printf("rate-limited; retry after %s\n", apiErr.RetryAfter)
			case llm.IsOverloaded(err):
				fmt.Println("overloaded; consider provider fallback")
			case llm.IsServerError(err):
				fmt.Println("5xx; retry with backoff")
			case llm.IsContextLength(err):
				fmt.Println("prompt too long; truncate or route to longer-context model")
			case llm.IsPolicyViolation(err):
				fmt.Println("rejected by safety policy; do not retry")
			case errors.Is(err, llm.ErrAuth):
				fmt.Println("auth failed; check API key")
			}
		}
		return
	}
}

func (*APIError) Error

func (e *APIError) Error() string

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

type Block

type Block interface {
	// contains filtered or unexported methods
}

Block is the sealed sum type for message content. Concrete implementations are TextBlock, ThinkingBlock, ToolCallBlock, and ToolResultBlock — all defined in this package. The unexported marker method keeps the set closed: provider converters need exhaustive type-switches to serialize content correctly, so new block types must be added inside the package.

type CacheRetention

type CacheRetention string

CacheRetention controls Anthropic prompt-cache breakpoint placement. It is a single-knob abstraction: callers pick a retention tier and the provider decides where to place the cache_control markers on the underlying wire format.

The marker tells Anthropic "everything from the start of the request up to and including this block is cacheable; on a subsequent request with byte-identical content up to this marker, return a cache hit and bill cache-read rates instead of full input rates."

Behavior by value:

• CacheRetentionNone (zero value, "") — no markers emitted.
• CacheRetentionShort — ephemeral markers with the default ~5 minute lifetime, placed at: (a) the System prompt's trailing block, (b) the final Tool in Request.Tools, (c) the last block (any type) of the most recent user-role message. The last-block placement is type-agnostic so that subsequent calls in a tool loop reuse the cached tool_result round-trip instead of re-billing it.
• CacheRetentionLong — same placement as Short with TTL "1h" and the "extended-cache-ttl-2025-04-11" beta header auto-attached to the outgoing HTTP request.

OpenAI's Chat Completions and Responses providers silently ignore CacheRetention — OpenAI caches automatically with no caller-side breakpoint API.

1h-TTL availability and fallback behavior:

All currently-shipped Claude 4 family models (Opus 4.7, Sonnet 4.6, Haiku 4.5) support 1h cache TTL. Older Claude 3.x models accept the beta header but may silently downgrade the hold to the 5-minute default — Anthropic does NOT error on an unsupported model. The silent fallback is a real cost-budgeting hazard for callers that assumed a 1h-cached prefix would survive across long iterations (Noumenal issue #12).

Detect silent fallback via the Usage breakdown: when CacheRetention=long was requested, inspect the response message's Usage.CacheWrite5mTokens vs CacheWrite1hTokens. If the 5m count is non-zero and the 1h count is zero, the model honored the request at 5min, not 1h — adjust cost projections accordingly. The two fields are populated from Anthropic's cache_creation response breakdown; other providers leave them at 0.

The heuristic (5m>0 && 1h==0) assumes UNIFORM TTL placement across all cache_control markers in the request, which CacheRetention=long guarantees today (every marker carries ttl:"1h"). If a future per-block-TTL feature lets callers mix 5min and 1h breakpoints in one request, this diagnostic would produce false positives and would need a different signal — likely a per-block annotation rather than aggregate counts.

Note that as of March 2026, Anthropic regressed the DEFAULT ephemeral TTL from 60min to 5min; the 1h tier is now opt-in via CacheRetentionLong + the extended-cache-ttl-2025-04-11 beta header (which the Anthropic provider auto-attaches when CacheRetention=long).

The caller owns prompt determinism: cached sections must be byte-stable across iterations for the cache to hit. Any change (timestamps, map iteration order, reordered items) invalidates the cache from that point forward in the request.

See https://docs.claude.com/en/docs/build-with-claude/prompt-caching for the full discipline.

Example

ExampleCacheRetention shows enabling Anthropic prompt caching at the 1-hour TTL tier. Cache breakpoints are auto-placed at the end of System / last Tool / last user message block. Cache hits surface via Usage.CacheReadTokens; the per-TTL breakdown (CacheWrite5mTokens / CacheWrite1hTokens) lets callers detect silent 5min fallback when a model doesn't honor the extended TTL.

package main

import (
	"context"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

	req := llm.Request{
		Model:          anthropic.ClaudeSonnet4_6,
		MaxTokens:      1024,
		System:         "Long static system prompt that's identical across iterations.",
		CacheRetention: llm.CacheRetentionLong, // 1h TTL; CacheRetentionShort for 5m
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "hello"}}},
		},
	}
	msg, _ := llm.Complete(context.Background(), p, req)

	if req.CacheRetention == llm.CacheRetentionLong &&
		msg.Usage.CacheWrite5mTokens > 0 && msg.Usage.CacheWrite1hTokens == 0 {
		fmt.Println("WARNING: model fell back to 5min cache — 1h tier not honored")
	}
}

const (
	// CacheRetentionNone disables prompt caching for this request. This is
	// the zero value of CacheRetention; an unset field and an explicit
	// CacheRetentionNone are byte-identical and produce no cache_control
	// markers.
	CacheRetentionNone CacheRetention = ""

	// CacheRetentionShort places ephemeral cache breakpoints with the
	// default ~5 minute lifetime. The right default for iterative agent
	// loops where the prefix is reused within a single session.
	CacheRetentionShort CacheRetention = "short"

	// CacheRetentionLong places ephemeral cache breakpoints with the 1-hour
	// TTL and auto-attaches the "extended-cache-ttl-2025-04-11" beta header.
	// For long-lived static prefixes (large system prompts, big tool sets)
	// that survive across many sessions.
	CacheRetentionLong CacheRetention = "long"
)

type Cost

type Cost struct {
	Input        float64
	Output       float64
	CacheRead    float64
	CacheWrite5m float64
	CacheWrite1h float64
}

Cost is the dollar breakdown of a single completion's Usage. Sum via Total() for a single number.

Categories track Usage fields directly so the diagnostic from Usage.CacheWrite5mTokens / CacheWrite1hTokens (silent 5min fallback detection) carries into the cost projection.

func ApplyPricing

func ApplyPricing(usage Usage, p Pricing) Cost

ApplyPricing is the pure-arithmetic form of ComputeCost: takes a caller-supplied Pricing rather than a model lookup. Useful when the caller maintains their own pricing source (e.g. a database, models.dev poll) outside the built-in table.

func ComputeCost

func ComputeCost(usage Usage, model string) (Cost, error)

ComputeCost applies the registered pricing for model to usage and returns the dollar breakdown. Returns ErrUnknownModel wrapped with the model ID when no pricing is registered.

The Anthropic-specific TTL breakdown on Usage.CacheWrite5mTokens / CacheWrite1hTokens is honored: each tier prices against its own rate so silent 5min fallback (Issue #12 diagnostic) is reflected in the cost projection automatically.

When Usage.CacheWriteTokens > 0 but both CacheWrite5mTokens and CacheWrite1hTokens are 0 (OpenAI / Gemini, or older Anthropic SDK versions that didn't surface the breakdown), the writeable tokens fall through to the 5m rate as a best-effort. On non-Anthropic providers this rate is 0, so the cost is 0 regardless — matching the wire reality (OpenAI's cache is automatic; Gemini doesn't separately meter the write).

Example

ExampleComputeCost shows projecting dollar cost from a Usage record. Pricing for Claude 4 / GPT-5 / Gemini 2.5+3.1 families ships in the seed table; register your own via llm.RegisterPricing for other models or for batch / regional pricing tiers.

package main

import (
	"fmt"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	usage := llm.Usage{
		InputTokens:        50_000,
		OutputTokens:       2_000,
		CacheReadTokens:    100_000,
		CacheWrite1hTokens: 30_000,
	}
	cost, err := llm.ComputeCost(usage, anthropic.ClaudeSonnet4_6)
	if err != nil {
		return
	}
	fmt.Printf("$%.4f total (in=$%.4f out=$%.4f cache_read=$%.4f cache_1h=$%.4f)\n",
		cost.Total(), cost.Input, cost.Output, cost.CacheRead, cost.CacheWrite1h)
}

func (Cost) Total

func (c Cost) Total() float64

Total returns the sum of all cost categories.

type Effort

type Effort string

Effort is the adaptive-thinking depth enum. Honored by Anthropic on Opus 4.6+ via the top-level `output_config.effort` field. Empty string is the zero value and means "don't request adaptive thinking"; the provider falls back to BudgetTokens (manual mode) if non-zero.

The OpenAI Responses provider has its own per-provider ReasoningEffort enum on its Options; the two are intentionally separate because the values are not 1:1.

const (
	EffortLow    Effort = "low"
	EffortMedium Effort = "medium"
	EffortHigh   Effort = "high"
)

Effort levels recognized by Anthropic's adaptive thinking. Match the wire enum exactly — do NOT alias to numbers or "minimal" / "xhigh" without confirming the model accepts them.

type EventMessageEnd

type EventMessageEnd struct {
	StopReason StopReason
	Usage      Usage
}

EventMessageEnd is the terminal event. It carries the normalized stop reason and the final usage tally.

type EventMessageStart

type EventMessageStart struct {
	Model string
}

EventMessageStart is emitted once at the start of an assistant turn, before any block events.

type EventTextDelta

type EventTextDelta struct {
	BlockIndex int
	Delta      string
}

EventTextDelta appends Delta to the text in the block at BlockIndex.

type EventTextEnd

type EventTextEnd struct {
	BlockIndex int
}

EventTextEnd marks the end of the TextBlock at BlockIndex.

type EventTextStart

type EventTextStart struct {
	BlockIndex int
}

EventTextStart marks the beginning of a TextBlock.

type EventThinkingDelta

type EventThinkingDelta struct {
	BlockIndex int
	Delta      string
}

EventThinkingDelta appends Delta to the thinking block at BlockIndex.

type EventThinkingEnd

type EventThinkingEnd struct {
	BlockIndex int
	Signature  string
}

EventThinkingEnd marks the end of the ThinkingBlock. Signature is the opaque provider-supplied token to round-trip on follow-up messages.

type EventThinkingStart

type EventThinkingStart struct {
	BlockIndex int
}

EventThinkingStart marks the beginning of a ThinkingBlock.

type EventToolCallDelta

type EventToolCallDelta struct {
	BlockIndex int
	Delta      string
}

EventToolCallDelta delivers a fragment of the streaming JSON arguments. Callers that need the assembled arguments should wait for EventToolCallEnd rather than accumulate Delta bytes themselves (provider JSON delta framing is not guaranteed to be at value boundaries).

type EventToolCallEnd

type EventToolCallEnd struct {
	BlockIndex int
	Arguments  json.RawMessage
}

EventToolCallEnd marks the end of a ToolCallBlock and carries the assembled arguments.

type EventToolCallStart

type EventToolCallStart struct {
	BlockIndex int
	ID         string
	Name       string
}

EventToolCallStart marks the beginning of a ToolCallBlock. ID and Name are available immediately; arguments stream as deltas and are emitted in fully assembled form on EventToolCallEnd.

type ImageBlock

type ImageBlock struct {
	// Data is the raw base64-encoded image bytes. Do NOT include the
	// "data:<mime>;base64," prefix — providers add it where required.
	Data string

	// MimeType is the image's MIME type (e.g. "image/png"). Required.
	MimeType string
}

ImageBlock holds image data for multimodal input. Data is the raw base64-encoded image bytes (no "data:" URI prefix); MimeType is the standard MIME identifier (e.g. "image/png"). Providers convert to their on-wire format at the boundary.

pi-llm-go does NOT fetch image URLs. Callers that want to attach a remote image must download it themselves first, then construct an ImageBlock with the resulting bytes encoded. This keeps the library network-free except for the LLM provider call itself — no surprise timeouts, no surprise 404s mid-stream.

Portable MIME types accepted by every built-in provider:

• "image/jpeg"
• "image/png"
• "image/gif"
• "image/webp"

Other types may work on specific providers (e.g. OpenAI accepts more) but pi-llm-go does not pre-validate — the provider returns an ErrInvalidRequest if the type is unsupported.

v0.3.0 supports ImageBlock as USER-message input only. Assistant image output is provider-specific and a separate, future feature.

func (ImageBlock) Validate

func (i ImageBlock) Validate() error

Validate enforces the ImageBlock contract: Data must be raw base64-encoded bytes (without the "data:<mime>;base64," URI prefix) and MimeType must be set. Providers call this at the wire boundary and surface a wrapped error if the contract is violated.

type LLM

type LLM interface {
	Stream(ctx context.Context, req Request) iter.Seq2[StreamEvent, error]
}

LLM is the provider-agnostic streaming interface. Anthropic and OpenAI providers implement this; third-party providers may do the same to plug into Complete and Accumulate.

Implementations must:

• Honor cancellation of ctx by terminating the underlying HTTP request and yielding (nil, ctx.Err()) from the iterator.
• Surface HTTP errors as *APIError values via the iterator's error half.
• Emit events in the order documented on StreamEvent.

type Message

type Message struct {
	Role    Role
	Content []Block

	Usage      Usage
	StopReason StopReason
	Model      string
}

Message is one turn in the transcript. Content holds a sequence of blocks — the model emits assistant messages with mixed text / thinking / tool-call content; the caller sends user messages with text and tool messages with tool-result content.

Usage, StopReason, and Model are populated on assistant messages produced by Complete or Accumulate; they are zero on user / tool messages and on messages sent into Stream.

func Complete

func Complete(ctx context.Context, l LLM, req Request) (*Message, error)

Complete drains a streaming completion and returns the final assistant message. It is equivalent to iterating Stream and folding each event into a Message via Accumulate.

Returns the partial message and a wrapped error if the stream terminates early; the partial may be useful for debugging or replay.

Example

ExampleComplete shows a one-shot synchronous completion. Use this when you don't care about per-token streaming and just need the final assistant Message.

package main

import (
	"context"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

	msg, err := llm.Complete(context.Background(), p, llm.Request{
		Model:     anthropic.ClaudeSonnet4_6,
		MaxTokens: 1024,
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "hello"}}},
		},
	})
	if err != nil {
		return
	}
	fmt.Println(msg.Usage.TotalTokens, "tokens")
}

type Pricing

type Pricing struct {
	// Input is the per-million-token cost for non-cached input tokens.
	Input float64

	// Output is the per-million-token cost for output tokens. Includes
	// thinking tokens on Anthropic (extended thinking is billed as
	// output).
	Output float64

	// CacheRead is the per-million-token cost for tokens served from
	// a cache hit. On Anthropic this is 0.1× Input by policy; on
	// OpenAI it's roughly 0.1× Input (the "cached input" rate); on
	// Gemini it's the published "context caching" rate.
	CacheRead float64

	// CacheWrite5m is the per-million-token cost for tokens cached at
	// the default ~5 minute TTL. Anthropic-specific (1.25× Input today).
	// Other providers leave this at 0.
	CacheWrite5m float64

	// CacheWrite1h is the per-million-token cost for tokens cached at
	// the extended 1-hour TTL. Anthropic-specific (2× Input today,
	// gated on the extended-cache-ttl-2025-04-11 beta header which the
	// provider auto-attaches when CacheRetention=long). Other providers
	// leave this at 0.
	CacheWrite1h float64
}

Pricing holds the per-token cost rates for a model. All rates are in dollars per million tokens (the industry-standard quoting unit).

Rate semantics by provider:

• Anthropic publishes input + output rates plus three cache rates (5m write, 1h write, read). The cache rates are deterministic multipliers on the base input rate today (1.25× / 2× / 0.1×) but the multiplier policy could change, so the seed table stores them explicitly.
• OpenAI publishes input + output + a single "cached input" rate; CacheRead is the appropriate field. OpenAI has no caller-visible cache-write category — cache reads are billed at the discounted rate; writes implicit in standard input.
• Gemini publishes input + output + a single context-caching rate. CacheRead is the appropriate field; CacheWrite5m and CacheWrite1h stay 0 (Gemini's cache is single-TTL and not separately metered for the write).

Pricing is forward-flexible: a provider that ships a new cache tier can populate the unused field without breaking existing callers.

func PricingFor

func PricingFor(model string) (Pricing, bool)

PricingFor returns the registered pricing for a model ID and true, or zero Pricing and false if the model is unknown.

Lookup order:

1. Pricing registered via RegisterPricing (caller-overrideable).
2. The built-in seed table (a small set of canonical models, last verified against the provider docs on the build date — see pricing_seed.go for the verification date).

Pricing changes. Production callers SHOULD verify rates against the provider's current pricing page and call RegisterPricing for any model they care about, rather than trusting the built-in table across upgrades.

type Request

type Request struct {
	Model       string
	System      string
	Messages    []Message
	Tools       []Tool
	Temperature *float64
	MaxTokens   int
	Thinking    *ThinkingConfig
	StopReasons []string

	// ToolChoice controls whether and which tool the model must call
	// on this turn. nil preserves the provider's default behavior
	// (auto when Tools is non-empty, none otherwise). See the
	// ToolChoice godoc for per-provider wire mapping and the
	// Anthropic extended-thinking incompatibility caveat.
	ToolChoice *ToolChoice

	// CacheRetention controls Anthropic prompt-cache breakpoint placement.
	// When unset or "none", no cache markers are emitted. When "short" or
	// "long", the Anthropic provider auto-places ephemeral cache_control
	// markers at the static prefix boundary: the last block of the System
	// prompt, the final Tool in Tools, and the last text block of the most
	// recent user message. "long" additionally selects the 1h TTL and
	// auto-attaches the extended-cache-ttl-2025-04-11 beta header.
	//
	// Ignored by OpenAI providers (their cache is automatic and opaque).
	CacheRetention CacheRetention
}

Request is the common payload for a completion. Provider-specific tunables that have no portable meaning live on the provider's own Options struct (passed to its constructor), not here.

Temperature is a pointer so the zero value can be distinguished from "unset"; callers that want temperature=0 must set *Temperature to 0.

Example (ToolCalling)

ExampleRequest_toolCalling shows declaring a tool, receiving the model's ToolCallBlock, and shipping a ToolResultBlock back on the next turn. For a built-in loop that does this for you, see https://github.com/amit-timalsina/pi-agent-go.

package main

import (
	"context"
	"encoding/json"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

	schema := json.RawMessage(`{
		"type":"object",
		"properties":{"city":{"type":"string"}},
		"required":["city"]
	}`)
	req := llm.Request{
		Model:     anthropic.ClaudeSonnet4_6,
		MaxTokens: 1024,
		Tools:     []llm.Tool{{Name: "get_weather", Description: "Get weather for a city.", InputSchema: schema}},
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "What's the weather in Tokyo?"}}},
		},
	}

	msg, err := llm.Complete(context.Background(), p, req)
	if err != nil {
		return
	}
	for _, block := range msg.Content {
		if tc, ok := block.(llm.ToolCallBlock); ok {
			fmt.Printf("model wants to call %s with %s\n", tc.Name, tc.Arguments)
			// Execute the tool, then send ToolResultBlock on next turn:
			req.Messages = append(req.Messages, *msg, llm.Message{
				Role: llm.RoleUser,
				Content: []llm.Block{llm.ToolResultBlock{
					ToolCallID: tc.ID,
					Content:    `{"temp": 22, "unit": "C"}`,
				}},
			})
		}
	}
}

type RetryPolicy

type RetryPolicy struct {
	// MaxAttempts caps total tries INCLUDING the first attempt. So
	// MaxAttempts=4 means up to 3 retries after the initial failure.
	// Zero or negative disables retry.
	MaxAttempts int

	// BaseDelay is the initial backoff before the first retry.
	// Subsequent attempts double the delay (exponential) up to MaxDelay.
	// Zero defaults to 1 second.
	BaseDelay time.Duration

	// MaxDelay caps each backoff. Server-supplied Retry-After hints
	// from APIError.RetryAfter are honored UP TO this cap (an
	// upstream "retry in 1 hour" hint becomes "wait MaxDelay seconds"
	// to bound the worst case). Zero defaults to 30 seconds.
	MaxDelay time.Duration
}

RetryPolicy configures provider-side retry on retriable errors. Zero-value `RetryPolicy{}` (MaxAttempts == 0) disables retry — the provider runs each Stream / CountTokens call exactly once and surfaces the first error. Set `MaxAttempts >= 2` to opt in.

Wire via the provider's Options.Retry field:

p, _ := anthropic.New(anthropic.Options{
    APIKey: key,
    Retry:  &llm.RetryPolicy{MaxAttempts: 4, BaseDelay: time.Second, MaxDelay: 30 * time.Second},
})

Retry SCOPE: only the initial HTTP attempt is retriable. Once the streaming response begins yielding events (200 OK + first SSE frame parsed) the run is committed — a mid-stream connection break terminates the iterator with an error rather than retrying, because resuming would replay events the consumer already saw. Callers that need at-least-once streaming should wrap pi-llm-go in their own idempotent replay layer.

Retriable categories: ErrRateLimit (429), ErrOverloaded (529), ErrServerError (other 5xx), and transient network errors (DNS, connection refused, TLS handshake timeout, etc.). ErrAuth, ErrInvalidRequest, and context.Canceled / context.DeadlineExceeded are NOT retried.

Example

ExampleRetryPolicy wires retry middleware into a provider. Retries retriable errors (429 / 529 / 5xx / transient network) with exponential backoff + full jitter; honors server-supplied Retry-After hints up to MaxDelay. Only the initial HTTP attempt is retried — mid-stream connection breaks are NOT replayed.

package main

import (
	"os"
	"time"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{
		APIKey: os.Getenv("ANTHROPIC_API_KEY"),
		Retry: &llm.RetryPolicy{
			MaxAttempts: 4,
			BaseDelay:   time.Second,
			MaxDelay:    30 * time.Second,
		},
	})
	_ = p

	// Or use the sane defaults:
	defaults := llm.DefaultRetryPolicy()
	p2, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY"), Retry: &defaults})
	_ = p2
}

func DefaultRetryPolicy

func DefaultRetryPolicy() RetryPolicy

DefaultRetryPolicy returns a sane starting point for production use: 4 attempts (3 retries), 1s base, 30s cap. Modify as needed before passing to the provider's Options.

type Role

type Role string

Role enumerates message roles in a transcript. RoleTool messages carry tool results back to the model and may hold only ToolResultBlock content.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type StopReason

type StopReason string

StopReason is the normalized reason a model stopped generating. Provider-specific stop reasons map to one of these; unmappable values surface as errors rather than leaking provider strings.

const (
	StopReasonEnd       StopReason = "end"        // natural end of turn
	StopReasonMaxTokens StopReason = "max_tokens" // hit MaxTokens cap
	StopReasonToolUse   StopReason = "tool_use"   // model requested tool calls
	StopReasonStop      StopReason = "stop"       // matched a stop sequence
)

type StreamEvent

type StreamEvent interface {
	// contains filtered or unexported methods
}

StreamEvent is the sealed sum type emitted during a streaming completion. Errors flow through the iterator's error half rather than as event values, so consumers do not need an EventError variant.

Event order for one assistant turn:

EventMessageStart
  ( EventTextStart, EventTextDelta*, EventTextEnd
  | EventThinkingStart, EventThinkingDelta*, EventThinkingEnd
  | EventToolCallStart, EventToolCallDelta*, EventToolCallEnd
  )*
EventMessageEnd

BlockIndex on per-block events is the position of the block inside the emitted message's Content slice — it lets consumers route deltas when rendering or reconstructing incrementally.

type TextBlock

type TextBlock struct {
	Text string
}

TextBlock holds plain text content.

type ThinkingBlock

type ThinkingBlock struct {
	Thinking  string
	Signature string
}

ThinkingBlock holds an extended-thinking segment emitted by reasoning models. Signature is an opaque provider-supplied token that must be preserved and replayed for multi-turn thinking continuity (Anthropic).

type ThinkingConfig

type ThinkingConfig struct {
	// Effort controls thinking depth on adaptive-thinking models (Opus
	// 4.6+, REQUIRED on 4.7+). Set to one of the Effort* constants.
	// Empty string means "don't request adaptive thinking"; provider
	// will fall back to BudgetTokens (manual mode) if non-zero.
	//
	// On Anthropic the wire shape is `{ "thinking": { "type":
	// "adaptive" }, "output_config": { "effort": <Effort> } }` (note
	// output_config is a TOP-LEVEL request field, not nested under
	// thinking).
	Effort Effort

	// BudgetTokens is the manual-mode thinking-token cap. Required on
	// Opus 4.5 and older / Sonnet 3.7. Deprecated on Opus/Sonnet 4.6.
	// Rejected on Opus 4.7+ — those models REQUIRE Effort and return
	// a 400 if budget_tokens appears on the wire.
	//
	// Anthropic requires Request.MaxTokens > BudgetTokens (thinking
	// tokens count against max_tokens). A common safe choice is
	// MaxTokens == BudgetTokens * 2.
	//
	// Provider minimum: Anthropic refuses BudgetTokens < 1024.
	//
	// When both Effort and BudgetTokens are set, Effort wins (adaptive
	// shape emitted, BudgetTokens ignored). This lets callers pre-set
	// both during the migration without per-call branching.
	BudgetTokens int

	// Display controls whether adaptive-thinking models emit summarized
	// thinking TEXT or signature-only blocks. It maps to Anthropic's
	// `thinking.display` field (nested under thinking, NOT output_config
	// — that trap is why Effort and Display land in different wire
	// places).
	//
	// Empty string means "don't send display" (provider default). The
	// provider default on Opus 4.7+ / Sonnet 5 / Fable 5 is "omitted":
	// thinking blocks stream with EMPTY text (signature only), so a
	// consumer capturing EventThinkingDelta records nothing. Set
	// DisplaySummarized to receive summarized thinking text via
	// thinking_delta events.
	//
	// Display only takes effect on the adaptive shape (set alongside
	// Effort). It is ignored in manual mode (BudgetTokens only): older
	// models return thinking text by default and don't accept the field.
	Display ThinkingDisplay
}

ThinkingConfig enables extended thinking on supported models.

Per-provider behavior of the two fields:

• **Anthropic**: dispatches on the field that's set. Effort emits the adaptive shape (Opus 4.6+, required on 4.7+). BudgetTokens emits the manual shape (Opus 4.5- / Sonnet 3.7, deprecated on 4.6 family).
• **Gemini**: honors BudgetTokens only (mapped to thinkingBudget; -1=dynamic, 0=disabled). Effort is currently ignored on Gemini because Gemini's wire shape is an integer budget rather than an enum; revisit if Gemini ships an effort-style knob.
• **OpenAI Chat Completions**: ignores the entire field; reasoning-effort dialects vary across compatible hosts and don't map portably.
• **OpenAI Responses**: ignores this field; the provider routes reasoning-effort via its own Options.ReasoningEffort enum.

Anthropic has two on-wire shapes for extended thinking, gated by model:

• **Adaptive** (Opus 4.7+, recommended on Opus/Sonnet 4.6+) — `thinking.type = "adaptive"` plus a TOP-LEVEL `output_config.effort` enum. The model decides how many thinking tokens to spend within the effort bucket. REQUIRED on Opus 4.7+; manual mode returns 400. An optional `thinking.display` opts back into summarized thinking text (empty text is the default on Opus 4.7+); see Display.
• **Manual** (Opus 4.5- and Sonnet 3.7, deprecated on 4.6 family) — `thinking.type = "enabled"` plus `thinking.budget_tokens`. The caller pins the exact token cap.

Set the field that matches the model's expectation. When both Effort and BudgetTokens are set on the same request, the provider emits the adaptive shape (Effort wins) — adaptive is the future, manual is deprecated.

See closes #20 for the live failure that motivated the dispatch.

Example

ExampleThinkingConfig shows the two Anthropic extended-thinking shapes pi-llm-go supports. Pick the shape that matches the model:

• Opus 4.7 / Opus 4.6 / Sonnet 4.6: set Effort.
• Opus 4.5 / Sonnet 4.5 / Sonnet 3.7: set BudgetTokens.

When both fields are set, Effort wins (adaptive shape emitted) — useful when a single config object flows through multiple models during a migration.

package main

import (
	"context"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

	// Adaptive thinking (Opus 4.7+): model decides depth within Effort.
	adaptiveReq := llm.Request{
		Model:     anthropic.ClaudeOpus4_7,
		MaxTokens: 4096,
		Thinking:  &llm.ThinkingConfig{Effort: llm.EffortMedium},
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "Reason carefully about X."}}},
		},
	}
	_, _ = llm.Complete(context.Background(), p, adaptiveReq)

	// Manual thinking (Opus 4.5 and older): caller pins the token cap.
	manualReq := llm.Request{
		Model:     "claude-opus-4-5",
		MaxTokens: 4096,
		Thinking:  &llm.ThinkingConfig{BudgetTokens: 2048},
		Messages: []llm.Message{
			{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "Reason carefully about X."}}},
		},
	}
	_, _ = llm.Complete(context.Background(), p, manualReq)
}

type ThinkingDisplay

type ThinkingDisplay string

ThinkingDisplay controls the visibility of adaptive-thinking text on Anthropic. It maps to the `thinking.display` wire field. Empty string is the zero value and means "don't send display" (use the provider default). Display affects visibility and billing of what's returned, not whether the model thinks — the raw chain of thought is never exposed under any setting.

const (
	// DisplaySummarized returns a readable summary of the reasoning via
	// thinking_delta events (surfaced as EventThinkingDelta).
	DisplaySummarized ThinkingDisplay = "summarized"
	// DisplayOmitted streams thinking blocks with empty text (no
	// thinking_delta events). This is the provider default on Opus 4.7+.
	DisplayOmitted ThinkingDisplay = "omitted"
)

Thinking-display levels recognized by Anthropic's adaptive thinking. Match the wire enum exactly.

type TokenCounter

type TokenCounter interface {
	// CountTokens returns the number of input tokens the request would
	// consume if streamed. The returned error follows the same wrapping
	// discipline as Stream — HTTP errors surface as *APIError wrapping
	// a sentinel; callers can branch via errors.Is.
	CountTokens(ctx context.Context, req Request) (int, error)
}

TokenCounter is the optional capability of counting input tokens for a Request without spending an inference call. Implemented by providers whose API exposes a dedicated count-tokens endpoint (Anthropic Messages and Gemini both do; both providers document the call as free today — confirm against current provider docs if billing matters to you).

Use via type assertion against an LLM value:

if c, ok := p.(llm.TokenCounter); ok {
    n, err := c.CountTokens(ctx, req)
    if err != nil { ... }
    fmt.Printf("would consume %d input tokens\n", n)
}

The OpenAI Chat Completions and Responses providers do NOT implement this — OpenAI's tokenization is local-only (tiktoken), which pi-llm-go does not bundle. Callers needing pre-flight counts for an OpenAI-hosted model should run tiktoken themselves; for OpenAI-compatible self-hosted endpoints (vLLM, Ollama), counts are tokenizer-specific and not portably reachable.

CountTokens does NOT consume the cache or interact with the inference path; calling it does not warm a cache breakpoint. Anthropic's count_tokens endpoint accepts the same body shape as /v1/messages (system, messages, tools, thinking) but ignores cache_control markers and max_tokens. Gemini's countTokens accepts the same contents array as generateContent.

Example

ExampleTokenCounter shows pre-flight input-token counting against a provider's dedicated count endpoint. Anthropic and Gemini implement llm.TokenCounter; OpenAI does not (no server-side count endpoint — run tiktoken yourself if you need a count for an OpenAI-hosted model).

package main

import (
	"context"
	"fmt"
	"os"

	llm "github.com/amit-timalsina/pi-llm-go"
	"github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
	p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

	if counter, ok := llm.LLM(p).(llm.TokenCounter); ok {
		n, err := counter.CountTokens(context.Background(), llm.Request{
			Model: anthropic.ClaudeSonnet4_6,
			Messages: []llm.Message{
				{Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "hello"}}},
			},
		})
		if err == nil {
			fmt.Printf("would consume %d input tokens\n", n)
		}
	}
}

type Tool

type Tool struct {
	Name        string
	Description string
	InputSchema json.RawMessage

	// Strict opts this tool into grammar-constrained sampling: the
	// model's token sampler is constrained to schema-valid tokens, so
	// the emitted tool input is GUARANTEED to match InputSchema (no
	// app-side validate-and-retry round-trip).
	//
	// Per-provider behavior:
	//   - Anthropic Messages: forwarded as the top-level `strict` field
	//     on the tool definition (peer of name / description /
	//     input_schema). Compiled into a grammar; schemas are cached
	//     server-side for up to 24h. JSON Schema subset applies (see
	//     Anthropic's "JSON Schema limitations" docs).
	//   - OpenAI Chat Completions: forwarded as `function.strict` on
	//     the tool definition. Requires `additionalProperties: false`
	//     at every object level and every property in `required`.
	//   - OpenAI Responses API: same as Chat Completions.
	//   - Gemini: ignored (Gemini has no per-tool strict mode; structured
	//     output uses a different surface, response_schema).
	//
	// Caveat: strict mode adds first-call latency for grammar compilation
	// (cached thereafter). For tools called once across many runs the
	// non-strict + caller-side-validate path may be cheaper.
	Strict bool
}

Tool is the wire-level declaration of a callable function exposed to the model. pi-llm-go does not execute tools — it surfaces ToolCallBlocks on the response and accepts ToolResultBlocks in follow-up messages. Execution lives in pi-agent-go.

InputSchema is a JSON Schema document describing the tool's expected input. Both Anthropic and OpenAI accept JSON Schema draft-07; the schema is forwarded to the provider as-is, so the caller is responsible for dialect choice.

type ToolCallBlock

type ToolCallBlock struct {
	ID        string
	Name      string
	Arguments json.RawMessage
}

ToolCallBlock represents a tool invocation requested by the model. Arguments is the raw JSON object the model emitted, matching the tool's declared InputSchema. The agent layer validates and dispatches.

type ToolChoice

type ToolChoice struct {
	// Type controls the choice mode. Must be one of the ToolChoice*
	// constants; the provider rejects unknown values at request build
	// time.
	Type ToolChoiceType

	// Name is the tool name the model is forced to call. REQUIRED when
	// Type == ToolChoiceTool; ignored otherwise.
	Name string
}

ToolChoice controls whether and which tool the model must call on a turn. nil on Request.ToolChoice preserves provider defaults: `auto` when Tools is non-empty, `none` otherwise.

Per-provider behavior:

• Anthropic: forwarded as `tool_choice` object on the wire. EXTENDED THINKING INCOMPATIBILITY — `Any` and `Tool` are rejected by Anthropic when `ThinkingConfig` is also set (only `Auto` / `None` are compatible with thinking). The provider does not validate this combination client-side; a 400 surfaces at request time.
• OpenAI Chat Completions: forwarded as `tool_choice`. The keyword mapping differs from Anthropic — `Any` becomes OpenAI's `"required"`. `Tool` becomes the `{"type":"function","function":{"name":"..."}}` object shape.
• OpenAI Responses API: forwarded the same as Chat Completions (same wire shape).
• Gemini: forwarded as `tool_config.function_calling_config` with `mode` mapping: `Auto`→AUTO, `Any`→ANY, `None`→NONE; `Tool` becomes ANY with `allowed_function_names: [Name]`. The allowlist constrains the model exclusively — even if Tools declares N other functions, the model can only emit a call to `Name`. Semantically stronger than Anthropic's `tool` mode (the model may also choose not to call any tool there).

type ToolChoiceType

type ToolChoiceType string

ToolChoiceType enumerates the four tool-choice modes pi-llm-go abstracts over. Provider converters map these onto the provider's concrete wire shape (Anthropic / OpenAI / Gemini all differ).

const (
	// ToolChoiceAuto lets the model decide whether to call a tool.
	// Provider default when Tools is non-empty.
	ToolChoiceAuto ToolChoiceType = "auto"

	// ToolChoiceAny forces the model to call SOME tool (which one is
	// the model's choice). Maps to Anthropic's `any`, OpenAI's
	// `required`, Gemini's `ANY`.
	ToolChoiceAny ToolChoiceType = "any"

	// ToolChoiceTool forces the model to call a specific tool by name.
	// ToolChoice.Name is required; an empty Name is a build-time error.
	ToolChoiceTool ToolChoiceType = "tool"

	// ToolChoiceNone disables tool use even when Tools is non-empty.
	// Useful for a final-answer-only turn after a tool round-trip.
	ToolChoiceNone ToolChoiceType = "none"
)

type ToolResultBlock

type ToolResultBlock struct {
	ToolCallID string
	Content    string
	IsError    bool
}

ToolResultBlock carries the result of a tool invocation back to the model. ToolCallID matches the ID on the originating ToolCallBlock.

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int

	// CacheReadTokens is the total tokens served from a cache hit on
	// this request. Anthropic populates this; OpenAI's cache is
	// opaque (no telemetry); Gemini does not yet surface it.
	CacheReadTokens int

	// CacheWriteTokens is the total tokens written to a cache on
	// this request (all TTL tiers summed for Anthropic).
	CacheWriteTokens int

	// CacheWrite5mTokens is the count of tokens cached at the default
	// ~5-minute TTL on this request. Anthropic-specific. Zero on
	// other providers and zero when no cache write happened or all
	// cached tokens went to a longer TTL.
	CacheWrite5mTokens int

	// CacheWrite1hTokens is the count of tokens cached at the
	// extended 1-hour TTL on this request. Anthropic-specific (gated
	// on the extended-cache-ttl-2025-04-11 beta header which the
	// provider auto-attaches when CacheRetention=long). Zero on
	// other providers and zero when the 1h tier was not honored —
	// the latter is the structured signal Noumenal issue #12 needs
	// to detect silent 5min fallback on unsupported models.
	CacheWrite1hTokens int

	TotalTokens int
}

Usage records token accounting returned by a provider for a single completion request. Cache fields are zero when the provider doesn't bill or report cache reads/writes separately.

Cache-write TTL breakdown:

• CacheWriteTokens is the TOTAL of cache_creation_input_tokens (all TTL tiers summed). Populated by every provider that emits a cache creation count.
• CacheWrite5mTokens and CacheWrite1hTokens are the Anthropic-specific breakdown from the `cache_creation.ephemeral_*_input_tokens` response fields. Other providers (OpenAI, Gemini) leave these at zero because their cache surface is opaque or single-TTL.

The TTL breakdown is the structured signal callers use to detect "I requested CacheRetention=long but the model silently fell back to 5min" (closes issue #12). When CacheRetention=long is requested AND CacheWrite5mTokens > 0 AND CacheWrite1hTokens == 0 on the response, the 1h hold did NOT take effect — caller-side cost budgeting that assumed a 1h-cached prefix needs to adjust.

Backward compat: CacheWriteTokens semantics unchanged — still the total, regardless of TTL. CacheWrite5mTokens + CacheWrite1hTokens equals CacheWriteTokens for Anthropic responses today; the sum may be LESS than CacheWriteTokens if Anthropic ships a new TTL tier we haven't surfaced yet. CacheWriteTokens is populated from the wire's pre-aggregated cache_creation_input_tokens (NOT summed client-side), so the aggregate stays consistent with Anthropic's billing even if our enum drifts behind the API.

type VideoBlock

type VideoBlock struct {
	// Data is the raw base64-encoded video bytes for inline emission.
	// Do NOT include the "data:" URI prefix. Mutually exclusive with URI.
	Data string

	// URI is a pre-uploaded reference (Files API handle, YouTube URL,
	// or provider-specific URI). Mutually exclusive with Data.
	URI string

	// MimeType is the video's MIME type (e.g. "video/mp4"). Required
	// when Data is set; optional when only URI is set.
	MimeType string

	// StartOffset, when non-nil, clips the start of the segment. The
	// provider must support clipping; Gemini does via videoMetadata.
	StartOffset *time.Duration

	// EndOffset, when non-nil, clips the end of the segment.
	EndOffset *time.Duration

	// FPS, when non-nil, overrides the provider's default sampling rate.
	// Gemini defaults to 1 FPS (1 video frame per second analyzed);
	// pass a higher value for action-dense content or lower for long
	// static footage. Float for fractional rates (0.5 = 1 frame per 2s).
	FPS *float64
}

VideoBlock holds video data for multimodal input. Today only the built-in Gemini provider accepts video natively; Anthropic and OpenAI providers reject VideoBlock at the wire boundary (callers wanting video understanding on those providers must extract frames client-side and submit them as ImageBlocks).

VideoBlock has three mutually-exclusive emission shapes:

• **Data + MimeType set**: inline base64. Total request body must stay under the provider's inline cap (Gemini: ~20 MB).
• **URI set**: a pre-uploaded reference. For Gemini, this is either an `https://generativelanguage.googleapis.com/v1beta/files/...` handle from the Files API (see providers/gemini/files) or a YouTube URL (public videos only; free-tier 8h/day cap).
• Exactly one of (Data, URI) must be non-empty; both empty or both set is a contract violation rejected by Validate().

Optional StartOffset, EndOffset, and FPS let callers clip the segment and override Gemini's default 1 FPS sampling. nil = use the provider's default. FPS is float for fractional rates (e.g. 0.5).

MimeType uses standard video MIME identifiers: video/mp4, video/quicktime, video/webm, video/mpeg, etc. Required when Data is set; ignored when only URI is set (server infers from the file).

func (VideoBlock) Validate

func (v VideoBlock) Validate() error

Validate enforces the VideoBlock contract:

• exactly one of (Data, URI) must be non-empty
• Data must not carry a "data:" URI prefix (raw base64 only)
• MimeType is required when Data is set