llm

README

pi-llm-go

A minimal, Go-native LLM adapter with streaming, tool calling, and extended thinking. Provider-agnostic interface with built-in support for Anthropic Messages, OpenAI Chat Completions (covering OpenAI, Azure OpenAI, Groq, Together, vLLM, OpenRouter, Ollama), and the OpenAI Responses API (GPT-5-family server-side state + reasoning summaries).

Status: v0.x — pre-1.0. API may change between minor versions; see CHANGELOG.md.

Why

The Go LLM library landscape forces you to pick between heavy vendor SDKs that don't compose, code-generated client surfaces that don't track new providers, or "framework" libraries that ship more concepts than you want. pi-llm-go is the opposite: ~1.5kLoc of plain Go that gives you an interface, two providers, and a streaming model that uses iter.Seq2 like a normal iterator. No HTTP wrappers, no model registries, no provider-specific magic.

Installation

go get github.com/amit-timalsina/pi-llm-go

Requires Go 1.23 or later (for iter.Seq2).

Quickstart

package main

import (
    "context"
    "fmt"
    "os"

    llm "github.com/amit-timalsina/pi-llm-go"
    "github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
    p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

    msg, err := llm.Complete(context.Background(), p, llm.Request{
        Model:     anthropic.ClaudeSonnet4_6,
        MaxTokens: 1024,
        Messages: []llm.Message{
            {Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "hello"}}},
        },
    })
    if err != nil { panic(err) }

    for _, block := range msg.Content {
        if tb, ok := block.(llm.TextBlock); ok {
            fmt.Println(tb.Text)
        }
    }
}

Streaming is iter.Seq2[llm.StreamEvent, error] — range over it:

for event, err := range p.Stream(ctx, req) {
    if err != nil { /* handle */ }
    if d, ok := event.(llm.EventTextDelta); ok {
        fmt.Print(d.Delta)
    }
}

Features

• Streaming-first. Stream() returns iter.Seq2[StreamEvent, error] — Go 1.23 iterators, no callbacks, no goroutine leaks. Complete() is the synchronous helper for one-shot use.
• Sealed sum types. Block and StreamEvent are interfaces with package-private marker methods. Type-switch exhaustively; the compiler tells you if you miss a case.
• Tool calling. Declare tools on Request.Tools; receive ToolCallBlocks on the response; send ToolResultBlocks back. Pi-llm-go does not execute tools — that's pi-agent-go's job.
• Extended thinking. ThinkingConfig{BudgetTokens: N} on requests, surfaced as ThinkingBlock content. Anthropic-only at v1.
• Open-closed providers. Implement LLM.Stream to add custom providers; no plugin registry needed.
• Errors that branch cleanly. errors.Is(err, llm.ErrRateLimit) works through *APIError wraps; errors.As(err, &apiErr) gives you status + body.
• Cancellation = context.Context. No bespoke abort signal types.

Providers

Anthropic

import "github.com/amit-timalsina/pi-llm-go/providers/anthropic"

p, _ := anthropic.New(anthropic.Options{
    APIKey: os.Getenv("ANTHROPIC_API_KEY"),
    // BaseURL: defaults to https://api.anthropic.com
    // Version: defaults to "2023-06-01"
    // Beta:    optional anthropic-beta header values
})

Honors Request.Thinking. Surfaces all content-block types (text, thinking, tool_use).

OpenAI-compatible

import "github.com/amit-timalsina/pi-llm-go/providers/openai"

p, _ := openai.New(openai.Options{
    APIKey:  os.Getenv("OPENAI_API_KEY"),
    BaseURL: "https://api.openai.com/v1",  // or "https://api.groq.com/openai/v1", etc.
})

Talks the Chat Completions wire format, so the same provider works against any compatible host. Request.Thinking is ignored at v1 — reasoning-effort dialects vary too much across compatible hosts to map portably.

Gemini

import "github.com/amit-timalsina/pi-llm-go/providers/gemini"

p, _ := gemini.New(gemini.Options{APIKey: os.Getenv("GEMINI_API_KEY")})

Native support for the Gemini 2.5 / 3 / Robotics ER 1.6 families. Same LLM interface as the other providers, plus llm.VideoBlock for native video input (Gemini is the only provider that accepts video natively; Anthropic and OpenAI reject VideoBlock at the wire boundary with a clear pointer to the frame-extraction workaround). YouTube URLs work directly:

llm.VideoBlock{URI: "https://www.youtube.com/watch?v=..."}

For files larger than ~20 MB, the sibling providers/gemini/files sub-package handles the multipart upload + ACTIVE-state polling:

import "github.com/amit-timalsina/pi-llm-go/providers/gemini/files"

fc, _ := files.New(files.Options{APIKey: os.Getenv("GEMINI_API_KEY")})
ref, _ := fc.Upload(ctx, mp4Reader, "video/mp4", files.UploadOptions{DisplayName: "demo.mp4"})
ref, _ = fc.Wait(ctx, ref, files.WaitOptions{}) // polls until ACTIVE
defer fc.Delete(context.Background(), ref.Name) // ~48h server-side TTL if you forget

// Now use the URI in a generateContent call:
content := []llm.Block{llm.TextBlock{Text: "describe"}, llm.VideoBlock{URI: ref.URI}}

Vertex AI (gs:// URIs + OAuth) is a planned future addition; v0.5 only supports the Google AI direct endpoint.

Error handling

Non-2xx HTTP responses surface as *llm.APIError wrapping one of the typed sentinels — errors.Is works through the wrapping so consumers branch on category, not status code:

ErrAuth           // 401, 403
ErrRateLimit      // 429
ErrInvalidRequest // other 4xx
ErrProvider       // generic provider problem (parent of the next two)
├─ ErrServerError // 5xx (excluding 529)
└─ ErrOverloaded  // 529 (Anthropic infra overload)

ErrServerError and ErrOverloaded both wrap ErrProvider via %w, so legacy errors.Is(err, llm.ErrProvider) keeps matching 5xx + 529 unchanged.

Sugar helpers and the parsed Retry-After:

for ev, err := range provider.Stream(ctx, req) {
    if err == nil { /* consume ev */ continue }

    var apiErr *llm.APIError
    if errors.As(err, &apiErr) {
        switch {
        case llm.IsRateLimited(err):     // 429 → respect apiErr.RetryAfter
        case llm.IsOverloaded(err):      // 529 → short backoff, consider failover
        case llm.IsServerError(err):     // 5xx → retry, escalate if sustained
        case errors.Is(err, llm.ErrAuth):
        }
    }
    return err
}

APIError.RetryAfter is populated by all four built-in providers when the response carries a Retry-After (RFC 7231 delta-seconds or HTTP-date) or retry-after-ms (OpenAI's millisecond form, which wins when both are present).

Cost telemetry

Every completion surfaces a typed Usage value on the final EventMessageEnd (and on the *llm.Message returned by Complete / Accumulate):

msg, _ := llm.Complete(ctx, provider, req)
fmt.Printf("in=%d out=%d cache_write=%d (5m=%d, 1h=%d) cache_read=%d total=%d\n",
    msg.Usage.InputTokens, msg.Usage.OutputTokens,
    msg.Usage.CacheWriteTokens, msg.Usage.CacheWrite5mTokens, msg.Usage.CacheWrite1hTokens,
    msg.Usage.CacheReadTokens, msg.Usage.TotalTokens)

CacheWrite5mTokens and CacheWrite1hTokens are Anthropic-specific — they break CacheWriteTokens down by TTL tier so consumers can detect silent 5min fallback when CacheRetention=long was requested but the model didn't honor the extended TTL:

if req.CacheRetention == llm.CacheRetentionLong &&
    msg.Usage.CacheWrite5mTokens > 0 && msg.Usage.CacheWrite1hTokens == 0 {
    // Model downgraded the cache hold to 5min. Cost projections that
    // assumed a 1h-cached prefix across iterations need to adjust.
}

OpenAI and Gemini leave the TTL-breakdown fields at zero (their cache surfaces are opaque or single-TTL).

llm.ComputeCost applies a built-in pricing table to a Usage value and returns a dollar breakdown:

cost, err := llm.ComputeCost(msg.Usage, req.Model)
if err == nil {
    fmt.Printf("$%.4f total (in=$%.4f out=$%.4f cache_read=$%.4f cache_write_1h=$%.4f)\n",
        cost.Total(), cost.Input, cost.Output, cost.CacheRead, cost.CacheWrite1h)
}

The seed table covers the Claude 4, GPT-5, and Gemini 2.5/3.1 families (verified 2026-05-13). For any model not in the table — older snapshots, deprecated IDs, regional/batch tiers — call llm.RegisterPricing(modelID, llm.Pricing{...}) at startup with rates pulled from your own source. Registered entries override the seed.

The TTL-breakdown fields from Usage carry through automatically: when CacheWrite5mTokens and CacheWrite1hTokens are populated, they price against their respective tiers independently — so silent 5min fallback is reflected in the cost projection without any caller-side branching.

Token counting

Anthropic and Gemini expose a count-tokens endpoint that returns the input-token count without spending an inference call. Providers implement llm.TokenCounter when they support this:

if c, ok := provider.(llm.TokenCounter); ok {
    n, err := c.CountTokens(ctx, req)
    if err == nil {
        fmt.Printf("request would consume %d input tokens\n", n)
    }
}

OpenAI's Chat Completions and Responses providers do NOT implement TokenCounter — OpenAI's tokenization is local-only via tiktoken, which pi-llm-go does not bundle. Callers needing a pre-flight count for an OpenAI-hosted model should run tiktoken themselves.

Examples

Runnable examples in examples/:

• examples/streaming — basic streaming of a text response.
• examples/tool_calling — hand-rolled tool-call loop against get_current_time.
• examples/multimodal — image input on Anthropic + OpenAI.
• examples/multimodal_gemini — text / image / video against Gemini; --video-upload PATH exercises the Files API end-to-end.
• examples/prompt_caching — CacheRetention knob driving Anthropic prompt-cache hits.

Run them with go run ./examples/streaming (set ANTHROPIC_API_KEY first).

Versioning

This package is pre-1.0. Anything can change between minor versions; refer to CHANGELOG.md for each release.

v1.0 lands when the API surface has been used in production for ≥4 weeks without churn. Post-1.0 follows semver strictly.

License

MIT. See LICENSE.

Acknowledgements

Designed and named after pi-ai by Mario Zechner (TypeScript, MIT). The wire-level vocabulary and event types follow the upstream's lead; the Go-native API surface (interface-based providers, iterator streaming, sealed sum types) is a from-scratch redesign for Go idioms.

Built and maintained by Amit Timalsina with Claude Code assistance — all design decisions and release calls are human-owned.

Documentation

Overview

Package llm is a minimal, provider-agnostic Go interface for streaming LLM completions with tool calling. Built-in providers cover the Anthropic Messages API and OpenAI-compatible Chat Completions APIs (OpenAI, Groq, Together, vLLM, OpenRouter, Ollama, and similar).

Two patterns of use:

• Streaming events: range over LLM.Stream to react to deltas in real time.
• One-shot completion: call Complete to drain the stream and receive the final assistant message.

Cancellation propagates through context.Context. Provider errors surface through the iterator's error return as *APIError wrapping a sentinel (ErrAuth, ErrRateLimit, ErrInvalidRequest, ErrProvider).

pi-llm-go does not execute tools. It declares tool schemas on requests and surfaces ToolCallBlocks on responses. The companion pi-agent-go module adds the execution loop.

Index

• Variables
• func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]
• func IsOverloaded(err error) bool
• func IsRateLimited(err error) bool
• func IsServerError(err error) bool
• func ParseRetryAfter(headers http.Header) time.Duration
• func RegisterPricing(model string, p Pricing)
• func SentinelForStatus(status int) error
• type APIError
  • func (e *APIError) Error() string
  • func (e *APIError) Unwrap() error
• type Block
• type CacheRetention
• type Cost
  • func ApplyPricing(usage Usage, p Pricing) Cost
  • func ComputeCost(usage Usage, model string) (Cost, error)
  • func (c Cost) Total() float64
• type EventMessageEnd
• type EventMessageStart
• type EventTextDelta
• type EventTextEnd
• type EventTextStart
• type EventThinkingDelta
• type EventThinkingEnd
• type EventThinkingStart
• type EventToolCallDelta
• type EventToolCallEnd
• type EventToolCallStart
• type ImageBlock
  • func (i ImageBlock) Validate() error
• type LLM
• type Message
  • func Complete(ctx context.Context, l LLM, req Request) (*Message, error)
• type Pricing
  • func PricingFor(model string) (Pricing, bool)
• type Request
• type Role
• type StopReason
• type StreamEvent
• type TextBlock
• type ThinkingBlock
• type ThinkingConfig
• type TokenCounter
• type Tool
• type ToolCallBlock
• type ToolResultBlock
• type Usage
• type VideoBlock
  • func (v VideoBlock) Validate() error

Variables

var (
	ErrAuth           = errors.New("llm: authentication failed")
	ErrRateLimit      = errors.New("llm: rate limited")
	ErrInvalidRequest = errors.New("llm: invalid request")
	ErrProvider       = errors.New("llm: provider error")
	// ErrServerError signals a generic 5xx response (excluding 529).
	// Recommended consumer policy: retry with backoff; if sustained
	// past a threshold, surface for engineer escalation.
	ErrServerError = fmt.Errorf("%w: server error (5xx)", ErrProvider)
	// ErrOverloaded signals an Anthropic-style 529 "overloaded"
	// response. Recommended consumer policy: short backoff (~60s)
	// then retry; consider provider fallback if sustained.
	ErrOverloaded = fmt.Errorf("%w: overloaded (529)", ErrProvider)
)

Sentinel errors. Wrap via APIError or return directly. Use errors.Is to branch on them in caller retry / fallback logic.

Hierarchy:

ErrProvider           // generic "something provider-side broke"
├─ ErrServerError     // HTTP 5xx (excluding 529)
└─ ErrOverloaded      // HTTP 529 (Anthropic infra overload)

ErrServerError and ErrOverloaded each wrap ErrProvider via fmt.Errorf("%w"), so existing callers using errors.Is(err, ErrProvider) continue to match 5xx + 529 responses (backward compatible). The two child sentinels add specificity for consumers that need distinct retry / escalation policies per the category guidance in issue #11.

var ErrUnknownModel = errors.New("llm: unknown model pricing")

ErrUnknownModel is returned by ComputeCost when the model is not in the built-in pricing table and has no caller-registered pricing. Callers can register pricing via RegisterPricing.

Functions

func Accumulate

func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]

Accumulate folds a stream of events into a stream of progressively-built messages. Each yield emits a snapshot of the assistant message as it stands after the most recent event. The final yielded value (when err is nil) is the fully-assembled assistant message.

Callers that only want the final message should prefer Complete. Use Accumulate when intermediate snapshots are useful (e.g. driving a UI that renders each delta against a complete message tree rather than against individual deltas).

Each snapshot is an independent value — internal slices and strings are copied on emit so callers can retain previous snapshots without aliasing.

func IsOverloaded

func IsOverloaded(err error) bool

IsOverloaded reports whether err is an Anthropic-style 529 overloaded response.

func IsRateLimited

func IsRateLimited(err error) bool

IsRateLimited reports whether err (or anything in its Unwrap chain) is a 429 rate-limit error. Equivalent to errors.Is(err, ErrRateLimit) — sugar for the common caller-side branch.

func IsServerError

func IsServerError(err error) bool

IsServerError reports whether err is a generic 5xx response (excluding 529; check IsOverloaded for that case).

func ParseRetryAfter

func ParseRetryAfter(headers http.Header) time.Duration

ParseRetryAfter extracts a wait hint from a provider response's Retry-After / retry-after-ms headers. Returns 0 if no header is present or none of them parse.

Header precedence (`retry-after-ms` wins when both are present because it carries sub-second precision):

• retry-after-ms: integer milliseconds (OpenAI convention).
• Retry-After: integer seconds (RFC 7231 delta-seconds).
• Retry-After: HTTP-date (RFC 7231; computed as now() delta).

Negative deltas (HTTP-date already in the past) are clamped to 0. Callers should still apply their own minimum bound — a 0-second server hint usually means "as soon as you can" but pummeling the API immediately is rarely productive.

func RegisterPricing

func RegisterPricing(model string, p Pricing)

RegisterPricing registers (or overrides) pricing for a model ID. Safe for concurrent use. Registered entries take precedence over the built-in seed table.

Production callers maintaining their own pricing source should call RegisterPricing once at startup for every model they bill against, rather than relying on the built-in table.

func SentinelForStatus

func SentinelForStatus(status int) error

SentinelForStatus maps an HTTP status code to the matching sentinel error. Used by provider implementations when constructing APIError.

Status to sentinel:

401, 403       → ErrAuth
429            → ErrRateLimit
other 4xx      → ErrInvalidRequest
529            → ErrOverloaded (Anthropic-style)
other 5xx      → ErrServerError
otherwise      → ErrProvider

ErrServerError and ErrOverloaded both wrap ErrProvider, so legacy errors.Is(err, ErrProvider) keeps working for 5xx / 529 responses.

Types

type APIError

type APIError struct {
	Provider string
	Status   int
	Body     []byte
	Inner    error
	// RetryAfter, when > 0, is the server's hint for how long the
	// caller should wait before retrying. Parsed from the response's
	// Retry-After header (RFC 7231 seconds-or-HTTP-date) or, for
	// OpenAI-family providers, the `retry-after-ms` header. Zero
	// means "no header present" — callers fall back to their own
	// backoff schedule.
	RetryAfter time.Duration
}

APIError wraps a non-2xx HTTP response from a provider. The Inner field is one of the sentinel errors above so that errors.Is works through the wrapping. Status and Body let callers inspect the raw failure (e.g. to parse a structured provider error payload). RetryAfter, when non-zero, is the parsed value of the response's Retry-After or retry-after-ms header — populated by providers for 429 / 529 responses to support caller-side rate-limit / overload backoff.

func (*APIError) Error

func (e *APIError) Error() string

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

type Block

type Block interface {
	// contains filtered or unexported methods
}

Block is the sealed sum type for message content. Concrete implementations are TextBlock, ThinkingBlock, ToolCallBlock, and ToolResultBlock — all defined in this package. The unexported marker method keeps the set closed: provider converters need exhaustive type-switches to serialize content correctly, so new block types must be added inside the package.

type CacheRetention

type CacheRetention string

CacheRetention controls Anthropic prompt-cache breakpoint placement. It is a single-knob abstraction: callers pick a retention tier and the provider decides where to place the cache_control markers on the underlying wire format.

The marker tells Anthropic "everything from the start of the request up to and including this block is cacheable; on a subsequent request with byte-identical content up to this marker, return a cache hit and bill cache-read rates instead of full input rates."

Behavior by value:

• CacheRetentionNone (zero value, "") — no markers emitted.
• CacheRetentionShort — ephemeral markers with the default ~5 minute lifetime, placed at: (a) the System prompt's trailing block, (b) the final Tool in Request.Tools, (c) the last block (any type) of the most recent user-role message. The last-block placement is type-agnostic so that subsequent calls in a tool loop reuse the cached tool_result round-trip instead of re-billing it.
• CacheRetentionLong — same placement as Short with TTL "1h" and the "extended-cache-ttl-2025-04-11" beta header auto-attached to the outgoing HTTP request.

OpenAI's Chat Completions and Responses providers silently ignore CacheRetention — OpenAI caches automatically with no caller-side breakpoint API.

1h-TTL availability and fallback behavior:

All currently-shipped Claude 4 family models (Opus 4.7, Sonnet 4.6, Haiku 4.5) support 1h cache TTL. Older Claude 3.x models accept the beta header but may silently downgrade the hold to the 5-minute default — Anthropic does NOT error on an unsupported model. The silent fallback is a real cost-budgeting hazard for callers that assumed a 1h-cached prefix would survive across long iterations (Noumenal issue #12).

Detect silent fallback via the Usage breakdown: when CacheRetention=long was requested, inspect the response message's Usage.CacheWrite5mTokens vs CacheWrite1hTokens. If the 5m count is non-zero and the 1h count is zero, the model honored the request at 5min, not 1h — adjust cost projections accordingly. The two fields are populated from Anthropic's cache_creation response breakdown; other providers leave them at 0.

The heuristic (5m>0 && 1h==0) assumes UNIFORM TTL placement across all cache_control markers in the request, which CacheRetention=long guarantees today (every marker carries ttl:"1h"). If a future per-block-TTL feature lets callers mix 5min and 1h breakpoints in one request, this diagnostic would produce false positives and would need a different signal — likely a per-block annotation rather than aggregate counts.

Note that as of March 2026, Anthropic regressed the DEFAULT ephemeral TTL from 60min to 5min; the 1h tier is now opt-in via CacheRetentionLong + the extended-cache-ttl-2025-04-11 beta header (which the Anthropic provider auto-attaches when CacheRetention=long).

The caller owns prompt determinism: cached sections must be byte-stable across iterations for the cache to hit. Any change (timestamps, map iteration order, reordered items) invalidates the cache from that point forward in the request.

See https://docs.claude.com/en/docs/build-with-claude/prompt-caching for the full discipline.

const (
	// CacheRetentionNone disables prompt caching for this request. This is
	// the zero value of CacheRetention; an unset field and an explicit
	// CacheRetentionNone are byte-identical and produce no cache_control
	// markers.
	CacheRetentionNone CacheRetention = ""

	// CacheRetentionShort places ephemeral cache breakpoints with the
	// default ~5 minute lifetime. The right default for iterative agent
	// loops where the prefix is reused within a single session.
	CacheRetentionShort CacheRetention = "short"

	// CacheRetentionLong places ephemeral cache breakpoints with the 1-hour
	// TTL and auto-attaches the "extended-cache-ttl-2025-04-11" beta header.
	// For long-lived static prefixes (large system prompts, big tool sets)
	// that survive across many sessions.
	CacheRetentionLong CacheRetention = "long"
)

type Cost

type Cost struct {
	Input        float64
	Output       float64
	CacheRead    float64
	CacheWrite5m float64
	CacheWrite1h float64
}

Cost is the dollar breakdown of a single completion's Usage. Sum via Total() for a single number.

Categories track Usage fields directly so the diagnostic from Usage.CacheWrite5mTokens / CacheWrite1hTokens (silent 5min fallback detection) carries into the cost projection.

func ApplyPricing

func ApplyPricing(usage Usage, p Pricing) Cost

ApplyPricing is the pure-arithmetic form of ComputeCost: takes a caller-supplied Pricing rather than a model lookup. Useful when the caller maintains their own pricing source (e.g. a database, models.dev poll) outside the built-in table.

func ComputeCost

func ComputeCost(usage Usage, model string) (Cost, error)

ComputeCost applies the registered pricing for model to usage and returns the dollar breakdown. Returns ErrUnknownModel wrapped with the model ID when no pricing is registered.

The Anthropic-specific TTL breakdown on Usage.CacheWrite5mTokens / CacheWrite1hTokens is honored: each tier prices against its own rate so silent 5min fallback (Issue #12 diagnostic) is reflected in the cost projection automatically.

When Usage.CacheWriteTokens > 0 but both CacheWrite5mTokens and CacheWrite1hTokens are 0 (OpenAI / Gemini, or older Anthropic SDK versions that didn't surface the breakdown), the writeable tokens fall through to the 5m rate as a best-effort. On non-Anthropic providers this rate is 0, so the cost is 0 regardless — matching the wire reality (OpenAI's cache is automatic; Gemini doesn't separately meter the write).

func (Cost) Total

func (c Cost) Total() float64

Total returns the sum of all cost categories.

type EventMessageEnd

type EventMessageEnd struct {
	StopReason StopReason
	Usage      Usage
}

EventMessageEnd is the terminal event. It carries the normalized stop reason and the final usage tally.

type EventMessageStart

type EventMessageStart struct {
	Model string
}

EventMessageStart is emitted once at the start of an assistant turn, before any block events.

type EventTextDelta

type EventTextDelta struct {
	BlockIndex int
	Delta      string
}

EventTextDelta appends Delta to the text in the block at BlockIndex.

type EventTextEnd

type EventTextEnd struct {
	BlockIndex int
}

EventTextEnd marks the end of the TextBlock at BlockIndex.

type EventTextStart

type EventTextStart struct {
	BlockIndex int
}

EventTextStart marks the beginning of a TextBlock.

type EventThinkingDelta

type EventThinkingDelta struct {
	BlockIndex int
	Delta      string
}

EventThinkingDelta appends Delta to the thinking block at BlockIndex.

type EventThinkingEnd

type EventThinkingEnd struct {
	BlockIndex int
	Signature  string
}

EventThinkingEnd marks the end of the ThinkingBlock. Signature is the opaque provider-supplied token to round-trip on follow-up messages.

type EventThinkingStart

type EventThinkingStart struct {
	BlockIndex int
}

EventThinkingStart marks the beginning of a ThinkingBlock.

type EventToolCallDelta

type EventToolCallDelta struct {
	BlockIndex int
	Delta      string
}

EventToolCallDelta delivers a fragment of the streaming JSON arguments. Callers that need the assembled arguments should wait for EventToolCallEnd rather than accumulate Delta bytes themselves (provider JSON delta framing is not guaranteed to be at value boundaries).

type EventToolCallEnd

type EventToolCallEnd struct {
	BlockIndex int
	Arguments  json.RawMessage
}

EventToolCallEnd marks the end of a ToolCallBlock and carries the assembled arguments.

type EventToolCallStart

type EventToolCallStart struct {
	BlockIndex int
	ID         string
	Name       string
}

EventToolCallStart marks the beginning of a ToolCallBlock. ID and Name are available immediately; arguments stream as deltas and are emitted in fully assembled form on EventToolCallEnd.

type ImageBlock

type ImageBlock struct {
	// Data is the raw base64-encoded image bytes. Do NOT include the
	// "data:<mime>;base64," prefix — providers add it where required.
	Data string

	// MimeType is the image's MIME type (e.g. "image/png"). Required.
	MimeType string
}

ImageBlock holds image data for multimodal input. Data is the raw base64-encoded image bytes (no "data:" URI prefix); MimeType is the standard MIME identifier (e.g. "image/png"). Providers convert to their on-wire format at the boundary.

pi-llm-go does NOT fetch image URLs. Callers that want to attach a remote image must download it themselves first, then construct an ImageBlock with the resulting bytes encoded. This keeps the library network-free except for the LLM provider call itself — no surprise timeouts, no surprise 404s mid-stream.

Portable MIME types accepted by every built-in provider:

• "image/jpeg"
• "image/png"
• "image/gif"
• "image/webp"

Other types may work on specific providers (e.g. OpenAI accepts more) but pi-llm-go does not pre-validate — the provider returns an ErrInvalidRequest if the type is unsupported.

v0.3.0 supports ImageBlock as USER-message input only. Assistant image output is provider-specific and a separate, future feature.

func (ImageBlock) Validate

func (i ImageBlock) Validate() error

Validate enforces the ImageBlock contract: Data must be raw base64-encoded bytes (without the "data:<mime>;base64," URI prefix) and MimeType must be set. Providers call this at the wire boundary and surface a wrapped error if the contract is violated.

type LLM

type LLM interface {
	Stream(ctx context.Context, req Request) iter.Seq2[StreamEvent, error]
}

LLM is the provider-agnostic streaming interface. Anthropic and OpenAI providers implement this; third-party providers may do the same to plug into Complete and Accumulate.

Implementations must:

• Honor cancellation of ctx by terminating the underlying HTTP request and yielding (nil, ctx.Err()) from the iterator.
• Surface HTTP errors as *APIError values via the iterator's error half.
• Emit events in the order documented on StreamEvent.

type Message

type Message struct {
	Role    Role
	Content []Block

	Usage      Usage
	StopReason StopReason
	Model      string
}

Message is one turn in the transcript. Content holds a sequence of blocks — the model emits assistant messages with mixed text / thinking / tool-call content; the caller sends user messages with text and tool messages with tool-result content.

Usage, StopReason, and Model are populated on assistant messages produced by Complete or Accumulate; they are zero on user / tool messages and on messages sent into Stream.

func Complete

func Complete(ctx context.Context, l LLM, req Request) (*Message, error)

Complete drains a streaming completion and returns the final assistant message. It is equivalent to iterating Stream and folding each event into a Message via Accumulate.

Returns the partial message and a wrapped error if the stream terminates early; the partial may be useful for debugging or replay.

type Pricing

type Pricing struct {
	// Input is the per-million-token cost for non-cached input tokens.
	Input float64

	// Output is the per-million-token cost for output tokens. Includes
	// thinking tokens on Anthropic (extended thinking is billed as
	// output).
	Output float64

	// CacheRead is the per-million-token cost for tokens served from
	// a cache hit. On Anthropic this is 0.1× Input by policy; on
	// OpenAI it's roughly 0.1× Input (the "cached input" rate); on
	// Gemini it's the published "context caching" rate.
	CacheRead float64

	// CacheWrite5m is the per-million-token cost for tokens cached at
	// the default ~5 minute TTL. Anthropic-specific (1.25× Input today).
	// Other providers leave this at 0.
	CacheWrite5m float64

	// CacheWrite1h is the per-million-token cost for tokens cached at
	// the extended 1-hour TTL. Anthropic-specific (2× Input today,
	// gated on the extended-cache-ttl-2025-04-11 beta header which the
	// provider auto-attaches when CacheRetention=long). Other providers
	// leave this at 0.
	CacheWrite1h float64
}

Pricing holds the per-token cost rates for a model. All rates are in dollars per million tokens (the industry-standard quoting unit).

Rate semantics by provider:

• Anthropic publishes input + output rates plus three cache rates (5m write, 1h write, read). The cache rates are deterministic multipliers on the base input rate today (1.25× / 2× / 0.1×) but the multiplier policy could change, so the seed table stores them explicitly.
• OpenAI publishes input + output + a single "cached input" rate; CacheRead is the appropriate field. OpenAI has no caller-visible cache-write category — cache reads are billed at the discounted rate; writes implicit in standard input.
• Gemini publishes input + output + a single context-caching rate. CacheRead is the appropriate field; CacheWrite5m and CacheWrite1h stay 0 (Gemini's cache is single-TTL and not separately metered for the write).

Pricing is forward-flexible: a provider that ships a new cache tier can populate the unused field without breaking existing callers.

func PricingFor

func PricingFor(model string) (Pricing, bool)

PricingFor returns the registered pricing for a model ID and true, or zero Pricing and false if the model is unknown.

Lookup order:

1. Pricing registered via RegisterPricing (caller-overrideable).
2. The built-in seed table (a small set of canonical models, last verified against the provider docs on the build date — see pricing_seed.go for the verification date).

Pricing changes. Production callers SHOULD verify rates against the provider's current pricing page and call RegisterPricing for any model they care about, rather than trusting the built-in table across upgrades.

type Request

type Request struct {
	Model       string
	System      string
	Messages    []Message
	Tools       []Tool
	Temperature *float64
	MaxTokens   int
	Thinking    *ThinkingConfig
	StopReasons []string

	// CacheRetention controls Anthropic prompt-cache breakpoint placement.
	// When unset or "none", no cache markers are emitted. When "short" or
	// "long", the Anthropic provider auto-places ephemeral cache_control
	// markers at the static prefix boundary: the last block of the System
	// prompt, the final Tool in Tools, and the last text block of the most
	// recent user message. "long" additionally selects the 1h TTL and
	// auto-attaches the extended-cache-ttl-2025-04-11 beta header.
	//
	// Ignored by OpenAI providers (their cache is automatic and opaque).
	CacheRetention CacheRetention
}

Request is the common payload for a completion. Provider-specific tunables that have no portable meaning live on the provider's own Options struct (passed to its constructor), not here.

Temperature is a pointer so the zero value can be distinguished from "unset"; callers that want temperature=0 must set *Temperature to 0.

type Role

type Role string

Role enumerates message roles in a transcript. RoleTool messages carry tool results back to the model and may hold only ToolResultBlock content.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type StopReason

type StopReason string

StopReason is the normalized reason a model stopped generating. Provider-specific stop reasons map to one of these; unmappable values surface as errors rather than leaking provider strings.

const (
	StopReasonEnd       StopReason = "end"        // natural end of turn
	StopReasonMaxTokens StopReason = "max_tokens" // hit MaxTokens cap
	StopReasonToolUse   StopReason = "tool_use"   // model requested tool calls
	StopReasonStop      StopReason = "stop"       // matched a stop sequence
)

type StreamEvent

type StreamEvent interface {
	// contains filtered or unexported methods
}

StreamEvent is the sealed sum type emitted during a streaming completion. Errors flow through the iterator's error half rather than as event values, so consumers do not need an EventError variant.

Event order for one assistant turn:

EventMessageStart
  ( EventTextStart, EventTextDelta*, EventTextEnd
  | EventThinkingStart, EventThinkingDelta*, EventThinkingEnd
  | EventToolCallStart, EventToolCallDelta*, EventToolCallEnd
  )*
EventMessageEnd

BlockIndex on per-block events is the position of the block inside the emitted message's Content slice — it lets consumers route deltas when rendering or reconstructing incrementally.

type TextBlock

type TextBlock struct {
	Text string
}

TextBlock holds plain text content.

type ThinkingBlock

type ThinkingBlock struct {
	Thinking  string
	Signature string
}

ThinkingBlock holds an extended-thinking segment emitted by reasoning models. Signature is an opaque provider-supplied token that must be preserved and replayed for multi-turn thinking continuity (Anthropic).

type ThinkingConfig

type ThinkingConfig struct {
	// BudgetTokens is the maximum number of thinking tokens the model may
	// emit before producing the final response. Required when ThinkingConfig
	// is non-nil. Provider minimums apply (Anthropic: 1024).
	//
	// IMPORTANT: Anthropic requires Request.MaxTokens > BudgetTokens because
	// thinking tokens are counted against max_tokens. A common safe choice
	// is MaxTokens == BudgetTokens * 2, giving roughly equal budget to the
	// reasoning trace and the visible answer.
	BudgetTokens int
}

ThinkingConfig enables extended thinking on supported models. Honored by the Anthropic provider. Ignored by the OpenAI-compatible provider in v1.

type TokenCounter

type TokenCounter interface {
	// CountTokens returns the number of input tokens the request would
	// consume if streamed. The returned error follows the same wrapping
	// discipline as Stream — HTTP errors surface as *APIError wrapping
	// a sentinel; callers can branch via errors.Is.
	CountTokens(ctx context.Context, req Request) (int, error)
}

TokenCounter is the optional capability of counting input tokens for a Request without spending an inference call. Implemented by providers whose API exposes a dedicated count-tokens endpoint (Anthropic Messages and Gemini both do; both providers document the call as free today — confirm against current provider docs if billing matters to you).

Use via type assertion against an LLM value:

if c, ok := p.(llm.TokenCounter); ok {
    n, err := c.CountTokens(ctx, req)
    if err != nil { ... }
    fmt.Printf("would consume %d input tokens\n", n)
}

The OpenAI Chat Completions and Responses providers do NOT implement this — OpenAI's tokenization is local-only (tiktoken), which pi-llm-go does not bundle. Callers needing pre-flight counts for an OpenAI-hosted model should run tiktoken themselves; for OpenAI-compatible self-hosted endpoints (vLLM, Ollama), counts are tokenizer-specific and not portably reachable.

CountTokens does NOT consume the cache or interact with the inference path; calling it does not warm a cache breakpoint. Anthropic's count_tokens endpoint accepts the same body shape as /v1/messages (system, messages, tools, thinking) but ignores cache_control markers and max_tokens. Gemini's countTokens accepts the same contents array as generateContent.

type Tool

type Tool struct {
	Name        string
	Description string
	InputSchema json.RawMessage
}

Tool is the wire-level declaration of a callable function exposed to the model. pi-llm-go does not execute tools — it surfaces ToolCallBlocks on the response and accepts ToolResultBlocks in follow-up messages. Execution lives in pi-agent-go.

InputSchema is a JSON Schema document describing the tool's expected input. Both Anthropic and OpenAI accept JSON Schema draft-07; the schema is forwarded to the provider as-is, so the caller is responsible for dialect choice.

type ToolCallBlock

type ToolCallBlock struct {
	ID        string
	Name      string
	Arguments json.RawMessage
}

ToolCallBlock represents a tool invocation requested by the model. Arguments is the raw JSON object the model emitted, matching the tool's declared InputSchema. The agent layer validates and dispatches.

type ToolResultBlock

type ToolResultBlock struct {
	ToolCallID string
	Content    string
	IsError    bool
}

ToolResultBlock carries the result of a tool invocation back to the model. ToolCallID matches the ID on the originating ToolCallBlock.

type Usage

type Usage struct {
	InputTokens  int
	OutputTokens int

	// CacheReadTokens is the total tokens served from a cache hit on
	// this request. Anthropic populates this; OpenAI's cache is
	// opaque (no telemetry); Gemini does not yet surface it.
	CacheReadTokens int

	// CacheWriteTokens is the total tokens written to a cache on
	// this request (all TTL tiers summed for Anthropic).
	CacheWriteTokens int

	// CacheWrite5mTokens is the count of tokens cached at the default
	// ~5-minute TTL on this request. Anthropic-specific. Zero on
	// other providers and zero when no cache write happened or all
	// cached tokens went to a longer TTL.
	CacheWrite5mTokens int

	// CacheWrite1hTokens is the count of tokens cached at the
	// extended 1-hour TTL on this request. Anthropic-specific (gated
	// on the extended-cache-ttl-2025-04-11 beta header which the
	// provider auto-attaches when CacheRetention=long). Zero on
	// other providers and zero when the 1h tier was not honored —
	// the latter is the structured signal Noumenal issue #12 needs
	// to detect silent 5min fallback on unsupported models.
	CacheWrite1hTokens int

	TotalTokens int
}

Usage records token accounting returned by a provider for a single completion request. Cache fields are zero when the provider doesn't bill or report cache reads/writes separately.

Cache-write TTL breakdown:

• CacheWriteTokens is the TOTAL of cache_creation_input_tokens (all TTL tiers summed). Populated by every provider that emits a cache creation count.
• CacheWrite5mTokens and CacheWrite1hTokens are the Anthropic-specific breakdown from the `cache_creation.ephemeral_*_input_tokens` response fields. Other providers (OpenAI, Gemini) leave these at zero because their cache surface is opaque or single-TTL.

The TTL breakdown is the structured signal callers use to detect "I requested CacheRetention=long but the model silently fell back to 5min" (closes issue #12). When CacheRetention=long is requested AND CacheWrite5mTokens > 0 AND CacheWrite1hTokens == 0 on the response, the 1h hold did NOT take effect — caller-side cost budgeting that assumed a 1h-cached prefix needs to adjust.

Backward compat: CacheWriteTokens semantics unchanged — still the total, regardless of TTL. CacheWrite5mTokens + CacheWrite1hTokens equals CacheWriteTokens for Anthropic responses today; the sum may be LESS than CacheWriteTokens if Anthropic ships a new TTL tier we haven't surfaced yet. CacheWriteTokens is populated from the wire's pre-aggregated cache_creation_input_tokens (NOT summed client-side), so the aggregate stays consistent with Anthropic's billing even if our enum drifts behind the API.

type VideoBlock

type VideoBlock struct {
	// Data is the raw base64-encoded video bytes for inline emission.
	// Do NOT include the "data:" URI prefix. Mutually exclusive with URI.
	Data string

	// URI is a pre-uploaded reference (Files API handle, YouTube URL,
	// or provider-specific URI). Mutually exclusive with Data.
	URI string

	// MimeType is the video's MIME type (e.g. "video/mp4"). Required
	// when Data is set; optional when only URI is set.
	MimeType string

	// StartOffset, when non-nil, clips the start of the segment. The
	// provider must support clipping; Gemini does via videoMetadata.
	StartOffset *time.Duration

	// EndOffset, when non-nil, clips the end of the segment.
	EndOffset *time.Duration

	// FPS, when non-nil, overrides the provider's default sampling rate.
	// Gemini defaults to 1 FPS (1 video frame per second analyzed);
	// pass a higher value for action-dense content or lower for long
	// static footage. Float for fractional rates (0.5 = 1 frame per 2s).
	FPS *float64
}

VideoBlock holds video data for multimodal input. Today only the built-in Gemini provider accepts video natively; Anthropic and OpenAI providers reject VideoBlock at the wire boundary (callers wanting video understanding on those providers must extract frames client-side and submit them as ImageBlocks).

VideoBlock has three mutually-exclusive emission shapes:

• **Data + MimeType set**: inline base64. Total request body must stay under the provider's inline cap (Gemini: ~20 MB).
• **URI set**: a pre-uploaded reference. For Gemini, this is either an `https://generativelanguage.googleapis.com/v1beta/files/...` handle from the Files API (see providers/gemini/files) or a YouTube URL (public videos only; free-tier 8h/day cap).
• Exactly one of (Data, URI) must be non-empty; both empty or both set is a contract violation rejected by Validate().

Optional StartOffset, EndOffset, and FPS let callers clip the segment and override Gemini's default 1 FPS sampling. nil = use the provider's default. FPS is float for fractional rates (e.g. 0.5).

MimeType uses standard video MIME identifiers: video/mp4, video/quicktime, video/webm, video/mpeg, etc. Required when Data is set; ignored when only URI is set (server infers from the file).

func (VideoBlock) Validate

func (v VideoBlock) Validate() error

Validate enforces the VideoBlock contract:

• exactly one of (Data, URI) must be non-empty
• Data must not carry a "data:" URI prefix (raw base64 only)
• MimeType is required when Data is set