ai

package module
v0.9.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 9, 2026 License: MIT Imports: 32 Imported by: 0

README


ai

CI Go Reference

Multimodal Go module for building across AI providers, with realistic cost tracking baked in, not just token counts.

From the obvious to the overlooked: chat, streaming, reasoning, tool use, structured output, embeddings, speech-to-text, text-to-speech, sound effects, music generation, realtime voice, moderation, image gen & editing, video gen.

Install

go get github.com/nurf-ai/ai

Usage

Chat

import "github.com/nurf-ai/ai"

llm := ai.NewLLMProvider(provider, apiKey, model)

resp, err := llm.Chat(ctx, []ai.Message{
    {Role: "user", Content: "Hello"},
}, nil)

Streaming

Falls back to Chat transparently.

Iterator (Go 1.23+):

llm := ai.NewLLMProvider(provider, apiKey, model)

chunks, result := ai.Stream(ctx, llm, messages, nil)

for chunk, err := range chunks {
    fmt.Print(chunk.Text)
}
resp, err := result.Response()

Channel (for select/fan-out):

ch, wait := ai.StreamWithChan(ctx, llm, messages, nil)

for chunk := range ch {
    fmt.Print(chunk.Text)
}
resp, err := wait()

Vision + structured output

Hand a multimodal turn (text + base64 images) to any provider and get schema-constrained JSON back. Providers that cannot see images return ai.ErrVisionUnsupported instead of silently dropping them; ai.WithMaxTokens raises the output budget (default 4096).

parts := []ai.Part{
    ai.ImagePart{MediaType: "image/jpeg", Data: b64},
    ai.TextPart{Text: "Describe every part of this object."},
}
ctx = ai.WithMaxTokens(ctx, 16000)
out, err := ai.StructuredOutputFromParts(ctx, llm, parts, sysPrompt, schemaJSON) // map[string]any

Embeddings

embedder := ai.NewEmbedder(provider, apiKey)
vectors, err := embedder.EmbedText(ctx, []string{"hello world"})

Image Generation

img, err := ai.NewImageProvider(ctx, provider, apiKey, model)
b64, err := img.Generate(ctx, "a cat in space", "", "")

Video Generation

// fal (LTX-2.3)
video, err := ai.NewVideoProvider("fal", apiKey, "")
res, err := video.Generate(ctx, ai.VideoRequest{
    Prompt:   "a robot adopts a stray cat",
    Image:    lastFrameJPEG, // optional first-frame conditioning
    Duration: 6, Resolution: "1080p", AspectRatio: "16:9",
})

// gemini (Omni Flash)
video, err := ai.NewVideoProvider("gemini", apiKey, "")
res, err := video.Generate(ctx, ai.VideoRequest{
    Prompt: "a robot adopts a stray cat",
    Resolution: "720p", AspectRatio: "16:9",
})

// veo (Veo 3.1 Fast — async, same key as gemini)
video, err := ai.NewVideoProvider("veo", apiKey, "")
res, err := video.Generate(ctx, ai.VideoRequest{
    Prompt: "a robot adopts a stray cat",
    Duration: 8, Resolution: "1080p",
})

// minimax (H3 direct API — async)
video, err := ai.NewVideoProvider("minimax", apiKey, "")
res, err := video.Generate(ctx, ai.VideoRequest{
    Prompt: "a robot adopts a stray cat",
    Duration: 5, Resolution: "768P",
})

// router — picks cheapest available provider per request
fal, _ := ai.NewVideoProvider("fal", falKey, "")
gemini, _ := ai.NewVideoProvider("gemini", geminiKey, "")
veo, _ := ai.NewVideoProvider("veo", geminiKey, "")
minimax, _ := ai.NewVideoProvider("minimax", minimaxKey, "")
router, _ := ai.NewVideoRouter(
    ai.WithRoute(fal),
    ai.WithRoute(gemini),
    ai.WithRoute(veo),
    ai.WithRoute(minimax),
    ai.RouteByPrice(0.7),
    ai.RouteByAvailability(0.3),
)
res, err := router.Generate(ctx, ai.VideoRequest{
    Prompt: "a robot adopts a stray cat",
    Resolution: "720p",
})
// res.Model tells you which provider was chosen

Speech-to-Text

stt := ai.NewSTTProvider(provider, apiKey, model)
text, err := stt.Transcribe(ctx, audioReader, "audio.mp3")

Text-to-Speech

Audio streams back progressively; close the reader when done. Voice, Format (mp3 default), Speed and Instructions are optional.

tts := ai.NewTTSProvider(provider, apiKey, model)
audio, err := tts.Synthesize(ctx, ai.TTSRequest{Text: "hello", Voice: "nova"})
defer audio.Close()
io.Copy(w, audio)

Sound Effects (TTSFX)

sfx := ai.NewFalAudioProvider(apiKey, "") // default: sonilo/v1.1/text-to-sound-effects
res, err := sfx.Generate(ctx, ai.AudioRequest{
    Prompt:   "thunder crack followed by heavy rain",
    Duration: 5,       // 1–180s, default 8
    Format:   "aac",   // wav, mp3, aac (default), flac
})
// res.URL, res.Duration, res.CostUSD

Music Generation (TTMusic)

music := ai.NewFalAudioProvider(apiKey, "sonilo/v1.1/text-to-music")
res, err := music.Generate(ctx, ai.AudioRequest{
    Prompt:   "upbeat lo-fi hip hop beat with soft piano chords",
    Duration: 30, // 1–600s, default 90; always AAC output
})

Realtime Voice

Bidirectional WebSocket session for conversational voice. Streams audio in and out, supports tool calling mid-conversation, and reports per-response usage with text+audio token breakdown.

rt, _ := ai.NewRealtimeProvider("openai", apiKey, "gpt-realtime-2.1-mini")

err := rt.Connect(ctx, ai.RealtimeSessionConfig{
    Voice:        "alloy",
    Instructions: "You are a helpful assistant.",
    Tools:        []ai.Tool{{Name: "lookup", Description: "Look up a record", Parameters: schema}},
})
defer rt.Close()

// send text (or audio via rt.SendAudio)
rt.AddMessage("user", "What's the weather?")
rt.CreateResponse()

for ev := range rt.Recv() {
    switch ev.Type {
    case ai.RTAudioDelta:
        speaker.Write(ev.Audio) // PCM16 24kHz mono
    case ai.RTTextDelta:
        fmt.Print(ev.Text)
    case ai.RTToolCall:
        result := handleTool(ev.ToolCall.Name, ev.ToolCall.Arguments)
        rt.SendToolResult(ev.ToolCall.CallID, result)
        rt.CreateResponse()
    case ai.RTResponseDone:
        log.Printf("tokens: %d in, %d out", ev.Usage.InputTokens, ev.Usage.OutputTokens)
    }
}

Models: gpt-realtime-2 (full), gpt-realtime-2.1-mini (faster, cheaper). Server-side VAD is on by default — configure via RealtimeSessionConfig.TurnDetection.

Metering & Pricing

ai.SetLLMMeter(llm, func(ev ai.UsageEvent) {
    log.Printf("%s: %d tokens, $%.6f", ev.Model, ev.TotalTokens, ev.EstimatedCostUSD)
})

Attribute calls via context — ai.WithMeterCallerID, ai.WithMeterOperation, ai.WithMeterMetadata(ctx, map[string]any{...}) — every provider merges stamped metadata into UsageEvent.Metadata (provider-set keys win).

Built-in per-model cost estimation via EstimateCostFull (tokens / flat per image), EstimateVideoCost (per second of video), EstimateTTSCost (per character of speech input), EstimateAudioCost (per second of generated audio), and EstimateVideoCostByTokens / EstimateImageCostByTokens (actual token counts from provider response). Rates and context windows for all supported models are maintained in models.json — the single source of truth, embedded at compile time.

Providers

Factory Providers Features
NewLLMProvider(provider, apiKey, model) anthropic, openai, gemini, ollama, huggingface Chat, streaming, structured output, tools
NewImageProvider(ctx, provider, apiKey, model) openai, gemini Image generation / editing
NewSTTProvider(provider, apiKey, model) openai Speech-to-text
NewTTSProvider(provider, apiKey, model) openai Text-to-speech
NewFalAudioProvider(apiKey, model) fal Sound effects & music generation (Sonilo)
NewVideoProvider(provider, apiKey, model) fal, gemini, veo, minimax Video generation (text/image-to-video)
NewRealtimeProvider(provider, apiKey, model) openai Realtime voice (WebSocket, bidirectional audio + tool calls)
NewEmbedder(provider, apiKey) openai Text embeddings

Each ✓ means the integration test passes, ✗ means it fails, and — means it hasn't been run yet:

Provider Model Chat Stream Reasoning Structured Output From Schema Tools Embeddings STT TTS TTSFX TTMusic RT Voice RT Voice Tools Moderation Image Gen Img Edit Img Edit Ref Txt2Vid Img2Vid Caching
Anthropic claude-haiku-4-5 ✓ ✓ ✓ ✓ ✓ ✓ ✓
OpenAI gpt-4o-mini ✓ ✓ ✓ ✓ ✓
OpenAI gpt-5-mini ✓
OpenAI text-embedding-3-small ✓
OpenAI whisper-1 ✓
OpenAI gpt-4o-mini-tts ✓
OpenAI gpt-realtime-2.1-mini ✓ ✓
OpenAI omni-moderation-latest ✓
OpenAI gpt-image-1 ✓ ✓
Gemini gemini-3.6-flash ✓ ✓ ✓ ✓ ✓
Gemini gemini-2.5-flash-image ✓ ✓ ✓
Gemini gemini-omni-1.1-flash ✓ ✓
Gemini veo-3.1-fast ✓
Hugging Face Kimi-K2-Instruct ✓ ✓ ✓ ✓ ✓
Hugging Face Kimi-K3 ✓ ✓ ✓ ✓ ✓
Ollama qwen3.5:0.8b ✓ ✓
Ollama gpt-oss:20b ✓ ✓ ✓
Ollama gemma4:e4b ✓ ✓
fal sonilo/v1.1 ✓
fal sonilo/v1.1/music ✓
fal ltx-2.3/t2v/fast ✓
fal ltx-2.3/i2v/fast ✓
fal minimax/h3-max/i2v ✓
MiniMax MiniMax-H3 — —

Contributing

Setup

git clone https://github.com/nurf-ai/ai.git
cd ai
go test ./...

Adding a provider

  1. Create <provider>.go implementing LLMProvider
  2. Add model entries to models.json (pricing + max_input_tokens)
  3. Register in provider.go (NewLLMProvider switch)
  4. Add a smoke test to integration_test.go
  5. Run both test suites before submitting

Adding a model

  1. Add entry to models.json under the provider key — include max_input_tokens
  2. go test ./... — TestPricingCoverage will fail if max_input_tokens is missing

Updating models

Edit models.json directly — it's embedded at build time via go:embed. No Go code changes needed.

Testing

Unit tests (no API keys needed, runs in CI):

go test ./...

Integration tests (real API calls, run locally):

cp .env.test.tpl .env.test   # fill in your API keys
go test -tags=integration -count=1 -json ./... | go run ./cmd/testmatrix

.env.test is loaded automatically via godotenv in TestMain. Only set the keys you have — providers with missing keys are skipped (∅).

testmatrix prints live progress with per-test token counts and cost, failure details, and per-provider cost totals to stderr:

  ✓ Anthropic/claude-haiku-4-5/Chat (1.8s)                     800 tok_in, 80 tok_out, $0.0003
  ✓ OpenAI/gpt-4o-mini/Chat (1.2s)                             500 tok_in, 50 tok_out, $0.0001
  ✓ OpenAI/gpt-4o-mini/Moderation (0.3s)
  ∅ Ollama

4 passed, 0 failed, 1 skipped

── cost ──
  Anthropic      $0.0003  (800 tok_in, 80 tok_out, 200 tok_cached)
  OpenAI         $0.0001  (500 tok_in, 50 tok_out)
  TOTAL          $0.0004  (1300 tok_in, 130 tok_out, 200 tok_cached)

[!NOTE] The capability/coverage matrix goes to stdout — paste it into the README between the <!-- testmatrix:start/end --> markers. PRs that add or change provider capabilities must include an updated matrix.

All provider tests use t.Parallel(), so subtests run concurrently within a single go test invocation. To split across CI jobs, use -run:

go test -tags=integration -v -run TestAnthropic ./...
go test -tags=integration -v -run TestOpenAI ./...
go test -tags=integration -v -run TestGemini ./...
go test -tags=integration -v -run TestHuggingFace ./...
go test -tags=integration -v -run TestOllama ./...

See .env.test.tpl for the full list of env vars.

Guidelines

  • CI runs go vet, golangci-lint, and go test -race — make sure they pass locally before submitting:
go vet ./...
gofmt -l .
golangci-lint run
go test -race ./...

  • Releases are automated via Release Please — use conventional commits and a release PR is created automatically on push to main. The prefix determines the version bump:

    Prefix Bump When to use
    feat: minor New capability in the library
    fix: patch Bug fix in library code
    docs:, chore:, test:, ci: none No release — README, tooling, tests, CI

Documentation

Overview

Package ai is the consumer-facing LLM contract — provider-agnostic types (Tool/Message/Response/ToolCall/Part) plus the LLMProvider interface in ai_contract.go. Concrete implementations (Anthropic/OpenAI/Ollama/ HuggingFace) live in sibling files; they satisfy LLMProvider structurally.

Index

Constants

View Source 
const (
	RTAudioDelta = "audio_delta"
	RTAudioDone  = "audio_done"
	RTTextDelta  = "text_delta"
	RTTextDone   = "text_done"
	// RTTranscript / RTTranscriptDone carry the assistant's spoken words as
	// text. For an audio-only response these are the only text the model
	// produces — RTTextDelta/RTTextDone fire only in the text modality.
	RTTranscript     = "transcript"
	RTTranscriptDone = "transcript_done"
	// RTInputTranscriptDelta / RTInputTranscript carry the *caller's* speech,
	// transcribed by the input transcription model.
	RTInputTranscriptDelta = "input_transcript_delta"
	RTInputTranscript      = "input_transcript"
	RTToolCall             = "tool_call"
	RTSpeechStarted        = "speech_started"
	RTSpeechStopped        = "speech_stopped"
	RTResponseDone         = "response_done"
	RTError                = "error"
)

Realtime event types emitted on RealtimeProvider.Recv().

View Source 
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants used in Message.Role.

View Source 
const DefaultFalAudioModel = "sonilo/v1.1/text-to-sound-effects"
View Source 
const DefaultFalQueueBase = "https://queue.fal.run"

DefaultFalQueueBase is the fal queue API root. Every fal model endpoint is addressed as {base}/{model_id}; the queue returns absolute status/result URLs which the client follows verbatim.

View Source 
const DefaultFalVideoModel = "fal-ai/ltx-2.3/image-to-video/fast"

DefaultFalVideoModel is the fal endpoint used when none is configured: LTX-2.3 fast, image-to-video (accepts a first-frame image_url).

View Source 
const (
	DefaultGeminiVideoModel = "gemini-omni-1.1-flash"
)
View Source 
const DefaultHuggingFaceBaseURL = "https://router.huggingface.co/v1"

DefaultHuggingFaceBaseURL is the HF Inference router's OpenAI-compatible endpoint. Override via HF_BASE_URL when targeting a dedicated endpoint or a specific upstream provider route (e.g. https://router.huggingface.co/<provider>/v1).

View Source 
const (
	DefaultMinimaxVideoModel = "MiniMax-H3"
)
View Source 
const (
	DefaultVeoVideoModel = "veo-3.1-fast-generate-preview"
)

Variables

View Source 
var ErrVisionUnsupported = errors.New("ai: provider does not accept image input for structured output")

ErrVisionUnsupported is returned when an ImagePart is handed to a provider that only accepts text for structured output. The image is never silently dropped — a spec authored without the reference it was asked to describe is worse than an error.

Functions

func CacheSysPromptFromCtx 

func CacheSysPromptFromCtx(ctx context.Context) bool

CacheSysPromptFromCtx reports whether WithCacheSysPrompt was stamped. Adapters call this while building the request to decide whether to mark the sys block as cacheable.

func CountTokens 

func CountTokens(s string) int

CountTokens returns an approximate token count for s using the o200k_base BPE encoding. Exact for GPT-4o / GPT-5 family; within ~15% for Anthropic and HF model families. Empty string returns 0.

Falls back to a len(s)/4 heuristic if the tiktoken encoder fails to initialize, so this never returns an error — a usable rough number is more helpful at call sites than a plumbed error path.

func CountTokensOver 

func CountTokensOver(s string, limit int64) bool

CountTokensOver reports whether counting tokens in s exceeds limit. Short-circuits using the cheap char-based upper bound before invoking the BPE encoder, so callers can gate long strings without paying the full tokenization cost when they're obviously under the limit.

func DataURI added in v0.2.0 

func DataURI(mediaType string, data []byte) string

DataURI encodes raw bytes as a base64 data: URI.

func DebugPromptsEnabled 

func DebugPromptsEnabled() bool

DebugPromptsEnabled reports whether prompt capture is active. Providers check this before building sys/user prompt strings for the debug panel.

func DebugSpanIDFromCtx 

func DebugSpanIDFromCtx(ctx context.Context) string

DebugSpanIDFromCtx returns the span ID on the context, or empty.

func EstimateAudioCost added in v0.9.0 

func EstimateAudioCost(model string, seconds float64) float64

EstimateAudioCost prices a sound-effect generation from its output duration. Unknown models are recorded as 0.

func EstimateCostFull 

func EstimateCostFull(model string, in, out, cw, cr int) float64

func EstimateImageCostByTokens added in v0.4.0 

func EstimateImageCostByTokens(model string, outputTokens int) float64

EstimateImageCostByTokens prices an image generation from output tokens. Falls back to FlatPerImage when the model has no per-token image pricing.

func EstimateRealtimeCost added in v0.8.0 

func EstimateRealtimeCost(model string, textIn, textOut, audioIn, audioOut int) float64

EstimateRealtimeCost prices a realtime voice response from its text and audio token counts. Text tokens use the standard input/output rates; audio tokens use the audio-specific rates.

func EstimateTTSCost added in v0.7.0 

func EstimateTTSCost(model string, chars int) float64

EstimateTTSCost prices a speech synthesis from its input character count. Unknown models are recorded as 0 (and reported via the unknown-model hook), like EstimateCostFull.

func EstimateVideoCost added in v0.2.0 

func EstimateVideoCost(model string, seconds float64, resolution string) float64

EstimateVideoCost prices seconds of generated video for a per-second model. resolution selects a per-resolution rate when the table has one; otherwise the base rate applies. Unknown models are recorded as 0 (and reported via the unknown-model hook), like EstimateCostFull.

func EstimateVideoCostByTokens added in v0.4.0 

func EstimateVideoCostByTokens(model string, inputTokens, videoOutputTokens int) float64

EstimateVideoCostByTokens prices a video generation from actual token counts returned by the provider. Falls back to 0 when the model has no per-token video pricing.

func GenerateTyped 

func GenerateTyped[T any](ctx context.Context, p LLMProvider, userPrompt, sysPrompt string) (*T, error)

GenerateTyped wraps CreateStructuredOutput with Go generics so callers get back a typed pointer without pre-allocating the output struct.

func IsAudioModel added in v0.9.0 

func IsAudioModel(model string) bool

IsAudioModel reports whether model is priced per second of audio output.

func IsFalError added in v0.2.0 

func IsFalError(err error) bool

IsFalError reports whether err wraps a *FalError.

func IsRealtimeModel added in v0.8.0 

func IsRealtimeModel(model string) bool

IsRealtimeModel reports whether model has audio token pricing.

func IsTTSModel added in v0.7.0 

func IsTTSModel(model string) bool

IsTTSModel reports whether model is priced per character of speech input.

func IsVideoModel added in v0.2.0 

func IsVideoModel(model string) bool

IsVideoModel reports whether model is priced per second of video.

func MaxInputTokensLLM 

func MaxInputTokensLLM(company, modelName string) (int64, error)

MaxInputTokensLLM returns the advertised maximum input context window in tokens for the given model.

company is the model-family owner / API namespace — one of "anthropic", "openai", or "huggingface". modelName is the exact model id as passed to the provider constructor (for HF that's the full "<org>/<model>" id). Trailing date suffixes like "-20250514" are stripped automatically.

Returns an error if the model is unknown. Callers that need a safe default should handle the error explicitly rather than rely on a fallback — silently returning 0 or a guess would mask config typos.

func MaxTokensFromCtx 

func MaxTokensFromCtx(ctx context.Context, fallback int) int

func MeterCallerIDFromCtx added in v0.0.2 

func MeterCallerIDFromCtx(ctx context.Context) uuid.UUID

func MeterMetadataFromCtx added in v0.3.0 

func MeterMetadataFromCtx(ctx context.Context) map[string]any

MeterMetadataFromCtx returns a copy of the metadata stamped on ctx via WithMeterMetadata, or nil when none. Mutating the result never touches the context.

func MeterOperationFromCtx 

func MeterOperationFromCtx(ctx context.Context) string

func ModelCompany added in v0.5.0 

func ModelCompany(model string) string

ModelCompany returns the models.json group a model id belongs to ("fal", "gemini", "minimax", …), or "" for an unknown model.

func ModelsForProvider 

func ModelsForProvider(provider string) []string

func PartsText added in v0.2.0 

func PartsText(parts []Part) (string, bool)

PartsText joins the TextParts of a multimodal turn with blank lines and reports whether any ImagePart was present. Used for moderation, usage attribution and text-only fallbacks.

func PricingTable 

func PricingTable() map[string]modelPricing

func ReasoningEffortFromCtx 

func ReasoningEffortFromCtx(ctx context.Context) string

func SetAudioMeter added in v0.9.0 

func SetAudioMeter(a AudioProvider, hook MeterHook)

SetAudioMeter attaches a meter hook to any AudioProvider that satisfies AudioMeterable.

func SetDebugPromptsEnabled 

func SetDebugPromptsEnabled(enabled bool)

SetDebugPromptsEnabled toggles prompt capture in UsageEvents. Call once at startup based on your dev-mode flag.

func SetImageMeter 

func SetImageMeter(img ImageProvider, hook MeterHook)

SetImageMeter attaches a meter hook to any ImageProvider that satisfies ImageMeterable.

func SetImageModeration 

func SetImageModeration(img ImageProvider, m ModerationProvider)

SetImageModeration attaches a moderation provider to any ImageProvider that satisfies the moderable interface.

func SetLLMMeter 

func SetLLMMeter(llm LLMProvider, hook MeterHook)

SetLLMMeter attaches a meter hook to any LLMProvider that satisfies LLMMeterable.

func SetLLMModeration 

func SetLLMModeration(llm LLMProvider, m ModerationProvider)

SetLLMModeration attaches a moderation provider to any LLMProvider that satisfies LLMModerable.

func SetLogger 

func SetLogger(l *zap.Logger)

func SetOnModerationError 

func SetOnModerationError(fn func(ctx context.Context, err error) error)

SetOnModerationError overrides the default moderation-error behavior. Default (nil): warn and allow. When set, the function's return value determines whether the call proceeds (nil) or is rejected (non-nil error).

func SetRealtimeMeter added in v0.8.0 

func SetRealtimeMeter(p RealtimeProvider, hook MeterHook)

SetRealtimeMeter attaches a meter hook to any RealtimeProvider that satisfies RealtimeMeterable.

func SetTTSMeter added in v0.7.0 

func SetTTSMeter(t TTSProvider, hook MeterHook)

SetTTSMeter attaches a meter hook to any TTSProvider that satisfies TTSMeterable.

func SetUnknownModelHook 

func SetUnknownModelHook(hook func(model string))

func SetVideoMeter added in v0.2.0 

func SetVideoMeter(v VideoProvider, hook MeterHook)

SetVideoMeter attaches a meter hook to any VideoProvider that satisfies VideoMeterable.

func SetVideoModeration added in v0.2.0 

func SetVideoModeration(v VideoProvider, m ModerationProvider)

SetVideoModeration attaches a moderation provider to any VideoProvider that satisfies the moderable interface.

func StreamWithChan added in v0.0.2 

func StreamWithChan(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (<-chan StreamChunk, func() (*Response, error))

StreamWithChan starts streaming and returns a channel of chunks plus a function that blocks until done and returns the accumulated response. The channel is closed when streaming completes. If the provider does not implement StreamingProvider, falls back to Chat.

func StructuredOutputFromParts added in v0.2.0 

func StructuredOutputFromParts(ctx context.Context, p LLMProvider, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

StructuredOutputFromParts routes a multimodal structured-output request to p. Providers implementing MultimodalStructuredProvider receive the parts verbatim; any other provider receives the text parts joined, and ErrVisionUnsupported when an ImagePart would otherwise be lost.

func TTSMimeType added in v0.7.0 

func TTSMimeType(format string) string

TTSMimeType returns the MIME type for a TTSRequest.Format value.

func TransparentBGFromCtx 

func TransparentBGFromCtx(ctx context.Context) bool

func TruncatePromptForDebug 

func TruncatePromptForDebug(s string) string

TruncatePromptForDebug is kept for callers that still reference the old helper name; it now just delegates to capturePromptForDebug — prompts are no longer truncated. Dev mode only carries them at all.

func VideoModels added in v0.3.0 

func VideoModels() []string

VideoModels lists every priced video model id, sorted.

func WithCacheSysPrompt 

func WithCacheSysPrompt(ctx context.Context) context.Context

WithCacheSysPrompt signals that the sys prompt sent in this call is a stable prefix worth provider-side prompt caching. Anthropic's adapter sets cache_control: ephemeral on the sys block; OpenAI/Ollama/HF are no-op (OpenAI auto-caches prompts ≥1024 tokens; Ollama uses KV-cache for matching prefixes at the inference layer; HF depends on backend).

Call this at the call site right before the LLM invocation. Marker is per-call, not global — stamp only when the caller knows the sys prompt is stable across turns (e.g. planner/router system prompts).

func WithDebugSpanID 

func WithDebugSpanID(ctx context.Context, spanID string) context.Context

WithDebugSpanID stamps the current dev-debug span ID on the context so providers can include it on UsageEvents. Caller must be in dev mode; in prod this is a no-op path (the middleware chain never calls StartSpan).

func WithMaxTokens 

func WithMaxTokens(ctx context.Context, n int) context.Context

func WithMeterCallerID added in v0.0.2 

func WithMeterCallerID(ctx context.Context, id uuid.UUID) context.Context

func WithMeterMetadata added in v0.3.0 

func WithMeterMetadata(ctx context.Context, kv map[string]any) context.Context

WithMeterMetadata stamps arbitrary key/values on the context so every provider merges them into UsageEvent.Metadata (surf handle, session id, feature flag — whatever the caller wants attributed). Stacking calls merges kv over what is already stamped, later keys win, into a fresh map: the stored map is copied, never mutated, and kv is copied too so the caller may reuse it. Empty/nil kv returns ctx unchanged.

func WithMeterOperation 

func WithMeterOperation(ctx context.Context, op string) context.Context

func WithPromptBlocks 

func WithPromptBlocks(ctx context.Context, contents map[string]string) context.Context

WithPromptBlocks stamps a per-block breakdown on the context so providers can include it on UsageEvents. Call-site passes the raw block strings keyed by block name — this helper computes both Chars (len(s)) and Tokens (CountTokens(s)) per entry.

Tokenizer cost is paid here, but `dev-only` callers and a single tokenize per call mean it's negligible vs the LLM round-trip. If you need to skip the tokenize (e.g. hot path that doesn't care about tokens), call WithPromptBlocksRaw with a pre-built map[string]BlockSize directly.

Empty maps are intentionally not stored — PromptBlocksFromCtx returning nil is the canonical "absent" signal.

func WithPromptBlocksRaw 

func WithPromptBlocksRaw(ctx context.Context, b PromptBlocks) context.Context

WithPromptBlocksRaw is the lower-level variant — caller pre-computes Chars and Tokens per block. Useful when the same block string is reused across calls and you want to cache the token count.

func WithReasoningEffort 

func WithReasoningEffort(ctx context.Context, effort string) context.Context

WithReasoningEffort sets the reasoning effort level ("low", "medium", "high") for providers that support it. OpenAI: maps to reasoning_effort in chat completions. Anthropic: maps to extended thinking with a budget derived from effort level.

func WithTransparentBG 

func WithTransparentBG(ctx context.Context) context.Context

WithTransparentBG signals image providers to produce transparent backgrounds.

Types

type AnthropicProvider 

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

func NewAnthropicProvider 

func NewAnthropicProvider(apiKey, model string) *AnthropicProvider

func (*AnthropicProvider) Chat 

func (p *AnthropicProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*AnthropicProvider) ChatStream added in v0.0.2 

func (p *AnthropicProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*AnthropicProvider) CreateStructuredOutput 

func (p *AnthropicProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*AnthropicProvider) CreateStructuredOutputBreakpointed 

func (p *AnthropicProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM by emitting two system blocks each marked with cache_control: ephemeral. The provider hashes the prefix up through each cache_control marker, so when sysPrompt + stableMid is byte-stable across turns we hit the bp2 entry (everything stable cached); when only sysPrompt is stable (e.g. the candidate list changed) we hit bp1 (sysPrompt cached, stableMid reprocessed). One marker per change-rate tier — see .wiki/context/cache-breakpoints.md.

dynamicTail is the per-call query and rides as the user message — never cached.

func (*AnthropicProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *AnthropicProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images). Honours WithMaxTokens (default 4096): a vision-authored spec routinely needs more.

func (*AnthropicProvider) CreateStructuredOutputFromSchema 

func (p *AnthropicProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*AnthropicProvider) MaxInputTokens 

func (p *AnthropicProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*AnthropicProvider) Model 

func (p *AnthropicProvider) Model() string

func (*AnthropicProvider) Name 

func (p *AnthropicProvider) Name() string

func (*AnthropicProvider) SetMeter 

func (p *AnthropicProvider) SetMeter(hook MeterHook)

func (*AnthropicProvider) SetModeration 

func (p *AnthropicProvider) SetModeration(m ModerationProvider)

func (*AnthropicProvider) WithMeter 

func (p *AnthropicProvider) WithMeter(hook MeterHook) *AnthropicProvider

func (*AnthropicProvider) WithModeration 

func (p *AnthropicProvider) WithModeration(m ModerationProvider) *AnthropicProvider

type AudioMeterable added in v0.9.0 

type AudioMeterable interface {
	SetMeter(MeterHook)
}

AudioMeterable is implemented by audio providers that accept a meter hook.

type AudioProvider added in v0.9.0 

type AudioProvider interface {
	Name() string
	Model() string
	Generate(ctx context.Context, req AudioRequest) (*AudioResult, error)
}

AudioProvider generates sound effects from a text prompt.

type AudioRequest added in v0.9.0 

type AudioRequest struct {
	Prompt string
	// Duration in seconds; 0 = provider default.
	Duration int
	// Format is the audio container: wav, mp3, aac (default), flac.
	Format string
}

AudioRequest describes a single sound-effect generation.

type AudioResult added in v0.9.0 

type AudioResult struct {
	URL         string
	ContentType string
	FileName    string
	FileSize    int64
	Duration    float64 // seconds actually generated
	Model       string
	CostUSD     float64
	Elapsed     time.Duration
}

AudioResult is the provider-agnostic outcome of a Generate call.

type BlockSize 

type BlockSize struct {
	Chars  int `json:"chars"`
	Tokens int `json:"tokens"`
}

BlockSize carries the two attribution signals per prompt block: chars (raw length) and tokens (tiktoken o200k_base count). Tokens are what bills you; chars exist for sanity (a high tokens/chars ratio per block flags tokenizer-hostile content like code, base64, or heavy unicode — surfd as the "density" warning in PromptBlocksBar).

type CacheControl 

type CacheControl struct {
	Type string `json:"type"` // "ephemeral"
}

CacheControl tells a provider to mark this message (or its last content block) as a cache breakpoint. Only Anthropic acts on it today — other providers ignore it silently.

type Dimension added in v0.4.0 

type Dimension struct {
	Name      string
	Weight    float64
	CanHandle func(provider string) bool
	Score     func(ctx context.Context, provider string) (float64, error)
}

Dimension scores providers on one optimization axis. Lower scores are better. Route normalizes scores across candidates before applying weights, so raw scale does not matter.

func AvailabilityDim added in v0.4.0 

func AvailabilityDim(weight float64, probeFn func(ctx context.Context, provider string) (float64, error)) Dimension

AvailabilityDim scores by estimated latency (seconds) — lower is better.

func CapabilityDim added in v0.4.0 

func CapabilityDim(canHandle func(provider string) bool) Dimension

CapabilityDim is a filter-only dimension that excludes providers that cannot handle the request. It has no score weight.

func PriceDim added in v0.4.0 

func PriceDim(weight float64, costFn func(provider string) float64) Dimension

PriceDim scores by cost — lower is better.

type Embedder 

type Embedder interface {
	EmbedText(ctx context.Context, texts []string) ([][]float32, error)
	EmbedDimensions() int
}

Embedder produces vector embeddings for text. Consumers that need semantic similarity (discovery, memory recall, search ranking) depend on this interface. Not all LLM providers support embeddings — wire a separate provider if the main LLM doesn't (e.g. Anthropic + OpenAI embedding sidecar).

func EmbedderFromLLM 

func EmbedderFromLLM(llm LLMProvider) Embedder

EmbedderFromLLM extracts an Embedder from an LLMProvider if it satisfies the Embedder interface (e.g. OpenAIProvider).

func NewEmbedder 

func NewEmbedder(providerName, apiKey string) Embedder

NewEmbedder creates an Embedder from a provider name + API key. Only OpenAI is supported for now — returns nil for other providers.

type ErrorKind 

type ErrorKind int
const (
	ErrUnknown ErrorKind = iota
	ErrBilling
	ErrAuth
	ErrRateLimit
	ErrModeration
	ErrProviderDown
)

func ClassifyError 

func ClassifyError(err error) (ErrorKind, string)

func (ErrorKind) Code added in v0.7.2 

func (k ErrorKind) Code() string

Code is the stable, greppable name of what happened: the field to switch on in a client, a log query or a dashboard, since the prose deliberately does not distinguish. `ai_unavailable` means stop retrying — a person has to fix an account somewhere; `ai_busy` clears on its own.

func (ErrorKind) Retryable 

func (k ErrorKind) Retryable() bool

func (ErrorKind) UserMessage 

func (k ErrorKind) UserMessage() string

func (ErrorKind) ViewerMessage added in v0.7.2 

func (k ErrorKind) ViewerMessage() string

ViewerMessage is what a person who cannot fix it should read. The provider account and its key are the platform's, not theirs and not the surf owner's, so the only thing worth telling them is whether waiting helps. Billing and auth deliberately collapse into one line: "spent" and "bad key" both mean the platform's setup is broken, and telling them apart is a free hint about somebody's secret. The true cause travels in the logs and in Code().

type FalAudioProvider added in v0.9.0 

type FalAudioProvider struct {
	// contains filtered or unexported fields
}

FalAudioProvider implements AudioProvider on top of fal.ai's Sonilo model.

func NewFalAudioProvider added in v0.9.0 

func NewFalAudioProvider(apiKey, model string, opts ...FalOption) *FalAudioProvider

func (*FalAudioProvider) Generate added in v0.9.0 

func (p *FalAudioProvider) Generate(ctx context.Context, req AudioRequest) (*AudioResult, error)

func (*FalAudioProvider) Model added in v0.9.0 

func (p *FalAudioProvider) Model() string

func (*FalAudioProvider) Name added in v0.9.0 

func (p *FalAudioProvider) Name() string

func (*FalAudioProvider) SetMeter added in v0.9.0 

func (p *FalAudioProvider) SetMeter(hook MeterHook)

type FalClient added in v0.2.0 

type FalClient struct {
	// contains filtered or unexported fields
}

FalClient is a minimal client for fal.ai's HTTP queue API (submit → poll status → fetch result). It is model-agnostic: any fal endpoint that takes a JSON input and returns a JSON output can be driven through Run.

func NewFalClient added in v0.2.0 

func NewFalClient(apiKey string, opts ...FalOption) *FalClient

NewFalClient creates a fal queue client authenticated with apiKey.

func (*FalClient) Cancel added in v0.2.0 

func (c *FalClient) Cancel(ctx context.Context, req *FalRequest) error

Cancel asks the queue to drop a request that has not started yet.

func (*FalClient) Result added in v0.2.0 

func (c *FalClient) Result(ctx context.Context, req *FalRequest) (json.RawMessage, error)

Result fetches the model output. fal answers with a non-2xx status and a `detail` body when the run failed; that surfaces as *FalError.

func (*FalClient) Run added in v0.2.0 

func (c *FalClient) Run(ctx context.Context, endpoint string, input any) (json.RawMessage, error)

Run submits input, waits for completion and returns the raw output JSON.

func (*FalClient) Status added in v0.2.0 

func (c *FalClient) Status(ctx context.Context, req *FalRequest) (*FalStatus, error)

Status polls the request once.

func (*FalClient) Submit added in v0.2.0 

func (c *FalClient) Submit(ctx context.Context, endpoint string, input any) (*FalRequest, error)

Submit enqueues input on endpoint (e.g. "fal-ai/ltx-2.3/image-to-video/fast").

func (*FalClient) Wait added in v0.2.0 

func (c *FalClient) Wait(ctx context.Context, req *FalRequest) (*FalStatus, error)

Wait polls until the request completes or ctx is done.

type FalError added in v0.2.0 

type FalError struct {
	Status   int
	Endpoint string
	Message  string
}

FalError is a non-2xx response from any fal endpoint.

func (*FalError) Error added in v0.2.0 

func (e *FalError) Error() string

type FalOption added in v0.2.0 

type FalOption func(*FalClient)

FalOption configures a FalClient.

func WithFalHTTPClient added in v0.2.0 

func WithFalHTTPClient(h *http.Client) FalOption

WithFalHTTPClient overrides the HTTP client (timeouts, transport, tests).

func WithFalPollInterval added in v0.2.0 

func WithFalPollInterval(d time.Duration) FalOption

WithFalPollInterval sets the initial status poll interval. It backs off geometrically up to 5s.

func WithFalQueueBase added in v0.2.0 

func WithFalQueueBase(base string) FalOption

WithFalQueueBase overrides the queue API root (tests, proxies).

type FalRequest added in v0.2.0 

type FalRequest struct {
	Endpoint      string `json:"-"`
	RequestID     string `json:"request_id"`
	StatusURL     string `json:"status_url"`
	ResponseURL   string `json:"response_url"`
	CancelURL     string `json:"cancel_url"`
	QueuePosition int    `json:"queue_position"`
}

FalRequest is the queue ticket returned by Submit.

type FalStatus added in v0.2.0 

type FalStatus struct {
	Status        string `json:"status"`
	QueuePosition int    `json:"queue_position"`
	Metrics       struct {
		InferenceTime float64 `json:"inference_time"`
	} `json:"metrics"`
}

FalStatus is one status poll.

func (*FalStatus) Done added in v0.2.0 

func (s *FalStatus) Done() bool

Done reports whether the request has finished (success or failure — the result fetch tells which).

type FalVideoProvider added in v0.2.0 

type FalVideoProvider struct {
	// contains filtered or unexported fields
}

FalVideoProvider implements VideoProvider on top of fal.ai's queue API.

The model string is the full fal endpoint id (e.g. "fal-ai/ltx-2.3/image-to-video/fast"). Input keys follow the LTX family schema (prompt, image_url, duration, resolution, aspect_ratio, fps, generate_audio, seed, negative_prompt); zero-valued request fields are omitted so other fal video endpoints that share those names keep working.

func NewFalVideoProviderWithClient added in v0.2.0 

func NewFalVideoProviderWithClient(client *FalClient, model string) *FalVideoProvider

NewFalVideoProviderWithClient builds a provider around an existing client (tests, shared clients).

func (*FalVideoProvider) Client added in v0.2.0 

func (p *FalVideoProvider) Client() *FalClient

func (*FalVideoProvider) Generate added in v0.2.0 

func (p *FalVideoProvider) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

Generate runs one clip generation and blocks until fal returns the file.

func (*FalVideoProvider) Model added in v0.2.0 

func (p *FalVideoProvider) Model() string

func (*FalVideoProvider) Name added in v0.2.0 

func (p *FalVideoProvider) Name() string

func (*FalVideoProvider) SetMeter added in v0.2.0 

func (p *FalVideoProvider) SetMeter(hook MeterHook)

func (*FalVideoProvider) SetModeration added in v0.2.0 

func (p *FalVideoProvider) SetModeration(m ModerationProvider)

type GeminiImageProvider 

type GeminiImageProvider struct {
	// contains filtered or unexported fields
}

GeminiImageProvider implements ImageProvider using Google's Gemini API.

func (*GeminiImageProvider) Edit 

func (p *GeminiImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) EditWithReference 

func (p *GeminiImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) Generate 

func (p *GeminiImageProvider) Generate(ctx context.Context, prompt, model, _ string) (string, error)

func (*GeminiImageProvider) SetMeter 

func (p *GeminiImageProvider) SetMeter(hook MeterHook)

func (*GeminiImageProvider) SetModeration 

func (p *GeminiImageProvider) SetModeration(m ModerationProvider)

func (*GeminiImageProvider) WithModeration 

func (p *GeminiImageProvider) WithModeration(m ModerationProvider) *GeminiImageProvider

type GeminiProvider added in v0.0.2 

type GeminiProvider struct {
	// contains filtered or unexported fields
}

func NewGeminiProvider added in v0.0.2 

func NewGeminiProvider(ctx context.Context, apiKey, model string) (*GeminiProvider, error)

func (*GeminiProvider) Chat added in v0.0.2 

func (p *GeminiProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*GeminiProvider) ChatStream added in v0.0.2 

func (p *GeminiProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*GeminiProvider) CreateStructuredOutput added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*GeminiProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *GeminiProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images).

func (*GeminiProvider) CreateStructuredOutputFromSchema added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*GeminiProvider) MaxInputTokens added in v0.0.2 

func (p *GeminiProvider) MaxInputTokens() (int64, error)

func (*GeminiProvider) Model added in v0.0.2 

func (p *GeminiProvider) Model() string

func (*GeminiProvider) Name added in v0.0.2 

func (p *GeminiProvider) Name() string

func (*GeminiProvider) SetMeter added in v0.0.2 

func (p *GeminiProvider) SetMeter(hook MeterHook)

func (*GeminiProvider) SetModeration added in v0.0.2 

func (p *GeminiProvider) SetModeration(m ModerationProvider)

func (*GeminiProvider) WithMeter added in v0.0.2 

func (p *GeminiProvider) WithMeter(hook MeterHook) *GeminiProvider

func (*GeminiProvider) WithModeration added in v0.0.2 

func (p *GeminiProvider) WithModeration(m ModerationProvider) *GeminiProvider

type GeminiVideoProvider added in v0.4.0 

type GeminiVideoProvider struct {
	// contains filtered or unexported fields
}

GeminiVideoProvider implements VideoProvider via the Gemini Interactions API.

func (*GeminiVideoProvider) Generate added in v0.4.0 

func (p *GeminiVideoProvider) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

func (*GeminiVideoProvider) Model added in v0.4.0 

func (p *GeminiVideoProvider) Model() string

func (*GeminiVideoProvider) Name added in v0.4.0 

func (p *GeminiVideoProvider) Name() string

func (*GeminiVideoProvider) SetMeter added in v0.4.0 

func (p *GeminiVideoProvider) SetMeter(hook MeterHook)

func (*GeminiVideoProvider) SetModeration added in v0.4.0 

func (p *GeminiVideoProvider) SetModeration(m ModerationProvider)

type HTTPMeterEmitter 

type HTTPMeterEmitter struct {
	// contains filtered or unexported fields
}

HTTPMeterEmitter buffers usage events and POSTs them to a meter service.

func NewHTTPMeterEmitter 

func NewHTTPMeterEmitter(opts HTTPMeterOpts) *HTTPMeterEmitter

func (*HTTPMeterEmitter) Hook 

func (e *HTTPMeterEmitter) Hook() MeterHook

type HTTPMeterOpts 

type HTTPMeterOpts struct {
	Endpoint      string                             // full URL — no suffix appended
	AuthHeader    string                             // sent as Authorization header when non-empty
	BatchSize     int                                // default 32
	FlushInterval time.Duration                      // default 2s
	Marshal       func([]UsageEvent) ([]byte, error) // default json.Marshal
	ContentType   string                             // default "application/json"
	OnError       func(error)                        // default logger.Warn
}

HTTPMeterOpts configures an HTTPMeterEmitter.

type HuggingFaceProvider 

type HuggingFaceProvider struct {
	// contains filtered or unexported fields
}

HuggingFaceProvider speaks the HF Inference Router, which exposes an OpenAI-compatible Chat Completions API. Models are addressed by their canonical HF id (e.g. "meta-llama/Llama-3.3-70B-Instruct").

func NewHuggingFaceProvider 

func NewHuggingFaceProvider(apiKey, model, baseURL string) *HuggingFaceProvider

NewHuggingFaceProvider builds a provider against the HF router. baseURL may be empty to use DefaultHuggingFaceBaseURL.

func (*HuggingFaceProvider) Chat 

func (p *HuggingFaceProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*HuggingFaceProvider) ChatStream added in v0.0.2 

func (p *HuggingFaceProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*HuggingFaceProvider) CreateStructuredOutput 

func (p *HuggingFaceProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*HuggingFaceProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *HuggingFaceProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images).

func (*HuggingFaceProvider) CreateStructuredOutputFromSchema 

func (p *HuggingFaceProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*HuggingFaceProvider) MaxInputTokens 

func (p *HuggingFaceProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*HuggingFaceProvider) Model 

func (p *HuggingFaceProvider) Model() string

func (*HuggingFaceProvider) Name 

func (p *HuggingFaceProvider) Name() string

func (*HuggingFaceProvider) RawClient 

func (p *HuggingFaceProvider) RawClient() *openai.Client

func (*HuggingFaceProvider) SetMeter 

func (p *HuggingFaceProvider) SetMeter(hook MeterHook)

func (*HuggingFaceProvider) SetModeration 

func (p *HuggingFaceProvider) SetModeration(m ModerationProvider)

func (*HuggingFaceProvider) WithMeter 

func (p *HuggingFaceProvider) WithMeter(hook MeterHook) *HuggingFaceProvider

func (*HuggingFaceProvider) WithModeration 

func (p *HuggingFaceProvider) WithModeration(m ModerationProvider) *HuggingFaceProvider

type ImageMeterable 

type ImageMeterable interface {
	SetMeter(MeterHook)
}

ImageMeterable is implemented by image providers that accept a meter hook.

type ImagePart 

type ImagePart struct {
	MediaType string // e.g. "image/png"
	Data      string // base64-encoded
}

ImagePart carries base64-encoded image data.

type ImageProvider 

type ImageProvider interface {
	Generate(ctx context.Context, prompt, model, size string) (string, error)                                 // base64
	Edit(ctx context.Context, image []byte, editPrompt string) (string, error)                                // base64
	EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error) // base64
}

ImageProvider generates and edits images via an AI model.

func NewImageProvider 

func NewImageProvider(ctx context.Context, providerName, apiKey, model string) (ImageProvider, error)

NewImageProvider creates an ImageProvider from a provider name ("openai" or "gemini").

type LLMMeterable 

type LLMMeterable interface {
	SetMeter(MeterHook)
}

LLMMeterable is implemented by providers that accept a meter hook. SetLLMMeter uses this instead of a type-switch so new providers work without updating the switch.

type LLMModerable 

type LLMModerable interface {
	SetModeration(ModerationProvider)
}

LLMModerable is implemented by providers that accept a moderation provider.

type LLMProvider 

type LLMProvider interface {
	Name() string
	Model() string
	CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error
	CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)
	Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)
}

LLMProvider is the consumer-facing contract for any LLM backend.

func NewLLMProvider 

func NewLLMProvider(providerName, apiKey, model string, opts ...ProviderOption) LLMProvider

NewLLMProvider creates an LLMProvider from a provider name ("openai", "anthropic", "gemini", "ollama", or "huggingface"). For ollama, apiKey is the base URL (e.g. "http://ollama:11434/v1"). For huggingface, apiKey is the HF token; base URL is read from HF_BASE_URL (defaults to DefaultHuggingFaceBaseURL). For gemini, apiKey is the Google AI API key.

type Message 

type Message struct {
	Role         string        `json:"role"`
	Content      string        `json:"content,omitempty"`
	Parts        []Part        `json:"-"`
	ToolCalls    []ToolCall    `json:"tool_calls,omitempty"`
	ToolCallID   string        `json:"tool_call_id,omitempty"`
	CacheControl *CacheControl `json:"cache_control,omitempty"`
}

Message is a provider-agnostic chat message.

func (Message) MarshalJSON 

func (m Message) MarshalJSON() ([]byte, error)

func (*Message) UnmarshalJSON 

func (m *Message) UnmarshalJSON(data []byte) error

type MeterHook 

type MeterHook func(UsageEvent)

MeterHook is called after each LLM/image call with usage data.

type Middleware 

type Middleware func(next ModelFunc) ModelFunc

Middleware wraps a ModelFunc with cross-cutting behaviour.

func Chain 

func Chain(mws ...Middleware) Middleware

Chain composes middleware so the first in the list is outermost.

type MinimaxVideoProvider added in v0.4.0 

type MinimaxVideoProvider struct {
	// contains filtered or unexported fields
}

func (*MinimaxVideoProvider) Generate added in v0.4.0 

func (p *MinimaxVideoProvider) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

func (*MinimaxVideoProvider) Model added in v0.4.0 

func (p *MinimaxVideoProvider) Model() string

func (*MinimaxVideoProvider) Name added in v0.4.0 

func (p *MinimaxVideoProvider) Name() string

func (*MinimaxVideoProvider) SetMeter added in v0.4.0 

func (p *MinimaxVideoProvider) SetMeter(hook MeterHook)

func (*MinimaxVideoProvider) SetModeration added in v0.4.0 

func (p *MinimaxVideoProvider) SetModeration(m ModerationProvider)

type ModelFunc 

type ModelFunc func(ctx context.Context, msgs []Message, tools []Tool) (*Response, error)

ModelFunc is the signature of an LLM Chat call, abstracted from any concrete provider.

type ModerationError 

type ModerationError struct {
	Categories map[string]bool
}

ModerationError is returned when a prompt is blocked by moderation.

func (*ModerationError) Error 

func (e *ModerationError) Error() string

type ModerationProvider 

type ModerationProvider interface {
	Check(ctx context.Context, input string) (*ModerationResult, error)
}

ModerationProvider checks user-supplied text for policy violations.

func NewModerationProvider 

func NewModerationProvider(providerName, apiKey string) ModerationProvider

NewModerationProvider creates a ModerationProvider from a provider name.

type ModerationResult 

type ModerationResult struct {
	Flagged    bool            `json:"flagged"`
	Categories map[string]bool `json:"categories,omitempty"`
}

ModerationResult holds the outcome of a moderation check.

type MultimodalStructuredProvider added in v0.2.0 

type MultimodalStructuredProvider interface {
	CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)
}

MultimodalStructuredProvider is implemented by providers that accept a multimodal user turn (text + images) for schema-constrained structured output. Every built-in provider implements it; third-party LLMProvider implementations may not, which is why StructuredOutputFromParts exists.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider wraps the OpenAI-compatible API exposed by Ollama.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) *OllamaProvider

NewOllamaProvider creates a provider pointing at an Ollama instance. baseURL is the Ollama server, e.g. "http://ollama:11434/v1".

func (*OllamaProvider) Chat 

func (p *OllamaProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OllamaProvider) ChatStream added in v0.0.2 

func (p *OllamaProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OllamaProvider) CreateStructuredOutput 

func (p *OllamaProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OllamaProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *OllamaProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn. Ollama models don't always honour tool calling, so the schema is prompted for directly; images ride along as chat parts for vision-capable models.

func (*OllamaProvider) CreateStructuredOutputFromSchema 

func (p *OllamaProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OllamaProvider) Model 

func (p *OllamaProvider) Model() string

func (*OllamaProvider) Name 

func (p *OllamaProvider) Name() string

func (*OllamaProvider) SetMeter 

func (p *OllamaProvider) SetMeter(hook MeterHook)

func (*OllamaProvider) SetModeration 

func (p *OllamaProvider) SetModeration(m ModerationProvider)

func (*OllamaProvider) WithMeter 

func (p *OllamaProvider) WithMeter(hook MeterHook) *OllamaProvider

func (*OllamaProvider) WithModeration 

func (p *OllamaProvider) WithModeration(m ModerationProvider) *OllamaProvider

type OpenAIImageProvider 

type OpenAIImageProvider struct {
	// contains filtered or unexported fields
}

OpenAIImageProvider implements ImageProvider using the OpenAI images API.

func (*OpenAIImageProvider) Edit 

func (p *OpenAIImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) EditWithReference 

func (p *OpenAIImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) Generate 

func (p *OpenAIImageProvider) Generate(ctx context.Context, prompt, model, size string) (string, error)

func (*OpenAIImageProvider) SetMeter 

func (p *OpenAIImageProvider) SetMeter(hook MeterHook)

func (*OpenAIImageProvider) SetModeration 

func (p *OpenAIImageProvider) SetModeration(m ModerationProvider)

func (*OpenAIImageProvider) WithModeration 

func (p *OpenAIImageProvider) WithModeration(m ModerationProvider) *OpenAIImageProvider

type OpenAIModerationProvider 

type OpenAIModerationProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIModerationProvider 

func NewOpenAIModerationProvider(apiKey string) *OpenAIModerationProvider

func (*OpenAIModerationProvider) Check 

func (p *OpenAIModerationProvider) Check(ctx context.Context, input string) (*ModerationResult, error)

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIProvider 

func NewOpenAIProvider(apiKey, model string) *OpenAIProvider

func NewOpenAIProviderWithBaseURL added in v0.5.3 

func NewOpenAIProviderWithBaseURL(apiKey, model, baseURL string) *OpenAIProvider

NewOpenAIProviderWithBaseURL targets an OpenAI-compatible endpoint (proxy, gateway, test server) instead of api.openai.com.

func (*OpenAIProvider) Chat 

func (p *OpenAIProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OpenAIProvider) ChatStream added in v0.0.2 

func (p *OpenAIProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OpenAIProvider) CreateStructuredOutput 

func (p *OpenAIProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OpenAIProvider) CreateStructuredOutputBreakpointed 

func (p *OpenAIProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM. OpenAI auto-prefix-caches any stable prefix ≥1024 tokens, so the breakpointing surf is implemented by concatenating sysPrompt and stableMid into the single system message. No explicit markers needed — the byte-stable prefix is the cache key.

dynamicTail rides as the user message (uncached).

func (*OpenAIProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *OpenAIProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images). Honours WithMaxTokens (default 4096) and forces the structured_output tool so the model cannot answer in prose.

func (*OpenAIProvider) CreateStructuredOutputFromSchema 

func (p *OpenAIProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OpenAIProvider) EmbedDimensions 

func (p *OpenAIProvider) EmbedDimensions() int

func (*OpenAIProvider) EmbedText 

func (p *OpenAIProvider) EmbedText(ctx context.Context, texts []string) ([][]float32, error)

func (*OpenAIProvider) MaxInputTokens 

func (p *OpenAIProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*OpenAIProvider) Model 

func (p *OpenAIProvider) Model() string

func (*OpenAIProvider) Name 

func (p *OpenAIProvider) Name() string

func (*OpenAIProvider) RawClient 

func (p *OpenAIProvider) RawClient() *openai.Client

func (*OpenAIProvider) SetMeter 

func (p *OpenAIProvider) SetMeter(hook MeterHook)

func (*OpenAIProvider) SetModeration 

func (p *OpenAIProvider) SetModeration(m ModerationProvider)

func (*OpenAIProvider) WithMeter 

func (p *OpenAIProvider) WithMeter(hook MeterHook) *OpenAIProvider

func (*OpenAIProvider) WithModeration 

func (p *OpenAIProvider) WithModeration(m ModerationProvider) *OpenAIProvider

type OpenAIRealtimeProvider added in v0.8.0 

type OpenAIRealtimeProvider struct {
	// contains filtered or unexported fields
}

OpenAIRealtimeProvider implements RealtimeProvider over OpenAI's Realtime WebSocket API (wss://api.openai.com/v1/realtime).

func NewOpenAIRealtimeProvider added in v0.8.0 

func NewOpenAIRealtimeProvider(apiKey, model string) *OpenAIRealtimeProvider

func (*OpenAIRealtimeProvider) AddMessage added in v0.8.0 

func (p *OpenAIRealtimeProvider) AddMessage(role, text string) error

func (*OpenAIRealtimeProvider) CancelResponse added in v0.8.0 

func (p *OpenAIRealtimeProvider) CancelResponse() error

func (*OpenAIRealtimeProvider) ClearAudio added in v0.8.0 

func (p *OpenAIRealtimeProvider) ClearAudio() error

func (*OpenAIRealtimeProvider) Close added in v0.8.0 

func (p *OpenAIRealtimeProvider) Close() error

func (*OpenAIRealtimeProvider) CommitAudio added in v0.8.0 

func (p *OpenAIRealtimeProvider) CommitAudio() error

func (*OpenAIRealtimeProvider) Connect added in v0.8.0 

func (p *OpenAIRealtimeProvider) Connect(ctx context.Context, cfg RealtimeSessionConfig) error

func (*OpenAIRealtimeProvider) CreateResponse added in v0.8.0 

func (p *OpenAIRealtimeProvider) CreateResponse() error

func (*OpenAIRealtimeProvider) Recv added in v0.8.0 

func (p *OpenAIRealtimeProvider) Recv() <-chan RealtimeEvent

func (*OpenAIRealtimeProvider) SendAudio added in v0.8.0 

func (p *OpenAIRealtimeProvider) SendAudio(data []byte) error

func (*OpenAIRealtimeProvider) SendToolResult added in v0.8.0 

func (p *OpenAIRealtimeProvider) SendToolResult(callID, output string) error

func (*OpenAIRealtimeProvider) SetMeter added in v0.8.0 

func (p *OpenAIRealtimeProvider) SetMeter(hook MeterHook)

type OpenAISTTProvider 

type OpenAISTTProvider struct {
	// contains filtered or unexported fields
}

func (*OpenAISTTProvider) Transcribe 

func (p *OpenAISTTProvider) Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)

type OpenAITTSProvider added in v0.7.0 

type OpenAITTSProvider struct {
	// contains filtered or unexported fields
}

func (*OpenAITTSProvider) SetMeter added in v0.7.0 

func (p *OpenAITTSProvider) SetMeter(hook MeterHook)

func (*OpenAITTSProvider) Synthesize added in v0.7.0 

func (p *OpenAITTSProvider) Synthesize(ctx context.Context, req TTSRequest) (io.ReadCloser, error)

type Part 

type Part interface {
	// contains filtered or unexported methods
}

Part is a single element of a multimodal message. Sealed — only TextPart and ImagePart satisfy this interface.

type PromptBlocks 

type PromptBlocks map[string]BlockSize

PromptBlocks is a per-call breakdown of the prompt into named blocks. Each entry carries chars + tokens. The meter hook stamps this onto UsageEvent.Metadata["blocks"] so downstream recorders can see the breakdown without needing each provider to crack the prompt apart.

func PromptBlocksFromCtx 

func PromptBlocksFromCtx(ctx context.Context) PromptBlocks

PromptBlocksFromCtx returns the prompt-block breakdown stamped on ctx, or nil if none. Providers call this inside emitUsage to enrich the UsageEvent.Metadata.

type ProviderOption 

type ProviderOption func(*providerConfig)

ProviderOption configures an LLMProvider created by NewLLMProvider.

func WithMeterOption 

func WithMeterOption(hook MeterHook) ProviderOption

WithMeterOption returns a ProviderOption that attaches a meter hook.

func WithModerationOption 

func WithModerationOption(m ModerationProvider) ProviderOption

WithModerationOption returns a ProviderOption that attaches a moderation provider.

type RealtimeEvent added in v0.8.0 

type RealtimeEvent struct {
	Type       string
	Audio      []byte
	Text       string
	ToolCall   *RealtimeToolCall
	Usage      *RealtimeUsage
	ResponseID string
	ItemID     string
	Error      error
}

RealtimeEvent is a decoded server event from a realtime session.

type RealtimeMeterable added in v0.8.0 

type RealtimeMeterable interface {
	SetMeter(MeterHook)
}

RealtimeMeterable is implemented by realtime providers that accept a meter hook.

type RealtimeProvider added in v0.8.0 

type RealtimeProvider interface {
	Connect(ctx context.Context, cfg RealtimeSessionConfig) error
	SendAudio(data []byte) error
	CommitAudio() error
	ClearAudio() error
	AddMessage(role, text string) error
	SendToolResult(callID, output string) error
	CreateResponse() error
	CancelResponse() error
	Recv() <-chan RealtimeEvent
	Close() error
}

RealtimeProvider is a bidirectional voice+text conversational session.

func NewRealtimeProvider added in v0.8.0 

func NewRealtimeProvider(providerName, apiKey, model string) (RealtimeProvider, error)

NewRealtimeProvider creates a RealtimeProvider from a provider name + API key + model.

type RealtimeSessionConfig added in v0.8.0 

type RealtimeSessionConfig struct {
	Model        string
	Voice        string
	Instructions string
	Tools        []Tool
	// InputAudioFormat / OutputAudioFormat name the PCM encoding. Accepts the
	// beta names (pcm16, g711_ulaw, g711_alaw) or the GA MIME names (pcm, pcmu,
	// pcma); both are mapped onto the GA wire format. Empty = provider default.
	InputAudioFormat        string
	OutputAudioFormat       string
	TurnDetection           *RealtimeTurnDetection
	InputAudioTranscription *RealtimeTranscriptionConfig
}

RealtimeSessionConfig configures a realtime session.

type RealtimeToolCall added in v0.8.0 

type RealtimeToolCall struct {
	CallID    string          `json:"call_id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

RealtimeToolCall is a completed function call from the model.

type RealtimeTranscriptionConfig added in v0.8.0 

type RealtimeTranscriptionConfig struct {
	Model string // e.g. "gpt-4o-transcribe"
}

RealtimeTranscriptionConfig configures input audio transcription.

type RealtimeTurnDetection added in v0.8.0 

type RealtimeTurnDetection struct {
	Type              string  // "server_vad" or empty (= server_vad)
	Threshold         float64 // 0.0–1.0
	PrefixPaddingMs   int
	SilenceDurationMs int
}

RealtimeTurnDetection configures server-side VAD.

type RealtimeUsage added in v0.8.0 

type RealtimeUsage struct {
	InputTokens       int `json:"input_tokens"`
	OutputTokens      int `json:"output_tokens"`
	InputAudioTokens  int `json:"input_audio_tokens"`
	OutputAudioTokens int `json:"output_audio_tokens"`
	TotalTokens       int `json:"total_tokens"`
}

RealtimeUsage carries token counts from a completed response.

type Response 

type Response struct {
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Response is the provider-agnostic result of a Chat call.

type RouteResult added in v0.4.0 

type RouteResult struct {
	Provider string
	Scores   map[string]float64 // dimension name → normalized [0,1]
	Total    float64
}

RouteResult holds the routing decision and per-dimension scores.

func Route added in v0.4.0 

func Route(ctx context.Context, candidates []string, dims ...Dimension) (*RouteResult, error)

Route picks the best provider from candidates using weighted dimensions. Each candidate is filtered by CanHandle, scored per dimension, normalized to [0,1] within each dimension, then ranked by weighted sum (lowest wins).

func RouteAll added in v0.4.0 

func RouteAll(ctx context.Context, candidates []string, dims ...Dimension) ([]RouteResult, error)

RouteAll returns all viable candidates sorted by score (best first).

type STTProvider 

type STTProvider interface {
	Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)
}

STTProvider transcribes audio into text.

func NewSTTProvider 

func NewSTTProvider(providerName, apiKey, model string) STTProvider

NewSTTProvider creates an STTProvider from a provider name + API key + model. Only OpenAI is supported for now — returns nil for other providers.

type StreamChunk added in v0.0.2 

type StreamChunk struct {
	Text     string
	ToolName string
	ToolArg  string
}

StreamChunk is a single piece of a streaming response. Text carries free-form content deltas. ToolName + ToolArg carry tool-call argument fragments so callers can stream tool output (e.g. the REPLY tool's text field) before the call finalizes.

type StreamResult added in v0.0.2 

type StreamResult struct {
	// contains filtered or unexported fields
}

StreamResult holds the final accumulated response from a stream.

func Stream added in v0.0.2 

func Stream(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (iter.Seq2[StreamChunk, error], *StreamResult)

Stream returns an iter.Seq2 that yields chunks as they arrive. The returned StreamResult provides the accumulated *Response after iteration. If the provider does not implement StreamingProvider, falls back to Chat.

func (*StreamResult) Response added in v0.0.2 

func (r *StreamResult) Response() (*Response, error)

Response blocks until streaming completes and returns the accumulated response.

type StreamingProvider added in v0.0.2 

type StreamingProvider interface {
	ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)
}

StreamingProvider extends LLMProvider with streaming chat support.

type TTSMeterable added in v0.7.0 

type TTSMeterable interface {
	SetMeter(MeterHook)
}

TTSMeterable is implemented by TTS providers that accept a meter hook.

type TTSProvider added in v0.7.0 

type TTSProvider interface {
	Synthesize(ctx context.Context, req TTSRequest) (io.ReadCloser, error)
}

TTSProvider turns text into speech audio. Synthesize returns the encoded audio as a stream the caller must close; bytes arrive progressively on providers that stream, so a reader can start playback before the call finishes.

func NewTTSProvider added in v0.7.0 

func NewTTSProvider(providerName, apiKey, model string) TTSProvider

NewTTSProvider creates a TTSProvider from a provider name + API key + model. Only OpenAI is supported for now — returns nil for other providers.

type TTSRequest added in v0.7.0 

type TTSRequest struct {
	Text string
	// Voice is a provider voice id; empty picks the provider default.
	Voice string
	// Format is the audio container: mp3 (default), opus, aac, flac, wav, pcm.
	Format string
	// Speed scales playback, 0.25–4.0; 0 means 1.0.
	Speed float64
	// Instructions steer delivery (tone, accent, pacing) on models that
	// accept them; ignored elsewhere.
	Instructions string
}

TTSRequest is one text-to-speech synthesis call.

type TextPart 

type TextPart struct {
	Text string
}

TextPart carries inline text.

type Tool 

type Tool struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema object
}

Tool defines a provider-agnostic tool the model can call.

type ToolCall 

type ToolCall struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

ToolCall represents the model requesting a tool invocation.

type UsageEvent 

type UsageEvent struct {
	CallerID         uuid.UUID `json:"caller_id"`
	Provider         string    `json:"provider"`
	Model            string    `json:"model"`
	Operation        string    `json:"operation"`
	InputTokens      int       `json:"input_tokens"`
	OutputTokens     int       `json:"output_tokens"`
	TotalTokens      int       `json:"total_tokens"`
	EstimatedCostUSD float64   `json:"estimated_cost_usd"`
	// CacheCreationInputTokens counts tokens written to the provider's
	// prompt cache on this call (Anthropic only, populated when cache
	// writes occur; priced at ~1.25x base). Zero on cache hit or when
	// caching isn't active.
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`
	// CacheReadInputTokens counts tokens served from the provider's
	// prompt cache (Anthropic only; priced at ~0.1x base). Nonzero means
	// WithCacheSysPrompt is actively paying off.
	CacheReadInputTokens int `json:"cache_read_input_tokens,omitempty"`
	// SystemPrompt/UserPrompt are captured for dev debug display only — empty
	// in production (see capturePromptForDebug).
	SystemPrompt string `json:"system_prompt,omitempty"`
	UserPrompt   string `json:"user_prompt,omitempty"`
	// DebugSpanID is the ID of the PromptMiddlewareChain span that was active
	// when this LLM call fired — used by the debug panel meter hook to attach
	// the llm_call event to the correct span in the tree. Empty in prod.
	DebugSpanID string         `json:"debug_span_id,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
}

UsageEvent represents a single LLM/image-gen usage event for metering.

type VeoVideoProvider added in v0.4.0 

type VeoVideoProvider struct {
	// contains filtered or unexported fields
}

func (*VeoVideoProvider) Generate added in v0.4.0 

func (p *VeoVideoProvider) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

func (*VeoVideoProvider) Model added in v0.4.0 

func (p *VeoVideoProvider) Model() string

func (*VeoVideoProvider) Name added in v0.4.0 

func (p *VeoVideoProvider) Name() string

func (*VeoVideoProvider) SetMeter added in v0.4.0 

func (p *VeoVideoProvider) SetMeter(hook MeterHook)

func (*VeoVideoProvider) SetModeration added in v0.4.0 

func (p *VeoVideoProvider) SetModeration(m ModerationProvider)

type VideoMeterable added in v0.2.0 

type VideoMeterable interface {
	SetMeter(MeterHook)
}

VideoMeterable is implemented by video providers that accept a meter hook.

type VideoProvider added in v0.2.0 

type VideoProvider interface {
	Name() string
	Model() string
	Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)
}

VideoProvider generates short video clips from a prompt and an optional conditioning image.

func NewVideoProvider added in v0.2.0 

func NewVideoProvider(providerName, apiKey, model string) (VideoProvider, error)

NewVideoProvider creates a VideoProvider from a provider name ("fal", "gemini", "minimax", "veo").

type VideoRequest added in v0.2.0 

type VideoRequest struct {
	Prompt         string
	NegativePrompt string
	// ImageURL is forwarded verbatim (https:// or data: URI).
	ImageURL string
	// Image is raw image bytes; encoded as a data: URI when ImageURL is empty.
	Image []byte
	// ImageMediaType is the MIME type of Image ("image/jpeg" when empty).
	ImageMediaType string
	// Duration in seconds. Providers snap to their supported set.
	Duration float64
	// Resolution label, e.g. "1080p" | "1440p" | "2160p".
	Resolution string
	// AspectRatio label, e.g. "auto" | "16:9" | "9:16".
	AspectRatio string
	// FPS of the generated clip; 0 = provider default.
	FPS int
	// Audio asks the model to generate a soundtrack when it can.
	Audio bool
	// Seed pins generation; nil = random per call.
	Seed *int64
	// Model overrides the provider's default model/endpoint for this call.
	Model string
}

VideoRequest describes a single clip generation.

Image / ImageURL are optional first-frame conditioning (image-to-video). Leave both empty for text-to-video. Zero-valued knobs mean "provider default" — providers only forward fields the caller set.

type VideoResult added in v0.2.0 

type VideoResult struct {
	URL         string
	ContentType string
	FileName    string
	FileSize    int64
	Width       int
	Height      int
	FPS         float64
	Duration    float64 // seconds
	NumFrames   int
	Model       string
	Seed        int64
	CostUSD     float64
	Elapsed     time.Duration
	// Data holds the clip bytes when URL is not publicly fetchable (Gemini and
	// Veo serve files behind the API key). Empty for providers with public URLs.
	Data []byte
}

VideoResult is the provider-agnostic outcome of a Generate call.

URL points at a provider-hosted file that is typically temporary — callers that need durability must download it promptly.

type VideoRouter added in v0.4.0 

type VideoRouter struct {
	// contains filtered or unexported fields
}

VideoRouter routes Generate calls across multiple VideoProviders using weighted Dimensions (price, availability, capability). It implements VideoProvider so callers can use it as a drop-in replacement.

func NewVideoRouter added in v0.4.0 

func NewVideoRouter(opts ...VideoRouterOption) (*VideoRouter, error)

NewVideoRouter creates a router that picks the best provider per request. Without explicit dimension options it defaults to price-only routing.

func (*VideoRouter) Generate added in v0.4.0 

func (r *VideoRouter) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

Generate picks the best provider for req and delegates to it. If the chosen provider fails, it falls back to the next best. Generate tries providers in order and returns the first success.

A request that names a model goes to that model's provider first (the caller chose it for a reason — price, native audio, …); only if that fails (balance exhausted, outage, moderation) do the others get a turn, each on its own default model since model ids are provider-specific. A request without a model follows the price/availability ranking as before.

func (*VideoRouter) Model added in v0.4.0 

func (r *VideoRouter) Model() string

func (*VideoRouter) Name added in v0.4.0 

func (r *VideoRouter) Name() string

func (*VideoRouter) SetMeter added in v0.4.0 

func (r *VideoRouter) SetMeter(hook MeterHook)

SetMeter forwards the meter hook to all underlying providers.

type VideoRouterOption added in v0.4.0 

type VideoRouterOption func(*VideoRouter)

VideoRouterOption configures a VideoRouter.

func RouteByAvailability added in v0.4.0 

func RouteByAvailability(weight float64) VideoRouterOption

RouteByAvailability enables availability-weighted scoring based on historical latency per provider.

func RouteByPrice added in v0.4.0 

func RouteByPrice(weight float64) VideoRouterOption

RouteByPrice enables price-weighted scoring.

func WithRoute added in v0.4.0 

func WithRoute(p VideoProvider) VideoRouterOption

WithRoute adds a VideoProvider to the router.

Source Files

View all Source files

Directories

Path Synopsis
cmd
testmatrix command
testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.
 testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.