ai

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 2, 2026 License: MIT Imports: 30 Imported by: 0

README


ai

CI Go Reference

Multimodal Go module for building across AI providers, with realistic cost tracking baked in, not just token counts.

From the obvious to the overlooked: chat, reasoning, tool use, structured output, image gen & editing, video gen, speech-to-text, embeddings, moderation.

Install

go get github.com/nurf-ai/ai

Usage

Chat

import "github.com/nurf-ai/ai"

llm := ai.NewLLMProvider(provider, apiKey, model)

resp, err := llm.Chat(ctx, []ai.Message{
    {Role: "user", Content: "Hello"},
}, nil)

Streaming

Falls back to Chat transparently.

Iterator (Go 1.23+):

llm := ai.NewLLMProvider(provider, apiKey, model)

chunks, result := ai.Stream(ctx, llm, messages, nil)

for chunk, err := range chunks {
    fmt.Print(chunk.Text)
}
resp, err := result.Response()

Channel (for select/fan-out):

ch, wait := ai.StreamWithChan(ctx, llm, messages, nil)

for chunk := range ch {
    fmt.Print(chunk.Text)
}
resp, err := wait()

Vision + structured output

Hand a multimodal turn (text + base64 images) to any provider and get schema-constrained JSON back. Providers that cannot see images return ai.ErrVisionUnsupported instead of silently dropping them; ai.WithMaxTokens raises the output budget (default 4096).

parts := []ai.Part{
    ai.ImagePart{MediaType: "image/jpeg", Data: b64},
    ai.TextPart{Text: "Describe every part of this object."},
}
ctx = ai.WithMaxTokens(ctx, 16000)
out, err := ai.StructuredOutputFromParts(ctx, llm, parts, sysPrompt, schemaJSON) // map[string]any

Embeddings

embedder := ai.NewEmbedder(provider, apiKey)
vectors, err := embedder.EmbedText(ctx, []string{"hello world"})

Image Generation

img, err := ai.NewImageProvider(ctx, provider, apiKey, model)
b64, err := img.Generate(ctx, "a cat in space", "", "")

Video Generation

video, err := ai.NewVideoProvider("fal", apiKey, "") // default: fal-ai/ltx-2.3/image-to-video/fast
res, err := video.Generate(ctx, ai.VideoRequest{
    Prompt:   "a robot adopts a stray cat",
    Image:    lastFrameJPEG, // optional first-frame conditioning
    Duration: 6, Resolution: "1080p", AspectRatio: "16:9",
})
fmt.Println(res.URL, res.Duration, res.CostUSD) // provider URL is temporary — download it

Speech-to-Text

stt := ai.NewSTTProvider(provider, apiKey, model)
text, err := stt.Transcribe(ctx, audioReader, "audio.mp3")

Metering & Pricing

ai.SetLLMMeter(llm, func(ev ai.UsageEvent) {
    log.Printf("%s: %d tokens, $%.6f", ev.Model, ev.TotalTokens, ev.EstimatedCostUSD)
})

Attribute calls via context — ai.WithMeterCallerID, ai.WithMeterOperation, ai.WithMeterMetadata(ctx, map[string]any{...}) — every provider merges stamped metadata into UsageEvent.Metadata (provider-set keys win).

Built-in per-model cost estimation via EstimateCostFull (tokens / flat per image) and EstimateVideoCost (per second of video, optionally per resolution). Rates and context windows for all supported models are maintained in models.json — the single source of truth, embedded at compile time.

Providers

Factory Providers Features
NewLLMProvider(provider, apiKey, model) anthropic, openai, gemini, ollama, huggingface Chat, streaming, structured output, tools
NewImageProvider(ctx, provider, apiKey, model) openai, gemini Image generation / editing
NewSTTProvider(provider, apiKey, model) openai Speech-to-text
NewVideoProvider(provider, apiKey, model) fal Video generation (text/image-to-video)
NewEmbedder(provider, apiKey) openai Text embeddings

Each ✓ means the integration test passes:

Provider Model Chat Stream Reasoning Structured Output From Schema Tools Embeddings STT Moderation Image Gen Img Edit Img Edit Ref Txt2Vid Img2Vid Caching
Anthropic claude-haiku-4-5 ✓ ✓ ✓ ✓ ✓ ✓ ✓
OpenAI gpt-4o-mini ✓ ✓ ✓ ✓ ✓
OpenAI gpt-5-mini ✓
OpenAI text-embedding-3-small ✓
OpenAI whisper-1 ✓
OpenAI omni-moderation-latest ✓
OpenAI gpt-image-1 ✓ ✓
Gemini gemini-3.6-flash ✓ ✓ ✓ ✓ ✓
Gemini gemini-2.5-flash-image ✓ ✓ ✓
Hugging Face Kimi-K2-Instruct ✓ ✓ ✓ ✓ ✓
Hugging Face Kimi-K3 ✓ ✓ ✓ ✓ ✓
Ollama qwen3.5:0.8b ✓ ✓
Ollama gpt-oss:20b ✓ ✓ ✓
Ollama gemma4:e4b ✓ ✓
fal ltx-2.3/t2v/fast ?
fal ltx-2.3/i2v/fast ?
fal minimax/h3-max/i2v ?

Contributing

Setup

git clone https://github.com/nurf-ai/ai.git
cd ai
go test ./...

Adding a provider

  1. Create <provider>.go implementing LLMProvider
  2. Add model entries to models.json (pricing + max_input_tokens)
  3. Register in provider.go (NewLLMProvider switch)
  4. Add a smoke test to integration_test.go
  5. Run both test suites before submitting

Adding a model

  1. Add entry to models.json under the provider key — include max_input_tokens
  2. go test ./... — TestPricingCoverage will fail if max_input_tokens is missing

Updating models

Edit models.json directly — it's embedded at build time via go:embed. No Go code changes needed.

Testing

Unit tests (no API keys needed, runs in CI):

go test ./...

Integration tests (real API calls, run locally):

cp .env.test.tpl .env.test   # fill in your API keys
go test -tags=integration -count=1 -json ./... | go run ./cmd/testmatrix

.env.test is loaded automatically via godotenv in TestMain. Only set the keys you have — providers with missing keys are skipped (∅).

testmatrix prints live progress with per-test token counts and cost, failure details, and per-provider cost totals to stderr:

  ✓ Anthropic/claude-haiku-4-5/Chat (1.8s)                     800 tok_in, 80 tok_out, $0.0003
  ✓ OpenAI/gpt-4o-mini/Chat (1.2s)                             500 tok_in, 50 tok_out, $0.0001
  ✓ OpenAI/gpt-4o-mini/Moderation (0.3s)
  ∅ Ollama

4 passed, 0 failed, 1 skipped

── cost ──
  Anthropic      $0.0003  (800 tok_in, 80 tok_out, 200 tok_cached)
  OpenAI         $0.0001  (500 tok_in, 50 tok_out)
  TOTAL          $0.0004  (1300 tok_in, 130 tok_out, 200 tok_cached)

[!NOTE] The capability/coverage matrix goes to stdout — paste it into the README between the <!-- testmatrix:start/end --> markers. PRs that add or change provider capabilities must include an updated matrix.

All provider tests use t.Parallel(), so subtests run concurrently within a single go test invocation. To split across CI jobs, use -run:

go test -tags=integration -v -run TestAnthropic ./...
go test -tags=integration -v -run TestOpenAI ./...
go test -tags=integration -v -run TestGemini ./...
go test -tags=integration -v -run TestHuggingFace ./...
go test -tags=integration -v -run TestOllama ./...

See .env.test.tpl for the full list of env vars.

Guidelines

  • CI runs go vet, golangci-lint, and go test -race — make sure they pass locally before submitting:
go vet ./...
gofmt -l .
golangci-lint run
go test -race ./...

  • Releases are automated via Release Please — use conventional commits and a release PR is created automatically on push to main. The prefix determines the version bump:

    Prefix Bump When to use
    feat: minor New capability in the library
    fix: patch Bug fix in library code
    docs:, chore:, test:, ci: none No release — README, tooling, tests, CI

Documentation

Overview

Package ai is the consumer-facing LLM contract — provider-agnostic types (Tool/Message/Response/ToolCall/Part) plus the LLMProvider interface in ai_contract.go. Concrete implementations (Anthropic/OpenAI/Ollama/ HuggingFace) live in sibling files; they satisfy LLMProvider structurally.

Index

Constants

View Source 
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants used in Message.Role.

View Source 
const DefaultFalQueueBase = "https://queue.fal.run"

DefaultFalQueueBase is the fal queue API root. Every fal model endpoint is addressed as {base}/{model_id}; the queue returns absolute status/result URLs which the client follows verbatim.

View Source 
const DefaultFalVideoModel = "fal-ai/ltx-2.3/image-to-video/fast"

DefaultFalVideoModel is the fal endpoint used when none is configured: LTX-2.3 fast, image-to-video (accepts a first-frame image_url).

View Source 
const DefaultHuggingFaceBaseURL = "https://router.huggingface.co/v1"

DefaultHuggingFaceBaseURL is the HF Inference router's OpenAI-compatible endpoint. Override via HF_BASE_URL when targeting a dedicated endpoint or a specific upstream provider route (e.g. https://router.huggingface.co/<provider>/v1).

Variables

View Source 
var ErrVisionUnsupported = errors.New("ai: provider does not accept image input for structured output")

ErrVisionUnsupported is returned when an ImagePart is handed to a provider that only accepts text for structured output. The image is never silently dropped — a spec authored without the reference it was asked to describe is worse than an error.

Functions

func CacheSysPromptFromCtx 

func CacheSysPromptFromCtx(ctx context.Context) bool

CacheSysPromptFromCtx reports whether WithCacheSysPrompt was stamped. Adapters call this while building the request to decide whether to mark the sys block as cacheable.

func CountTokens 

func CountTokens(s string) int

CountTokens returns an approximate token count for s using the o200k_base BPE encoding. Exact for GPT-4o / GPT-5 family; within ~15% for Anthropic and HF model families. Empty string returns 0.

Falls back to a len(s)/4 heuristic if the tiktoken encoder fails to initialize, so this never returns an error — a usable rough number is more helpful at call sites than a plumbed error path.

func CountTokensOver 

func CountTokensOver(s string, limit int64) bool

CountTokensOver reports whether counting tokens in s exceeds limit. Short-circuits using the cheap char-based upper bound before invoking the BPE encoder, so callers can gate long strings without paying the full tokenization cost when they're obviously under the limit.

func DataURI added in v0.2.0 

func DataURI(mediaType string, data []byte) string

DataURI encodes raw bytes as a base64 data: URI.

func DebugPromptsEnabled 

func DebugPromptsEnabled() bool

DebugPromptsEnabled reports whether prompt capture is active. Providers check this before building sys/user prompt strings for the debug panel.

func DebugSpanIDFromCtx 

func DebugSpanIDFromCtx(ctx context.Context) string

DebugSpanIDFromCtx returns the span ID on the context, or empty.

func EstimateCostFull 

func EstimateCostFull(model string, in, out, cw, cr int) float64

func EstimateVideoCost added in v0.2.0 

func EstimateVideoCost(model string, seconds float64, resolution string) float64

EstimateVideoCost prices seconds of generated video for a per-second model. resolution selects a per-resolution rate when the table has one; otherwise the base rate applies. Unknown models are recorded as 0 (and reported via the unknown-model hook), like EstimateCostFull.

func GenerateTyped 

func GenerateTyped[T any](ctx context.Context, p LLMProvider, userPrompt, sysPrompt string) (*T, error)

GenerateTyped wraps CreateStructuredOutput with Go generics so callers get back a typed pointer without pre-allocating the output struct.

func IsFalError added in v0.2.0 

func IsFalError(err error) bool

IsFalError reports whether err wraps a *FalError.

func IsVideoModel added in v0.2.0 

func IsVideoModel(model string) bool

IsVideoModel reports whether model is priced per second of video.

func MaxInputTokensLLM 

func MaxInputTokensLLM(company, modelName string) (int64, error)

MaxInputTokensLLM returns the advertised maximum input context window in tokens for the given model.

company is the model-family owner / API namespace — one of "anthropic", "openai", or "huggingface". modelName is the exact model id as passed to the provider constructor (for HF that's the full "<org>/<model>" id). Trailing date suffixes like "-20250514" are stripped automatically.

Returns an error if the model is unknown. Callers that need a safe default should handle the error explicitly rather than rely on a fallback — silently returning 0 or a guess would mask config typos.

func MaxTokensFromCtx 

func MaxTokensFromCtx(ctx context.Context, fallback int) int

func MeterCallerIDFromCtx added in v0.0.2 

func MeterCallerIDFromCtx(ctx context.Context) uuid.UUID

func MeterMetadataFromCtx added in v0.3.0 

func MeterMetadataFromCtx(ctx context.Context) map[string]any

MeterMetadataFromCtx returns a copy of the metadata stamped on ctx via WithMeterMetadata, or nil when none. Mutating the result never touches the context.

func MeterOperationFromCtx 

func MeterOperationFromCtx(ctx context.Context) string

func ModelsForProvider 

func ModelsForProvider(provider string) []string

func PartsText added in v0.2.0 

func PartsText(parts []Part) (string, bool)

PartsText joins the TextParts of a multimodal turn with blank lines and reports whether any ImagePart was present. Used for moderation, usage attribution and text-only fallbacks.

func PricingTable 

func PricingTable() map[string]modelPricing

func ReasoningEffortFromCtx 

func ReasoningEffortFromCtx(ctx context.Context) string

func SetDebugPromptsEnabled 

func SetDebugPromptsEnabled(enabled bool)

SetDebugPromptsEnabled toggles prompt capture in UsageEvents. Call once at startup based on your dev-mode flag.

func SetImageMeter 

func SetImageMeter(img ImageProvider, hook MeterHook)

SetImageMeter attaches a meter hook to any ImageProvider that satisfies ImageMeterable.

func SetImageModeration 

func SetImageModeration(img ImageProvider, m ModerationProvider)

SetImageModeration attaches a moderation provider to any ImageProvider that satisfies the moderable interface.

func SetLLMMeter 

func SetLLMMeter(llm LLMProvider, hook MeterHook)

SetLLMMeter attaches a meter hook to any LLMProvider that satisfies LLMMeterable.

func SetLLMModeration 

func SetLLMModeration(llm LLMProvider, m ModerationProvider)

SetLLMModeration attaches a moderation provider to any LLMProvider that satisfies LLMModerable.

func SetLogger 

func SetLogger(l *zap.Logger)

func SetOnModerationError 

func SetOnModerationError(fn func(ctx context.Context, err error) error)

SetOnModerationError overrides the default moderation-error behavior. Default (nil): warn and allow. When set, the function's return value determines whether the call proceeds (nil) or is rejected (non-nil error).

func SetUnknownModelHook 

func SetUnknownModelHook(hook func(model string))

func SetVideoMeter added in v0.2.0 

func SetVideoMeter(v VideoProvider, hook MeterHook)

SetVideoMeter attaches a meter hook to any VideoProvider that satisfies VideoMeterable.

func SetVideoModeration added in v0.2.0 

func SetVideoModeration(v VideoProvider, m ModerationProvider)

SetVideoModeration attaches a moderation provider to any VideoProvider that satisfies the moderable interface.

func StreamWithChan added in v0.0.2 

func StreamWithChan(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (<-chan StreamChunk, func() (*Response, error))

StreamWithChan starts streaming and returns a channel of chunks plus a function that blocks until done and returns the accumulated response. The channel is closed when streaming completes. If the provider does not implement StreamingProvider, falls back to Chat.

func StructuredOutputFromParts added in v0.2.0 

func StructuredOutputFromParts(ctx context.Context, p LLMProvider, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

StructuredOutputFromParts routes a multimodal structured-output request to p. Providers implementing MultimodalStructuredProvider receive the parts verbatim; any other provider receives the text parts joined, and ErrVisionUnsupported when an ImagePart would otherwise be lost.

func TransparentBGFromCtx 

func TransparentBGFromCtx(ctx context.Context) bool

func TruncatePromptForDebug 

func TruncatePromptForDebug(s string) string

TruncatePromptForDebug is kept for callers that still reference the old helper name; it now just delegates to capturePromptForDebug — prompts are no longer truncated. Dev mode only carries them at all.

func VideoModels added in v0.3.0 

func VideoModels() []string

VideoModels lists every priced video model id, sorted.

func WithCacheSysPrompt 

func WithCacheSysPrompt(ctx context.Context) context.Context

WithCacheSysPrompt signals that the sys prompt sent in this call is a stable prefix worth provider-side prompt caching. Anthropic's adapter sets cache_control: ephemeral on the sys block; OpenAI/Ollama/HF are no-op (OpenAI auto-caches prompts ≥1024 tokens; Ollama uses KV-cache for matching prefixes at the inference layer; HF depends on backend).

Call this at the call site right before the LLM invocation. Marker is per-call, not global — stamp only when the caller knows the sys prompt is stable across turns (e.g. planner/router system prompts).

func WithDebugSpanID 

func WithDebugSpanID(ctx context.Context, spanID string) context.Context

WithDebugSpanID stamps the current dev-debug span ID on the context so providers can include it on UsageEvents. Caller must be in dev mode; in prod this is a no-op path (the middleware chain never calls StartSpan).

func WithMaxTokens 

func WithMaxTokens(ctx context.Context, n int) context.Context

func WithMeterCallerID added in v0.0.2 

func WithMeterCallerID(ctx context.Context, id uuid.UUID) context.Context

func WithMeterMetadata added in v0.3.0 

func WithMeterMetadata(ctx context.Context, kv map[string]any) context.Context

WithMeterMetadata stamps arbitrary key/values on the context so every provider merges them into UsageEvent.Metadata (surf handle, session id, feature flag — whatever the caller wants attributed). Stacking calls merges kv over what is already stamped, later keys win, into a fresh map: the stored map is copied, never mutated, and kv is copied too so the caller may reuse it. Empty/nil kv returns ctx unchanged.

func WithMeterOperation 

func WithMeterOperation(ctx context.Context, op string) context.Context

func WithPromptBlocks 

func WithPromptBlocks(ctx context.Context, contents map[string]string) context.Context

WithPromptBlocks stamps a per-block breakdown on the context so providers can include it on UsageEvents. Call-site passes the raw block strings keyed by block name — this helper computes both Chars (len(s)) and Tokens (CountTokens(s)) per entry.

Tokenizer cost is paid here, but `dev-only` callers and a single tokenize per call mean it's negligible vs the LLM round-trip. If you need to skip the tokenize (e.g. hot path that doesn't care about tokens), call WithPromptBlocksRaw with a pre-built map[string]BlockSize directly.

Empty maps are intentionally not stored — PromptBlocksFromCtx returning nil is the canonical "absent" signal.

func WithPromptBlocksRaw 

func WithPromptBlocksRaw(ctx context.Context, b PromptBlocks) context.Context

WithPromptBlocksRaw is the lower-level variant — caller pre-computes Chars and Tokens per block. Useful when the same block string is reused across calls and you want to cache the token count.

func WithReasoningEffort 

func WithReasoningEffort(ctx context.Context, effort string) context.Context

WithReasoningEffort sets the reasoning effort level ("low", "medium", "high") for providers that support it. OpenAI: maps to reasoning_effort in chat completions. Anthropic: maps to extended thinking with a budget derived from effort level.

func WithTransparentBG 

func WithTransparentBG(ctx context.Context) context.Context

WithTransparentBG signals image providers to produce transparent backgrounds.

Types

type AnthropicProvider 

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

func NewAnthropicProvider 

func NewAnthropicProvider(apiKey, model string) *AnthropicProvider

func (*AnthropicProvider) Chat 

func (p *AnthropicProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*AnthropicProvider) ChatStream added in v0.0.2 

func (p *AnthropicProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*AnthropicProvider) CreateStructuredOutput 

func (p *AnthropicProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*AnthropicProvider) CreateStructuredOutputBreakpointed 

func (p *AnthropicProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM by emitting two system blocks each marked with cache_control: ephemeral. The provider hashes the prefix up through each cache_control marker, so when sysPrompt + stableMid is byte-stable across turns we hit the bp2 entry (everything stable cached); when only sysPrompt is stable (e.g. the candidate list changed) we hit bp1 (sysPrompt cached, stableMid reprocessed). One marker per change-rate tier — see .wiki/context/cache-breakpoints.md.

dynamicTail is the per-call query and rides as the user message — never cached.

func (*AnthropicProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *AnthropicProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images). Honours WithMaxTokens (default 4096): a vision-authored spec routinely needs more.

func (*AnthropicProvider) CreateStructuredOutputFromSchema 

func (p *AnthropicProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*AnthropicProvider) MaxInputTokens 

func (p *AnthropicProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*AnthropicProvider) Model 

func (p *AnthropicProvider) Model() string

func (*AnthropicProvider) Name 

func (p *AnthropicProvider) Name() string

func (*AnthropicProvider) SetMeter 

func (p *AnthropicProvider) SetMeter(hook MeterHook)

func (*AnthropicProvider) SetModeration 

func (p *AnthropicProvider) SetModeration(m ModerationProvider)

func (*AnthropicProvider) WithMeter 

func (p *AnthropicProvider) WithMeter(hook MeterHook) *AnthropicProvider

func (*AnthropicProvider) WithModeration 

func (p *AnthropicProvider) WithModeration(m ModerationProvider) *AnthropicProvider

type BlockSize 

type BlockSize struct {
	Chars  int `json:"chars"`
	Tokens int `json:"tokens"`
}

BlockSize carries the two attribution signals per prompt block: chars (raw length) and tokens (tiktoken o200k_base count). Tokens are what bills you; chars exist for sanity (a high tokens/chars ratio per block flags tokenizer-hostile content like code, base64, or heavy unicode — surfd as the "density" warning in PromptBlocksBar).

type CacheControl 

type CacheControl struct {
	Type string `json:"type"` // "ephemeral"
}

CacheControl tells a provider to mark this message (or its last content block) as a cache breakpoint. Only Anthropic acts on it today — other providers ignore it silently.

type Embedder 

type Embedder interface {
	EmbedText(ctx context.Context, texts []string) ([][]float32, error)
	EmbedDimensions() int
}

Embedder produces vector embeddings for text. Consumers that need semantic similarity (discovery, memory recall, search ranking) depend on this interface. Not all LLM providers support embeddings — wire a separate provider if the main LLM doesn't (e.g. Anthropic + OpenAI embedding sidecar).

func EmbedderFromLLM 

func EmbedderFromLLM(llm LLMProvider) Embedder

EmbedderFromLLM extracts an Embedder from an LLMProvider if it satisfies the Embedder interface (e.g. OpenAIProvider).

func NewEmbedder 

func NewEmbedder(providerName, apiKey string) Embedder

NewEmbedder creates an Embedder from a provider name + API key. Only OpenAI is supported for now — returns nil for other providers.

type ErrorKind 

type ErrorKind int
const (
	ErrUnknown ErrorKind = iota
	ErrBilling
	ErrAuth
	ErrRateLimit
	ErrModeration
	ErrProviderDown
)

func ClassifyError 

func ClassifyError(err error) (ErrorKind, string)

func (ErrorKind) Retryable 

func (k ErrorKind) Retryable() bool

func (ErrorKind) UserMessage 

func (k ErrorKind) UserMessage() string

type FalClient added in v0.2.0 

type FalClient struct {
	// contains filtered or unexported fields
}

FalClient is a minimal client for fal.ai's HTTP queue API (submit → poll status → fetch result). It is model-agnostic: any fal endpoint that takes a JSON input and returns a JSON output can be driven through Run.

func NewFalClient added in v0.2.0 

func NewFalClient(apiKey string, opts ...FalOption) *FalClient

NewFalClient creates a fal queue client authenticated with apiKey.

func (*FalClient) Cancel added in v0.2.0 

func (c *FalClient) Cancel(ctx context.Context, req *FalRequest) error

Cancel asks the queue to drop a request that has not started yet.

func (*FalClient) Result added in v0.2.0 

func (c *FalClient) Result(ctx context.Context, req *FalRequest) (json.RawMessage, error)

Result fetches the model output. fal answers with a non-2xx status and a `detail` body when the run failed; that surfaces as *FalError.

func (*FalClient) Run added in v0.2.0 

func (c *FalClient) Run(ctx context.Context, endpoint string, input any) (json.RawMessage, error)

Run submits input, waits for completion and returns the raw output JSON.

func (*FalClient) Status added in v0.2.0 

func (c *FalClient) Status(ctx context.Context, req *FalRequest) (*FalStatus, error)

Status polls the request once.

func (*FalClient) Submit added in v0.2.0 

func (c *FalClient) Submit(ctx context.Context, endpoint string, input any) (*FalRequest, error)

Submit enqueues input on endpoint (e.g. "fal-ai/ltx-2.3/image-to-video/fast").

func (*FalClient) Wait added in v0.2.0 

func (c *FalClient) Wait(ctx context.Context, req *FalRequest) (*FalStatus, error)

Wait polls until the request completes or ctx is done.

type FalError added in v0.2.0 

type FalError struct {
	Status   int
	Endpoint string
	Message  string
}

FalError is a non-2xx response from any fal endpoint.

func (*FalError) Error added in v0.2.0 

func (e *FalError) Error() string

type FalOption added in v0.2.0 

type FalOption func(*FalClient)

FalOption configures a FalClient.

func WithFalHTTPClient added in v0.2.0 

func WithFalHTTPClient(h *http.Client) FalOption

WithFalHTTPClient overrides the HTTP client (timeouts, transport, tests).

func WithFalPollInterval added in v0.2.0 

func WithFalPollInterval(d time.Duration) FalOption

WithFalPollInterval sets the initial status poll interval. It backs off geometrically up to 5s.

func WithFalQueueBase added in v0.2.0 

func WithFalQueueBase(base string) FalOption

WithFalQueueBase overrides the queue API root (tests, proxies).

type FalRequest added in v0.2.0 

type FalRequest struct {
	Endpoint      string `json:"-"`
	RequestID     string `json:"request_id"`
	StatusURL     string `json:"status_url"`
	ResponseURL   string `json:"response_url"`
	CancelURL     string `json:"cancel_url"`
	QueuePosition int    `json:"queue_position"`
}

FalRequest is the queue ticket returned by Submit.

type FalStatus added in v0.2.0 

type FalStatus struct {
	Status        string `json:"status"`
	QueuePosition int    `json:"queue_position"`
	Metrics       struct {
		InferenceTime float64 `json:"inference_time"`
	} `json:"metrics"`
}

FalStatus is one status poll.

func (*FalStatus) Done added in v0.2.0 

func (s *FalStatus) Done() bool

Done reports whether the request has finished (success or failure — the result fetch tells which).

type FalVideoProvider added in v0.2.0 

type FalVideoProvider struct {
	// contains filtered or unexported fields
}

FalVideoProvider implements VideoProvider on top of fal.ai's queue API.

The model string is the full fal endpoint id (e.g. "fal-ai/ltx-2.3/image-to-video/fast"). Input keys follow the LTX family schema (prompt, image_url, duration, resolution, aspect_ratio, fps, generate_audio, seed, negative_prompt); zero-valued request fields are omitted so other fal video endpoints that share those names keep working.

func NewFalVideoProviderWithClient added in v0.2.0 

func NewFalVideoProviderWithClient(client *FalClient, model string) *FalVideoProvider

NewFalVideoProviderWithClient builds a provider around an existing client (tests, shared clients).

func (*FalVideoProvider) Client added in v0.2.0 

func (p *FalVideoProvider) Client() *FalClient

func (*FalVideoProvider) Generate added in v0.2.0 

func (p *FalVideoProvider) Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)

Generate runs one clip generation and blocks until fal returns the file.

func (*FalVideoProvider) Model added in v0.2.0 

func (p *FalVideoProvider) Model() string

func (*FalVideoProvider) Name added in v0.2.0 

func (p *FalVideoProvider) Name() string

func (*FalVideoProvider) SetMeter added in v0.2.0 

func (p *FalVideoProvider) SetMeter(hook MeterHook)

func (*FalVideoProvider) SetModeration added in v0.2.0 

func (p *FalVideoProvider) SetModeration(m ModerationProvider)

type GeminiImageProvider 

type GeminiImageProvider struct {
	// contains filtered or unexported fields
}

GeminiImageProvider implements ImageProvider using Google's Gemini API.

func (*GeminiImageProvider) Edit 

func (p *GeminiImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) EditWithReference 

func (p *GeminiImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) Generate 

func (p *GeminiImageProvider) Generate(ctx context.Context, prompt, model, _ string) (string, error)

func (*GeminiImageProvider) SetMeter 

func (p *GeminiImageProvider) SetMeter(hook MeterHook)

func (*GeminiImageProvider) SetModeration 

func (p *GeminiImageProvider) SetModeration(m ModerationProvider)

func (*GeminiImageProvider) WithModeration 

func (p *GeminiImageProvider) WithModeration(m ModerationProvider) *GeminiImageProvider

type GeminiProvider added in v0.0.2 

type GeminiProvider struct {
	// contains filtered or unexported fields
}

func NewGeminiProvider added in v0.0.2 

func NewGeminiProvider(ctx context.Context, apiKey, model string) (*GeminiProvider, error)

func (*GeminiProvider) Chat added in v0.0.2 

func (p *GeminiProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*GeminiProvider) ChatStream added in v0.0.2 

func (p *GeminiProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*GeminiProvider) CreateStructuredOutput added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*GeminiProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *GeminiProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images).

func (*GeminiProvider) CreateStructuredOutputFromSchema added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*GeminiProvider) MaxInputTokens added in v0.0.2 

func (p *GeminiProvider) MaxInputTokens() (int64, error)

func (*GeminiProvider) Model added in v0.0.2 

func (p *GeminiProvider) Model() string

func (*GeminiProvider) Name added in v0.0.2 

func (p *GeminiProvider) Name() string

func (*GeminiProvider) SetMeter added in v0.0.2 

func (p *GeminiProvider) SetMeter(hook MeterHook)

func (*GeminiProvider) SetModeration added in v0.0.2 

func (p *GeminiProvider) SetModeration(m ModerationProvider)

func (*GeminiProvider) WithMeter added in v0.0.2 

func (p *GeminiProvider) WithMeter(hook MeterHook) *GeminiProvider

func (*GeminiProvider) WithModeration added in v0.0.2 

func (p *GeminiProvider) WithModeration(m ModerationProvider) *GeminiProvider

type HTTPMeterEmitter 

type HTTPMeterEmitter struct {
	// contains filtered or unexported fields
}

HTTPMeterEmitter buffers usage events and POSTs them to a meter service.

func NewHTTPMeterEmitter 

func NewHTTPMeterEmitter(opts HTTPMeterOpts) *HTTPMeterEmitter

func (*HTTPMeterEmitter) Hook 

func (e *HTTPMeterEmitter) Hook() MeterHook

type HTTPMeterOpts 

type HTTPMeterOpts struct {
	Endpoint      string                             // full URL — no suffix appended
	AuthHeader    string                             // sent as Authorization header when non-empty
	BatchSize     int                                // default 32
	FlushInterval time.Duration                      // default 2s
	Marshal       func([]UsageEvent) ([]byte, error) // default json.Marshal
	ContentType   string                             // default "application/json"
	OnError       func(error)                        // default logger.Warn
}

HTTPMeterOpts configures an HTTPMeterEmitter.

type HuggingFaceProvider 

type HuggingFaceProvider struct {
	// contains filtered or unexported fields
}

HuggingFaceProvider speaks the HF Inference Router, which exposes an OpenAI-compatible Chat Completions API. Models are addressed by their canonical HF id (e.g. "meta-llama/Llama-3.3-70B-Instruct").

func NewHuggingFaceProvider 

func NewHuggingFaceProvider(apiKey, model, baseURL string) *HuggingFaceProvider

NewHuggingFaceProvider builds a provider against the HF router. baseURL may be empty to use DefaultHuggingFaceBaseURL.

func (*HuggingFaceProvider) Chat 

func (p *HuggingFaceProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*HuggingFaceProvider) ChatStream added in v0.0.2 

func (p *HuggingFaceProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*HuggingFaceProvider) CreateStructuredOutput 

func (p *HuggingFaceProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*HuggingFaceProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *HuggingFaceProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images).

func (*HuggingFaceProvider) CreateStructuredOutputFromSchema 

func (p *HuggingFaceProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*HuggingFaceProvider) MaxInputTokens 

func (p *HuggingFaceProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*HuggingFaceProvider) Model 

func (p *HuggingFaceProvider) Model() string

func (*HuggingFaceProvider) Name 

func (p *HuggingFaceProvider) Name() string

func (*HuggingFaceProvider) RawClient 

func (p *HuggingFaceProvider) RawClient() *openai.Client

func (*HuggingFaceProvider) SetMeter 

func (p *HuggingFaceProvider) SetMeter(hook MeterHook)

func (*HuggingFaceProvider) SetModeration 

func (p *HuggingFaceProvider) SetModeration(m ModerationProvider)

func (*HuggingFaceProvider) WithMeter 

func (p *HuggingFaceProvider) WithMeter(hook MeterHook) *HuggingFaceProvider

func (*HuggingFaceProvider) WithModeration 

func (p *HuggingFaceProvider) WithModeration(m ModerationProvider) *HuggingFaceProvider

type ImageMeterable 

type ImageMeterable interface {
	SetMeter(MeterHook)
}

ImageMeterable is implemented by image providers that accept a meter hook.

type ImagePart 

type ImagePart struct {
	MediaType string // e.g. "image/png"
	Data      string // base64-encoded
}

ImagePart carries base64-encoded image data.

type ImageProvider 

type ImageProvider interface {
	Generate(ctx context.Context, prompt, model, size string) (string, error)                                 // base64
	Edit(ctx context.Context, image []byte, editPrompt string) (string, error)                                // base64
	EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error) // base64
}

ImageProvider generates and edits images via an AI model.

func NewImageProvider 

func NewImageProvider(ctx context.Context, providerName, apiKey, model string) (ImageProvider, error)

NewImageProvider creates an ImageProvider from a provider name ("openai" or "gemini").

type LLMMeterable 

type LLMMeterable interface {
	SetMeter(MeterHook)
}

LLMMeterable is implemented by providers that accept a meter hook. SetLLMMeter uses this instead of a type-switch so new providers work without updating the switch.

type LLMModerable 

type LLMModerable interface {
	SetModeration(ModerationProvider)
}

LLMModerable is implemented by providers that accept a moderation provider.

type LLMProvider 

type LLMProvider interface {
	Name() string
	Model() string
	CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error
	CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)
	Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)
}

LLMProvider is the consumer-facing contract for any LLM backend.

func NewLLMProvider 

func NewLLMProvider(providerName, apiKey, model string, opts ...ProviderOption) LLMProvider

NewLLMProvider creates an LLMProvider from a provider name ("openai", "anthropic", "gemini", "ollama", or "huggingface"). For ollama, apiKey is the base URL (e.g. "http://ollama:11434/v1"). For huggingface, apiKey is the HF token; base URL is read from HF_BASE_URL (defaults to DefaultHuggingFaceBaseURL). For gemini, apiKey is the Google AI API key.

type Message 

type Message struct {
	Role         string        `json:"role"`
	Content      string        `json:"content,omitempty"`
	Parts        []Part        `json:"-"`
	ToolCalls    []ToolCall    `json:"tool_calls,omitempty"`
	ToolCallID   string        `json:"tool_call_id,omitempty"`
	CacheControl *CacheControl `json:"cache_control,omitempty"`
}

Message is a provider-agnostic chat message.

func (Message) MarshalJSON 

func (m Message) MarshalJSON() ([]byte, error)

func (*Message) UnmarshalJSON 

func (m *Message) UnmarshalJSON(data []byte) error

type MeterHook 

type MeterHook func(UsageEvent)

MeterHook is called after each LLM/image call with usage data.

type Middleware 

type Middleware func(next ModelFunc) ModelFunc

Middleware wraps a ModelFunc with cross-cutting behaviour.

func Chain 

func Chain(mws ...Middleware) Middleware

Chain composes middleware so the first in the list is outermost.

type ModelFunc 

type ModelFunc func(ctx context.Context, msgs []Message, tools []Tool) (*Response, error)

ModelFunc is the signature of an LLM Chat call, abstracted from any concrete provider.

type ModerationError 

type ModerationError struct {
	Categories map[string]bool
}

ModerationError is returned when a prompt is blocked by moderation.

func (*ModerationError) Error 

func (e *ModerationError) Error() string

type ModerationProvider 

type ModerationProvider interface {
	Check(ctx context.Context, input string) (*ModerationResult, error)
}

ModerationProvider checks user-supplied text for policy violations.

func NewModerationProvider 

func NewModerationProvider(providerName, apiKey string) ModerationProvider

NewModerationProvider creates a ModerationProvider from a provider name.

type ModerationResult 

type ModerationResult struct {
	Flagged    bool            `json:"flagged"`
	Categories map[string]bool `json:"categories,omitempty"`
}

ModerationResult holds the outcome of a moderation check.

type MultimodalStructuredProvider added in v0.2.0 

type MultimodalStructuredProvider interface {
	CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)
}

MultimodalStructuredProvider is implemented by providers that accept a multimodal user turn (text + images) for schema-constrained structured output. Every built-in provider implements it; third-party LLMProvider implementations may not, which is why StructuredOutputFromParts exists.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider wraps the OpenAI-compatible API exposed by Ollama.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) *OllamaProvider

NewOllamaProvider creates a provider pointing at an Ollama instance. baseURL is the Ollama server, e.g. "http://ollama:11434/v1".

func (*OllamaProvider) Chat 

func (p *OllamaProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OllamaProvider) ChatStream added in v0.0.2 

func (p *OllamaProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OllamaProvider) CreateStructuredOutput 

func (p *OllamaProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OllamaProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *OllamaProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn. Ollama models don't always honour tool calling, so the schema is prompted for directly; images ride along as chat parts for vision-capable models.

func (*OllamaProvider) CreateStructuredOutputFromSchema 

func (p *OllamaProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OllamaProvider) Model 

func (p *OllamaProvider) Model() string

func (*OllamaProvider) Name 

func (p *OllamaProvider) Name() string

func (*OllamaProvider) SetMeter 

func (p *OllamaProvider) SetMeter(hook MeterHook)

func (*OllamaProvider) SetModeration 

func (p *OllamaProvider) SetModeration(m ModerationProvider)

func (*OllamaProvider) WithMeter 

func (p *OllamaProvider) WithMeter(hook MeterHook) *OllamaProvider

func (*OllamaProvider) WithModeration 

func (p *OllamaProvider) WithModeration(m ModerationProvider) *OllamaProvider

type OpenAIImageProvider 

type OpenAIImageProvider struct {
	// contains filtered or unexported fields
}

OpenAIImageProvider implements ImageProvider using the OpenAI images API.

func (*OpenAIImageProvider) Edit 

func (p *OpenAIImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) EditWithReference 

func (p *OpenAIImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) Generate 

func (p *OpenAIImageProvider) Generate(ctx context.Context, prompt, model, size string) (string, error)

func (*OpenAIImageProvider) SetMeter 

func (p *OpenAIImageProvider) SetMeter(hook MeterHook)

func (*OpenAIImageProvider) SetModeration 

func (p *OpenAIImageProvider) SetModeration(m ModerationProvider)

func (*OpenAIImageProvider) WithModeration 

func (p *OpenAIImageProvider) WithModeration(m ModerationProvider) *OpenAIImageProvider

type OpenAIModerationProvider 

type OpenAIModerationProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIModerationProvider 

func NewOpenAIModerationProvider(apiKey string) *OpenAIModerationProvider

func (*OpenAIModerationProvider) Check 

func (p *OpenAIModerationProvider) Check(ctx context.Context, input string) (*ModerationResult, error)

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIProvider 

func NewOpenAIProvider(apiKey, model string) *OpenAIProvider

func (*OpenAIProvider) Chat 

func (p *OpenAIProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OpenAIProvider) ChatStream added in v0.0.2 

func (p *OpenAIProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OpenAIProvider) CreateStructuredOutput 

func (p *OpenAIProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OpenAIProvider) CreateStructuredOutputBreakpointed 

func (p *OpenAIProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM. OpenAI auto-prefix-caches any stable prefix ≥1024 tokens, so the breakpointing surf is implemented by concatenating sysPrompt and stableMid into the single system message. No explicit markers needed — the byte-stable prefix is the cache key.

dynamicTail rides as the user message (uncached).

func (*OpenAIProvider) CreateStructuredOutputFromParts added in v0.2.0 

func (p *OpenAIProvider) CreateStructuredOutputFromParts(ctx context.Context, parts []Part, sysPrompt string, schema json.RawMessage) (map[string]any, error)

CreateStructuredOutputFromParts is CreateStructuredOutputFromSchema with a multimodal user turn (text + base64 images). Honours WithMaxTokens (default 4096) and forces the structured_output tool so the model cannot answer in prose.

func (*OpenAIProvider) CreateStructuredOutputFromSchema 

func (p *OpenAIProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OpenAIProvider) EmbedDimensions 

func (p *OpenAIProvider) EmbedDimensions() int

func (*OpenAIProvider) EmbedText 

func (p *OpenAIProvider) EmbedText(ctx context.Context, texts []string) ([][]float32, error)

func (*OpenAIProvider) MaxInputTokens 

func (p *OpenAIProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*OpenAIProvider) Model 

func (p *OpenAIProvider) Model() string

func (*OpenAIProvider) Name 

func (p *OpenAIProvider) Name() string

func (*OpenAIProvider) RawClient 

func (p *OpenAIProvider) RawClient() *openai.Client

func (*OpenAIProvider) SetMeter 

func (p *OpenAIProvider) SetMeter(hook MeterHook)

func (*OpenAIProvider) SetModeration 

func (p *OpenAIProvider) SetModeration(m ModerationProvider)

func (*OpenAIProvider) WithMeter 

func (p *OpenAIProvider) WithMeter(hook MeterHook) *OpenAIProvider

func (*OpenAIProvider) WithModeration 

func (p *OpenAIProvider) WithModeration(m ModerationProvider) *OpenAIProvider

type OpenAISTTProvider 

type OpenAISTTProvider struct {
	// contains filtered or unexported fields
}

func (*OpenAISTTProvider) Transcribe 

func (p *OpenAISTTProvider) Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)

type Part 

type Part interface {
	// contains filtered or unexported methods
}

Part is a single element of a multimodal message. Sealed — only TextPart and ImagePart satisfy this interface.

type PromptBlocks 

type PromptBlocks map[string]BlockSize

PromptBlocks is a per-call breakdown of the prompt into named blocks. Each entry carries chars + tokens. The meter hook stamps this onto UsageEvent.Metadata["blocks"] so downstream recorders can see the breakdown without needing each provider to crack the prompt apart.

func PromptBlocksFromCtx 

func PromptBlocksFromCtx(ctx context.Context) PromptBlocks

PromptBlocksFromCtx returns the prompt-block breakdown stamped on ctx, or nil if none. Providers call this inside emitUsage to enrich the UsageEvent.Metadata.

type ProviderOption 

type ProviderOption func(*providerConfig)

ProviderOption configures an LLMProvider created by NewLLMProvider.

func WithMeterOption 

func WithMeterOption(hook MeterHook) ProviderOption

WithMeterOption returns a ProviderOption that attaches a meter hook.

func WithModerationOption 

func WithModerationOption(m ModerationProvider) ProviderOption

WithModerationOption returns a ProviderOption that attaches a moderation provider.

type Response 

type Response struct {
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Response is the provider-agnostic result of a Chat call.

type STTProvider 

type STTProvider interface {
	Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)
}

STTProvider transcribes audio into text.

func NewSTTProvider 

func NewSTTProvider(providerName, apiKey, model string) STTProvider

NewSTTProvider creates an STTProvider from a provider name + API key + model. Only OpenAI is supported for now — returns nil for other providers.

type StreamChunk added in v0.0.2 

type StreamChunk struct {
	Text string
}

StreamChunk is a single piece of a streaming response.

type StreamResult added in v0.0.2 

type StreamResult struct {
	// contains filtered or unexported fields
}

StreamResult holds the final accumulated response from a stream.

func Stream added in v0.0.2 

func Stream(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (iter.Seq2[StreamChunk, error], *StreamResult)

Stream returns an iter.Seq2 that yields chunks as they arrive. The returned StreamResult provides the accumulated *Response after iteration. If the provider does not implement StreamingProvider, falls back to Chat.

func (*StreamResult) Response added in v0.0.2 

func (r *StreamResult) Response() (*Response, error)

Response blocks until streaming completes and returns the accumulated response.

type StreamingProvider added in v0.0.2 

type StreamingProvider interface {
	ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)
}

StreamingProvider extends LLMProvider with streaming chat support.

type TextPart 

type TextPart struct {
	Text string
}

TextPart carries inline text.

type Tool 

type Tool struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema object
}

Tool defines a provider-agnostic tool the model can call.

type ToolCall 

type ToolCall struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

ToolCall represents the model requesting a tool invocation.

type UsageEvent 

type UsageEvent struct {
	CallerID         uuid.UUID `json:"caller_id"`
	Provider         string    `json:"provider"`
	Model            string    `json:"model"`
	Operation        string    `json:"operation"`
	InputTokens      int       `json:"input_tokens"`
	OutputTokens     int       `json:"output_tokens"`
	TotalTokens      int       `json:"total_tokens"`
	EstimatedCostUSD float64   `json:"estimated_cost_usd"`
	// CacheCreationInputTokens counts tokens written to the provider's
	// prompt cache on this call (Anthropic only, populated when cache
	// writes occur; priced at ~1.25x base). Zero on cache hit or when
	// caching isn't active.
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`
	// CacheReadInputTokens counts tokens served from the provider's
	// prompt cache (Anthropic only; priced at ~0.1x base). Nonzero means
	// WithCacheSysPrompt is actively paying off.
	CacheReadInputTokens int `json:"cache_read_input_tokens,omitempty"`
	// SystemPrompt/UserPrompt are captured for dev debug display only — empty
	// in production (see capturePromptForDebug).
	SystemPrompt string `json:"system_prompt,omitempty"`
	UserPrompt   string `json:"user_prompt,omitempty"`
	// DebugSpanID is the ID of the PromptMiddlewareChain span that was active
	// when this LLM call fired — used by the debug panel meter hook to attach
	// the llm_call event to the correct span in the tree. Empty in prod.
	DebugSpanID string         `json:"debug_span_id,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
}

UsageEvent represents a single LLM/image-gen usage event for metering.

type VideoMeterable added in v0.2.0 

type VideoMeterable interface {
	SetMeter(MeterHook)
}

VideoMeterable is implemented by video providers that accept a meter hook.

type VideoProvider added in v0.2.0 

type VideoProvider interface {
	Name() string
	Model() string
	Generate(ctx context.Context, req VideoRequest) (*VideoResult, error)
}

VideoProvider generates short video clips from a prompt and an optional conditioning image.

func NewVideoProvider added in v0.2.0 

func NewVideoProvider(providerName, apiKey, model string) (VideoProvider, error)

NewVideoProvider creates a VideoProvider from a provider name ("fal"). For fal, model is the full endpoint id (default DefaultFalVideoModel).

type VideoRequest added in v0.2.0 

type VideoRequest struct {
	Prompt         string
	NegativePrompt string
	// ImageURL is forwarded verbatim (https:// or data: URI).
	ImageURL string
	// Image is raw image bytes; encoded as a data: URI when ImageURL is empty.
	Image []byte
	// ImageMediaType is the MIME type of Image ("image/jpeg" when empty).
	ImageMediaType string
	// Duration in seconds. Providers snap to their supported set.
	Duration float64
	// Resolution label, e.g. "1080p" | "1440p" | "2160p".
	Resolution string
	// AspectRatio label, e.g. "auto" | "16:9" | "9:16".
	AspectRatio string
	// FPS of the generated clip; 0 = provider default.
	FPS int
	// Audio asks the model to generate a soundtrack when it can.
	Audio bool
	// Seed pins generation; nil = random per call.
	Seed *int64
	// Model overrides the provider's default model/endpoint for this call.
	Model string
}

VideoRequest describes a single clip generation.

Image / ImageURL are optional first-frame conditioning (image-to-video). Leave both empty for text-to-video. Zero-valued knobs mean "provider default" — providers only forward fields the caller set.

type VideoResult added in v0.2.0 

type VideoResult struct {
	URL         string
	ContentType string
	FileName    string
	FileSize    int64
	Width       int
	Height      int
	FPS         float64
	Duration    float64 // seconds
	NumFrames   int
	Model       string
	Seed        int64
	CostUSD     float64
	Elapsed     time.Duration
}

VideoResult is the provider-agnostic outcome of a Generate call.

URL points at a provider-hosted file that is typically temporary — callers that need durability must download it promptly.

Source Files

View all Source files

Directories

Path Synopsis
cmd
testmatrix command
testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.
 testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.