ai

package module
v0.0.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 18, 2026 License: MIT Imports: 26 Imported by: 0

README


ai

CI Coverage Go Reference Go Report Card

Multi-provider AI module for Go with token and cost tracking.

Install

go get github.com/nurf-ai/ai

Usage

import "github.com/nurf-ai/ai"

provider := ai.NewLLMProvider("anthropic", apiKey, "claude-sonnet-5")

resp, err := provider.Chat(ctx, []ai.Message{
    {Role: "user", Content: "Hello"},
}, nil)

Providers

Factory Providers Features
NewLLMProvider(name, key, model) anthropic, openai, ollama, huggingface Chat, structured output
NewImageProvider(ctx, name, key, model) openai, gemini Image generation / editing
NewSTTProvider(name, key, model) openai Speech-to-text
NewEmbedder(name, key) openai Text embeddings

Capability Matrix

Provider Chat Structured Output Tool Use Reasoning Moderation Prompt Caching Embeddings STT Img: Generate Img: Edit Img: Edit w/ Ref
Anthropic (Claude) x x x x x
OpenAI (GPT) x x x x x x x x
Ollama (local) x x x
Hugging Face x x x
Gemini x x x

Pricing

Built-in per-model cost estimation via EstimateCostFull. Rates for all supported models (input, output, cache write, cache read, image generation) are maintained in pricing.json — the single source of truth, embedded at compile time.

Metering

Built-in usage metering via MeterHook. Attach a hook to track tokens, cost, and prompt block attribution per call.

ai.SetLLMMeter(provider, func(ev ai.UsageEvent) {
    log.Printf("%s: %d tokens, $%.6f", ev.Model, ev.TotalTokens, ev.EstimatedCostUSD)
})

Image Generation

img, err := ai.NewImageProvider(ctx, "gemini", apiKey, "gemini-2.5-flash-image")
b64, err := img.Generate(ctx, "a cat in space", "", "")

Embeddings

embedder := ai.NewEmbedder("openai", apiKey)
vectors, err := embedder.EmbedText(ctx, []string{"hello world"})

Speech-to-Text

stt := ai.NewSTTProvider("openai", apiKey, "whisper-1")
text, err := stt.Transcribe(ctx, audioReader, "audio.mp3")

Contributing

Setup

git clone https://github.com/nurf-ai/ai.git
cd ai
go test ./...

Adding a provider

  1. Create <provider>.go implementing LLMProvider
  2. Add model entries to model_limits.go (modelMaxInputTokens)
  3. Add pricing entries to pricing.json under the provider key
  4. Register in provider.go (NewLLMProvider switch)
  5. Run go test ./... — TestPricingCoverage enforces limits/pricing sync

Updating pricing

Edit pricing.json directly — it's embedded at build time via go:embed. No Go code changes needed for price updates.

Guidelines

  • Tests are unit-only (no API calls)
  • CI runs go vet, golangci-lint, and go test -race — make sure they pass locally before submitting:
go vet ./...
gofmt -l .
golangci-lint run
go test -race ./...

Documentation

Overview

Package ai is the consumer-facing LLM contract — provider-agnostic types (Tool/Message/Response/ToolCall/Part) plus the LLMProvider interface in ai_contract.go. Concrete implementations (Anthropic/OpenAI/Ollama/ HuggingFace) live in sibling files; they satisfy LLMProvider structurally.

Index

Constants

View Source 
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants used in Message.Role.

View Source 
const DefaultHuggingFaceBaseURL = "https://router.huggingface.co/v1"

DefaultHuggingFaceBaseURL is the HF Inference router's OpenAI-compatible endpoint. Override via HF_BASE_URL when targeting a dedicated endpoint or a specific upstream provider route (e.g. https://router.huggingface.co/<provider>/v1).

Variables

This section is empty.

Functions

func CacheSysPromptFromCtx 

func CacheSysPromptFromCtx(ctx context.Context) bool

CacheSysPromptFromCtx reports whether WithCacheSysPrompt was stamped. Adapters call this while building the request to decide whether to mark the sys block as cacheable.

func CountTokens 

func CountTokens(s string) int

CountTokens returns an approximate token count for s using the o200k_base BPE encoding. Exact for GPT-4o / GPT-5 family; within ~15% for Anthropic and HF model families. Empty string returns 0.

Falls back to a len(s)/4 heuristic if the tiktoken encoder fails to initialize, so this never returns an error — a usable rough number is more helpful at call sites than a plumbed error path.

func CountTokensOver 

func CountTokensOver(s string, limit int64) bool

CountTokensOver reports whether counting tokens in s exceeds limit. Short-circuits using the cheap char-based upper bound before invoking the BPE encoder, so callers can gate long strings without paying the full tokenization cost when they're obviously under the limit.

func DebugPromptsEnabled 

func DebugPromptsEnabled() bool

DebugPromptsEnabled reports whether prompt capture is active. Providers check this before building sys/user prompt strings for the debug panel.

func DebugSpanIDFromCtx 

func DebugSpanIDFromCtx(ctx context.Context) string

DebugSpanIDFromCtx returns the span ID on the context, or empty.

func EstimateCostFull 

func EstimateCostFull(model string, in, out, cw, cr int) float64

func GenerateTyped 

func GenerateTyped[T any](ctx context.Context, p LLMProvider, userPrompt, sysPrompt string) (*T, error)

GenerateTyped wraps CreateStructuredOutput with Go generics so callers get back a typed pointer without pre-allocating the output struct.

func MaxInputTokensLLM 

func MaxInputTokensLLM(company, modelName string) (int64, error)

MaxInputTokensLLM returns the advertised maximum input context window in tokens for the given model.

company is the model-family owner / API namespace — one of "anthropic", "openai", or "huggingface". modelName is the exact model id as passed to the provider constructor (for HF that's the full "<org>/<model>" id). Trailing date suffixes like "-20250514" are stripped automatically.

Returns an error if the model is unknown. Callers that need a safe default should handle the error explicitly rather than rely on a fallback — silently returning 0 or a guess would mask config typos.

func MaxTokensFromCtx 

func MaxTokensFromCtx(ctx context.Context, fallback int) int

func MeterOperationFromCtx 

func MeterOperationFromCtx(ctx context.Context) string

func MeterPeerIDFromCtx 

func MeterPeerIDFromCtx(ctx context.Context) uuid.UUID

func ModelsForProvider 

func ModelsForProvider(provider string) []string

func PricingTable 

func PricingTable() map[string]modelPricing

func ReasoningEffortFromCtx 

func ReasoningEffortFromCtx(ctx context.Context) string

func SetDebugPromptsEnabled 

func SetDebugPromptsEnabled(enabled bool)

SetDebugPromptsEnabled toggles prompt capture in UsageEvents. Call once at startup based on your dev-mode flag.

func SetImageMeter 

func SetImageMeter(img ImageProvider, hook MeterHook)

SetImageMeter attaches a meter hook to any ImageProvider that satisfies ImageMeterable.

func SetImageModeration 

func SetImageModeration(img ImageProvider, m ModerationProvider)

SetImageModeration attaches a moderation provider to any ImageProvider that satisfies the moderable interface.

func SetLLMMeter 

func SetLLMMeter(llm LLMProvider, hook MeterHook)

SetLLMMeter attaches a meter hook to any LLMProvider that satisfies LLMMeterable.

func SetLLMModeration 

func SetLLMModeration(llm LLMProvider, m ModerationProvider)

SetLLMModeration attaches a moderation provider to any LLMProvider that satisfies LLMModerable.

func SetLogger 

func SetLogger(l *zap.Logger)

func SetOnModerationError 

func SetOnModerationError(fn func(ctx context.Context, err error) error)

SetOnModerationError overrides the default moderation-error behavior. Default (nil): warn and allow. When set, the function's return value determines whether the call proceeds (nil) or is rejected (non-nil error).

func SetUnknownModelHook 

func SetUnknownModelHook(hook func(model string))

func TransparentBGFromCtx 

func TransparentBGFromCtx(ctx context.Context) bool

func TruncatePromptForDebug 

func TruncatePromptForDebug(s string) string

TruncatePromptForDebug is kept for callers that still reference the old helper name; it now just delegates to capturePromptForDebug — prompts are no longer truncated. Dev mode only carries them at all.

func WithCacheSysPrompt 

func WithCacheSysPrompt(ctx context.Context) context.Context

WithCacheSysPrompt signals that the sys prompt sent in this call is a stable prefix worth provider-side prompt caching. Anthropic's adapter sets cache_control: ephemeral on the sys block; OpenAI/Ollama/HF are no-op (OpenAI auto-caches prompts ≥1024 tokens; Ollama uses KV-cache for matching prefixes at the inference layer; HF depends on backend).

Call this at the call site right before the LLM invocation. Marker is per-call, not global — stamp only when the caller knows the sys prompt is stable across turns (e.g. planner/router system prompts).

func WithDebugSpanID 

func WithDebugSpanID(ctx context.Context, spanID string) context.Context

WithDebugSpanID stamps the current dev-debug span ID on the context so providers can include it on UsageEvents. Caller must be in dev mode; in prod this is a no-op path (the middleware chain never calls StartSpan).

func WithMaxTokens 

func WithMaxTokens(ctx context.Context, n int) context.Context

func WithMeterOperation 

func WithMeterOperation(ctx context.Context, op string) context.Context

func WithMeterPeerID 

func WithMeterPeerID(ctx context.Context, id uuid.UUID) context.Context

func WithPromptBlocks 

func WithPromptBlocks(ctx context.Context, contents map[string]string) context.Context

WithPromptBlocks stamps a per-block breakdown on the context so providers can include it on UsageEvents. Call-site passes the raw block strings keyed by block name — this helper computes both Chars (len(s)) and Tokens (CountTokens(s)) per entry.

Tokenizer cost is paid here, but `dev-only` callers and a single tokenize per call mean it's negligible vs the LLM round-trip. If you need to skip the tokenize (e.g. hot path that doesn't care about tokens), call WithPromptBlocksRaw with a pre-built map[string]BlockSize directly.

Empty maps are intentionally not stored — PromptBlocksFromCtx returning nil is the canonical "absent" signal.

func WithPromptBlocksRaw 

func WithPromptBlocksRaw(ctx context.Context, b PromptBlocks) context.Context

WithPromptBlocksRaw is the lower-level variant — caller pre-computes Chars and Tokens per block. Useful when the same block string is reused across calls and you want to cache the token count.

func WithReasoningEffort 

func WithReasoningEffort(ctx context.Context, effort string) context.Context

WithReasoningEffort sets the reasoning effort level ("low", "medium", "high") for providers that support it. OpenAI: maps to reasoning_effort in chat completions. Anthropic: maps to extended thinking with a budget derived from effort level.

func WithTransparentBG 

func WithTransparentBG(ctx context.Context) context.Context

WithTransparentBG signals image providers to produce transparent backgrounds.

Types

type AnthropicProvider 

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

func NewAnthropicProvider 

func NewAnthropicProvider(apiKey, model string) *AnthropicProvider

func (*AnthropicProvider) Chat 

func (p *AnthropicProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*AnthropicProvider) CreateStructuredOutput 

func (p *AnthropicProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*AnthropicProvider) CreateStructuredOutputBreakpointed 

func (p *AnthropicProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM by emitting two system blocks each marked with cache_control: ephemeral. The provider hashes the prefix up through each cache_control marker, so when sysPrompt + stableMid is byte-stable across turns we hit the bp2 entry (everything stable cached); when only sysPrompt is stable (e.g. the candidate list changed) we hit bp1 (sysPrompt cached, stableMid reprocessed). One marker per change-rate tier — see .wiki/context/cache-breakpoints.md.

dynamicTail is the per-call query and rides as the user message — never cached.

func (*AnthropicProvider) CreateStructuredOutputFromSchema 

func (p *AnthropicProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*AnthropicProvider) MaxInputTokens 

func (p *AnthropicProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*AnthropicProvider) Model 

func (p *AnthropicProvider) Model() string

func (*AnthropicProvider) Name 

func (p *AnthropicProvider) Name() string

func (*AnthropicProvider) SetMeter 

func (p *AnthropicProvider) SetMeter(hook MeterHook)

func (*AnthropicProvider) SetModeration 

func (p *AnthropicProvider) SetModeration(m ModerationProvider)

func (*AnthropicProvider) WithMeter 

func (p *AnthropicProvider) WithMeter(hook MeterHook) *AnthropicProvider

func (*AnthropicProvider) WithModeration 

func (p *AnthropicProvider) WithModeration(m ModerationProvider) *AnthropicProvider

type BlockSize 

type BlockSize struct {
	Chars  int `json:"chars"`
	Tokens int `json:"tokens"`
}

BlockSize carries the two attribution signals per prompt block: chars (raw length) and tokens (tiktoken o200k_base count). Tokens are what bills you; chars exist for sanity (a high tokens/chars ratio per block flags tokenizer-hostile content like code, base64, or heavy unicode — surfd as the "density" warning in PromptBlocksBar).

type CacheControl 

type CacheControl struct {
	Type string `json:"type"` // "ephemeral"
}

CacheControl tells a provider to mark this message (or its last content block) as a cache breakpoint. Only Anthropic acts on it today — other providers ignore it silently.

type Embedder 

type Embedder interface {
	EmbedText(ctx context.Context, texts []string) ([][]float32, error)
	EmbedDimensions() int
}

Embedder produces vector embeddings for text. Consumers that need semantic similarity (discovery, memory recall, search ranking) depend on this interface. Not all LLM providers support embeddings — wire a separate provider if the main LLM doesn't (e.g. Anthropic + OpenAI embedding sidecar).

func EmbedderFromLLM 

func EmbedderFromLLM(llm LLMProvider) Embedder

EmbedderFromLLM extracts an Embedder from an LLMProvider if it satisfies the Embedder interface (e.g. OpenAIProvider).

func NewEmbedder 

func NewEmbedder(providerName, apiKey string) Embedder

NewEmbedder creates an Embedder from a provider name + API key. Only OpenAI is supported for now — returns nil for other providers.

type ErrorKind 

type ErrorKind int
const (
	ErrUnknown ErrorKind = iota
	ErrBilling
	ErrAuth
	ErrRateLimit
	ErrModeration
	ErrProviderDown
)

func ClassifyError 

func ClassifyError(err error) (ErrorKind, string)

func (ErrorKind) Retryable 

func (k ErrorKind) Retryable() bool

func (ErrorKind) UserMessage 

func (k ErrorKind) UserMessage() string

type GeminiImageProvider 

type GeminiImageProvider struct {
	// contains filtered or unexported fields
}

GeminiImageProvider implements ImageProvider using Google's Gemini API.

func (*GeminiImageProvider) Edit 

func (p *GeminiImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) EditWithReference 

func (p *GeminiImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) Generate 

func (p *GeminiImageProvider) Generate(ctx context.Context, prompt, model, _ string) (string, error)

func (*GeminiImageProvider) SetMeter 

func (p *GeminiImageProvider) SetMeter(hook MeterHook)

func (*GeminiImageProvider) SetModeration 

func (p *GeminiImageProvider) SetModeration(m ModerationProvider)

func (*GeminiImageProvider) WithModeration 

func (p *GeminiImageProvider) WithModeration(m ModerationProvider) *GeminiImageProvider

type HTTPMeterEmitter 

type HTTPMeterEmitter struct {
	// contains filtered or unexported fields
}

HTTPMeterEmitter buffers usage events and POSTs them to a meter service.

func NewHTTPMeterEmitter 

func NewHTTPMeterEmitter(opts HTTPMeterOpts) *HTTPMeterEmitter

func (*HTTPMeterEmitter) Hook 

func (e *HTTPMeterEmitter) Hook() MeterHook

type HTTPMeterOpts 

type HTTPMeterOpts struct {
	Endpoint      string                             // full URL — no suffix appended
	AuthHeader    string                             // sent as Authorization header when non-empty
	BatchSize     int                                // default 32
	FlushInterval time.Duration                      // default 2s
	Marshal       func([]UsageEvent) ([]byte, error) // default json.Marshal
	ContentType   string                             // default "application/json"
	OnError       func(error)                        // default logger.Warn
}

HTTPMeterOpts configures an HTTPMeterEmitter.

type HuggingFaceProvider 

type HuggingFaceProvider struct {
	// contains filtered or unexported fields
}

HuggingFaceProvider speaks the HF Inference Router, which exposes an OpenAI-compatible Chat Completions API. Models are addressed by their canonical HF id (e.g. "meta-llama/Llama-3.3-70B-Instruct").

func NewHuggingFaceProvider 

func NewHuggingFaceProvider(apiKey, model, baseURL string) *HuggingFaceProvider

NewHuggingFaceProvider builds a provider against the HF router. baseURL may be empty to use DefaultHuggingFaceBaseURL.

func (*HuggingFaceProvider) Chat 

func (p *HuggingFaceProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*HuggingFaceProvider) CreateStructuredOutput 

func (p *HuggingFaceProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*HuggingFaceProvider) CreateStructuredOutputFromSchema 

func (p *HuggingFaceProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*HuggingFaceProvider) MaxInputTokens 

func (p *HuggingFaceProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*HuggingFaceProvider) Model 

func (p *HuggingFaceProvider) Model() string

func (*HuggingFaceProvider) Name 

func (p *HuggingFaceProvider) Name() string

func (*HuggingFaceProvider) RawClient 

func (p *HuggingFaceProvider) RawClient() *openai.Client

func (*HuggingFaceProvider) SetMeter 

func (p *HuggingFaceProvider) SetMeter(hook MeterHook)

func (*HuggingFaceProvider) SetModeration 

func (p *HuggingFaceProvider) SetModeration(m ModerationProvider)

func (*HuggingFaceProvider) WithMeter 

func (p *HuggingFaceProvider) WithMeter(hook MeterHook) *HuggingFaceProvider

func (*HuggingFaceProvider) WithModeration 

func (p *HuggingFaceProvider) WithModeration(m ModerationProvider) *HuggingFaceProvider

type ImageMeterable 

type ImageMeterable interface {
	SetMeter(MeterHook)
}

ImageMeterable is implemented by image providers that accept a meter hook.

type ImagePart 

type ImagePart struct {
	MediaType string // e.g. "image/png"
	Data      string // base64-encoded
}

ImagePart carries base64-encoded image data.

type ImageProvider 

type ImageProvider interface {
	Generate(ctx context.Context, prompt, model, size string) (string, error)                                 // base64
	Edit(ctx context.Context, image []byte, editPrompt string) (string, error)                                // base64
	EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error) // base64
}

ImageProvider generates and edits images via an AI model.

func NewImageProvider 

func NewImageProvider(ctx context.Context, providerName, apiKey, model string) (ImageProvider, error)

NewImageProvider creates an ImageProvider from a provider name ("openai" or "gemini").

type LLMMeterable 

type LLMMeterable interface {
	SetMeter(MeterHook)
}

LLMMeterable is implemented by providers that accept a meter hook. SetLLMMeter uses this instead of a type-switch so new providers work without updating the switch.

type LLMModerable 

type LLMModerable interface {
	SetModeration(ModerationProvider)
}

LLMModerable is implemented by providers that accept a moderation provider.

type LLMProvider 

type LLMProvider interface {
	Name() string
	Model() string
	CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error
	CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)
	Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)
}

LLMProvider is the consumer-facing contract for any LLM backend.

func NewLLMProvider 

func NewLLMProvider(providerName, apiKey, model string, opts ...ProviderOption) LLMProvider

NewLLMProvider creates an LLMProvider from a provider name ("openai", "anthropic", "ollama", or "huggingface"). For ollama, apiKey is the base URL (e.g. "http://ollama:11434/v1"). For huggingface, apiKey is the HF token; base URL is read from HF_BASE_URL (defaults to DefaultHuggingFaceBaseURL).

type Message 

type Message struct {
	Role         string        `json:"role"`
	Content      string        `json:"content,omitempty"`
	Parts        []Part        `json:"-"`
	ToolCalls    []ToolCall    `json:"tool_calls,omitempty"`
	ToolCallID   string        `json:"tool_call_id,omitempty"`
	CacheControl *CacheControl `json:"cache_control,omitempty"`
}

Message is a provider-agnostic chat message.

func (Message) MarshalJSON 

func (m Message) MarshalJSON() ([]byte, error)

func (*Message) UnmarshalJSON 

func (m *Message) UnmarshalJSON(data []byte) error

type MeterHook 

type MeterHook func(UsageEvent)

MeterHook is called after each LLM/image call with usage data.

type Middleware 

type Middleware func(next ModelFunc) ModelFunc

Middleware wraps a ModelFunc with cross-cutting behaviour.

func Chain 

func Chain(mws ...Middleware) Middleware

Chain composes middleware so the first in the list is outermost.

type ModelFunc 

type ModelFunc func(ctx context.Context, msgs []Message, tools []Tool) (*Response, error)

ModelFunc is the signature of an LLM Chat call, abstracted from any concrete provider.

type ModerationError 

type ModerationError struct {
	Categories map[string]bool
}

ModerationError is returned when a prompt is blocked by moderation.

func (*ModerationError) Error 

func (e *ModerationError) Error() string

type ModerationProvider 

type ModerationProvider interface {
	Check(ctx context.Context, input string) (*ModerationResult, error)
}

ModerationProvider checks user-supplied text for policy violations.

func NewModerationProvider 

func NewModerationProvider(providerName, apiKey string) ModerationProvider

NewModerationProvider creates a ModerationProvider from a provider name.

type ModerationResult 

type ModerationResult struct {
	Flagged    bool            `json:"flagged"`
	Categories map[string]bool `json:"categories,omitempty"`
}

ModerationResult holds the outcome of a moderation check.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider wraps the OpenAI-compatible API exposed by Ollama.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) *OllamaProvider

NewOllamaProvider creates a provider pointing at an Ollama instance. baseURL is the Ollama server, e.g. "http://ollama:11434/v1".

func (*OllamaProvider) Chat 

func (p *OllamaProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OllamaProvider) CreateStructuredOutput 

func (p *OllamaProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OllamaProvider) CreateStructuredOutputFromSchema 

func (p *OllamaProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OllamaProvider) Model 

func (p *OllamaProvider) Model() string

func (*OllamaProvider) Name 

func (p *OllamaProvider) Name() string

func (*OllamaProvider) SetMeter 

func (p *OllamaProvider) SetMeter(hook MeterHook)

func (*OllamaProvider) SetModeration 

func (p *OllamaProvider) SetModeration(m ModerationProvider)

func (*OllamaProvider) WithMeter 

func (p *OllamaProvider) WithMeter(hook MeterHook) *OllamaProvider

func (*OllamaProvider) WithModeration 

func (p *OllamaProvider) WithModeration(m ModerationProvider) *OllamaProvider

type OpenAIImageProvider 

type OpenAIImageProvider struct {
	// contains filtered or unexported fields
}

OpenAIImageProvider implements ImageProvider using the OpenAI images API.

func (*OpenAIImageProvider) Edit 

func (p *OpenAIImageProvider) Edit(ctx context.Context, _ []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) EditWithReference 

func (p *OpenAIImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) Generate 

func (p *OpenAIImageProvider) Generate(ctx context.Context, prompt, model, size string) (string, error)

func (*OpenAIImageProvider) SetMeter 

func (p *OpenAIImageProvider) SetMeter(hook MeterHook)

func (*OpenAIImageProvider) SetModeration 

func (p *OpenAIImageProvider) SetModeration(m ModerationProvider)

func (*OpenAIImageProvider) WithModeration 

func (p *OpenAIImageProvider) WithModeration(m ModerationProvider) *OpenAIImageProvider

type OpenAIModerationProvider 

type OpenAIModerationProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIModerationProvider 

func NewOpenAIModerationProvider(apiKey string) *OpenAIModerationProvider

func (*OpenAIModerationProvider) Check 

func (p *OpenAIModerationProvider) Check(ctx context.Context, input string) (*ModerationResult, error)

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIProvider 

func NewOpenAIProvider(apiKey, model string) *OpenAIProvider

func (*OpenAIProvider) Chat 

func (p *OpenAIProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OpenAIProvider) CreateStructuredOutput 

func (p *OpenAIProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OpenAIProvider) CreateStructuredOutputBreakpointed 

func (p *OpenAIProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM. OpenAI auto-prefix-caches any stable prefix ≥1024 tokens, so the breakpointing surf is implemented by concatenating sysPrompt and stableMid into the single system message. No explicit markers needed — the byte-stable prefix is the cache key.

dynamicTail rides as the user message (uncached).

func (*OpenAIProvider) CreateStructuredOutputFromSchema 

func (p *OpenAIProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OpenAIProvider) EmbedDimensions 

func (p *OpenAIProvider) EmbedDimensions() int

func (*OpenAIProvider) EmbedText 

func (p *OpenAIProvider) EmbedText(ctx context.Context, texts []string) ([][]float32, error)

func (*OpenAIProvider) MaxInputTokens 

func (p *OpenAIProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*OpenAIProvider) Model 

func (p *OpenAIProvider) Model() string

func (*OpenAIProvider) Name 

func (p *OpenAIProvider) Name() string

func (*OpenAIProvider) RawClient 

func (p *OpenAIProvider) RawClient() *openai.Client

func (*OpenAIProvider) SetMeter 

func (p *OpenAIProvider) SetMeter(hook MeterHook)

func (*OpenAIProvider) SetModeration 

func (p *OpenAIProvider) SetModeration(m ModerationProvider)

func (*OpenAIProvider) WithMeter 

func (p *OpenAIProvider) WithMeter(hook MeterHook) *OpenAIProvider

func (*OpenAIProvider) WithModeration 

func (p *OpenAIProvider) WithModeration(m ModerationProvider) *OpenAIProvider

type OpenAISTTProvider 

type OpenAISTTProvider struct {
	// contains filtered or unexported fields
}

func (*OpenAISTTProvider) Transcribe 

func (p *OpenAISTTProvider) Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)

type Part 

type Part interface {
	// contains filtered or unexported methods
}

Part is a single element of a multimodal message. Sealed — only TextPart and ImagePart satisfy this interface.

type PromptBlocks 

type PromptBlocks map[string]BlockSize

PromptBlocks is a per-call breakdown of the prompt into named blocks. Each entry carries chars + tokens. The meter hook stamps this onto UsageEvent.Metadata["blocks"] so downstream recorders can see the breakdown without needing each provider to crack the prompt apart.

func PromptBlocksFromCtx 

func PromptBlocksFromCtx(ctx context.Context) PromptBlocks

PromptBlocksFromCtx returns the prompt-block breakdown stamped on ctx, or nil if none. Providers call this inside emitUsage to enrich the UsageEvent.Metadata.

type ProviderOption 

type ProviderOption func(*providerConfig)

ProviderOption configures an LLMProvider created by NewLLMProvider.

func WithMeterOption 

func WithMeterOption(hook MeterHook) ProviderOption

WithMeterOption returns a ProviderOption that attaches a meter hook.

func WithModerationOption 

func WithModerationOption(m ModerationProvider) ProviderOption

WithModerationOption returns a ProviderOption that attaches a moderation provider.

type Response 

type Response struct {
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Response is the provider-agnostic result of a Chat call.

type STTProvider 

type STTProvider interface {
	Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)
}

STTProvider transcribes audio into text.

func NewSTTProvider 

func NewSTTProvider(providerName, apiKey, model string) STTProvider

NewSTTProvider creates an STTProvider from a provider name + API key + model. Only OpenAI is supported for now — returns nil for other providers.

type TextPart 

type TextPart struct {
	Text string
}

TextPart carries inline text.

type Tool 

type Tool struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema object
}

Tool defines a provider-agnostic tool the model can call.

type ToolCall 

type ToolCall struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

ToolCall represents the model requesting a tool invocation.

type UsageEvent 

type UsageEvent struct {
	PeerID           uuid.UUID `json:"peer_id"`
	Provider         string    `json:"provider"`
	Model            string    `json:"model"`
	Operation        string    `json:"operation"`
	InputTokens      int       `json:"input_tokens"`
	OutputTokens     int       `json:"output_tokens"`
	TotalTokens      int       `json:"total_tokens"`
	EstimatedCostUSD float64   `json:"estimated_cost_usd"`
	// CacheCreationInputTokens counts tokens written to the provider's
	// prompt cache on this call (Anthropic only, populated when cache
	// writes occur; priced at ~1.25x base). Zero on cache hit or when
	// caching isn't active.
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`
	// CacheReadInputTokens counts tokens served from the provider's
	// prompt cache (Anthropic only; priced at ~0.1x base). Nonzero means
	// WithCacheSysPrompt is actively paying off.
	CacheReadInputTokens int `json:"cache_read_input_tokens,omitempty"`
	// SystemPrompt/UserPrompt are captured for dev debug display only — empty
	// in production (see capturePromptForDebug).
	SystemPrompt string `json:"system_prompt,omitempty"`
	UserPrompt   string `json:"user_prompt,omitempty"`
	// DebugSpanID is the ID of the PromptMiddlewareChain span that was active
	// when this LLM call fired — used by the debug panel meter hook to attach
	// the llm_call event to the correct span in the tree. Empty in prod.
	DebugSpanID string         `json:"debug_span_id,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
}

UsageEvent represents a single LLM/image-gen usage event for metering.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.