ai

package module
v0.0.3 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2026 License: MIT Imports: 29 Imported by: 0

README


ai

CI Go Reference

Multimodal Go module for building across AI providers, with realistic cost tracking baked in, not just token counts.

From the obvious to the overlooked: chat, reasoning, tool use, structured output, image gen & editing, speech-to-text, embeddings, moderation.

Install

go get github.com/nurf-ai/ai

Usage

Chat

import "github.com/nurf-ai/ai"

llm := ai.NewLLMProvider(provider, apiKey, model)

resp, err := llm.Chat(ctx, []ai.Message{
    {Role: "user", Content: "Hello"},
}, nil)

Streaming

Falls back to Chat transparently.

Iterator (Go 1.23+):

llm := ai.NewLLMProvider(provider, apiKey, model)

chunks, result := ai.Stream(ctx, llm, messages, nil)

for chunk, err := range chunks {
    fmt.Print(chunk.Text)
}
resp, err := result.Response()

Channel (for select/fan-out):

ch, wait := ai.StreamWithChan(ctx, llm, messages, nil)

for chunk := range ch {
    fmt.Print(chunk.Text)
}
resp, err := wait()

Image Generation

img, err := ai.NewImageProvider(ctx, provider, apiKey, model)
b64, err := img.Generate(ctx, "a cat in space", "", "")

Embeddings

embedder := ai.NewEmbedder(provider, apiKey)
vectors, err := embedder.EmbedText(ctx, []string{"hello world"})

Speech-to-Text

stt := ai.NewSTTProvider(provider, apiKey, model)
text, err := stt.Transcribe(ctx, audioReader, "audio.mp3")

Metering

ai.SetLLMMeter(llm, func(ev ai.UsageEvent) {
    log.Printf("%s: %d tokens, $%.6f", ev.Model, ev.TotalTokens, ev.EstimatedCostUSD)
})

Providers

Factory Providers Features
NewLLMProvider(provider, apiKey, model) anthropic, openai, gemini, ollama, huggingface Chat, streaming, structured output, tools
NewImageProvider(ctx, provider, apiKey, model) openai, gemini Image generation / editing
NewSTTProvider(provider, apiKey, model) openai Speech-to-text
NewEmbedder(provider, apiKey) openai Text embeddings

Capability Matrix

Provider Chat Streaming Structured Output Tool Use Reasoning Moderation Prompt Caching Embeddings STT Img: Generate Img: Edit Img: Edit w/ Ref
Anthropic x x x x x x
OpenAI x x x x x x x x x x
Gemini x x x x x x x
Ollama x x x x
Hugging Face x x x x

Pricing

Built-in per-model cost estimation via EstimateCostFull. Rates and context windows for all supported models are maintained in models.json — the single source of truth, embedded at compile time.

Contributing

Setup

git clone https://github.com/nurf-ai/ai.git
cd ai
go test ./...

Adding a provider

  1. Create <provider>.go implementing LLMProvider
  2. Add model entries to models.json (pricing + max_input_tokens)
  3. Register in provider.go (NewLLMProvider switch)
  4. Add a smoke test to integration_test.go
  5. Run both test suites before submitting

Adding a model

  1. Add entry to models.json under the provider key — include max_input_tokens
  2. go test ./... — TestPricingCoverage will fail if max_input_tokens is missing

Updating models

Edit models.json directly — it's embedded at build time via go:embed. No Go code changes needed.

Testing

Unit tests (no API keys needed, runs in CI):

go test ./...

Integration tests (real API calls, run locally):

cp .env.test.tpl .env.test   # fill in your API keys
go test -tags=integration -count=1 -json ./... | go run ./cmd/testmatrix

.env.test is loaded automatically via godotenv in TestMain. Only set the keys you have — providers with missing keys are skipped (∅).

testmatrix prints live progress with per-test token counts and cost, failure details, and per-provider cost totals to stderr:

  ✓ Anthropic/claude-haiku-4-5/Chat (1.8s)                     800 tok_in, 80 tok_out, $0.0003
  ✓ OpenAI/gpt-4o-mini/Chat (1.2s)                             500 tok_in, 50 tok_out, $0.0001
  ✓ OpenAI/gpt-4o-mini/Moderation (0.3s)
  ∅ Ollama

4 passed, 0 failed, 1 skipped

── cost ──
  Anthropic      $0.0003  (800 tok_in, 80 tok_out, 200 tok_cached)
  OpenAI         $0.0001  (500 tok_in, 50 tok_out)
  TOTAL          $0.0004  (1300 tok_in, 130 tok_out, 200 tok_cached)

[!NOTE] The coverage matrix goes to stdout — paste it into the README between the <!-- testmatrix:start/end --> markers. PRs that add or change provider capabilities must include an updated matrix.

All provider tests use t.Parallel(), so subtests run concurrently within a single go test invocation. To split across CI jobs, use -run:

go test -tags=integration -v -run TestAnthropic ./...
go test -tags=integration -v -run TestOpenAI ./...
go test -tags=integration -v -run TestGemini ./...
go test -tags=integration -v -run TestHuggingFace ./...
go test -tags=integration -v -run TestOllama ./...

Integration test coverage per provider:

Provider Model Chat Stream Reasoning Structured Output From Schema Tools Embeddings STT Moderation Image Gen Img Edit Img Edit Ref Caching
Anthropic claude-haiku-4-5 ✓ ✓ ✓ ✓ ✓ ✓ ✓
OpenAI gpt-4o-mini ✓ ✓ ✓ ✓ ✓
OpenAI gpt-5-mini ✓
OpenAI text-embedding-3-small ✓
OpenAI whisper-1 ✓
OpenAI omni-moderation-latest ✓
OpenAI gpt-image-1 ✓ ✓
Gemini gemini-3.6-flash ✓ ✓ ✓ ✓ ✓
Gemini gemini-2.5-flash-image ✓ ✓ ✓
Hugging Face Kimi-K2-Instruct ✓ ✓ ✓ ✓ ✓
Hugging Face Kimi-K3 ✓ ✓ ✓ ✓ ✓
Ollama qwen3.5:0.8b ✓ ✓
Ollama gpt-oss:20b ✓ ✓ ✓
Ollama gemma4:e4b ✓ ✓

See .env.test.tpl for the full list of env vars.

Guidelines

  • CI runs go vet, golangci-lint, and go test -race — make sure they pass locally before submitting:
go vet ./...
gofmt -l .
golangci-lint run
go test -race ./...

Documentation

Overview

Package ai is the consumer-facing LLM contract — provider-agnostic types (Tool/Message/Response/ToolCall/Part) plus the LLMProvider interface in ai_contract.go. Concrete implementations (Anthropic/OpenAI/Ollama/ HuggingFace) live in sibling files; they satisfy LLMProvider structurally.

Index

Constants

View Source 
const (
	RoleSystem    = "system"
	RoleUser      = "user"
	RoleAssistant = "assistant"
	RoleTool      = "tool"
)

Role constants used in Message.Role.

View Source 
const DefaultHuggingFaceBaseURL = "https://router.huggingface.co/v1"

DefaultHuggingFaceBaseURL is the HF Inference router's OpenAI-compatible endpoint. Override via HF_BASE_URL when targeting a dedicated endpoint or a specific upstream provider route (e.g. https://router.huggingface.co/<provider>/v1).

Variables

This section is empty.

Functions

func CacheSysPromptFromCtx 

func CacheSysPromptFromCtx(ctx context.Context) bool

CacheSysPromptFromCtx reports whether WithCacheSysPrompt was stamped. Adapters call this while building the request to decide whether to mark the sys block as cacheable.

func CountTokens 

func CountTokens(s string) int

CountTokens returns an approximate token count for s using the o200k_base BPE encoding. Exact for GPT-4o / GPT-5 family; within ~15% for Anthropic and HF model families. Empty string returns 0.

Falls back to a len(s)/4 heuristic if the tiktoken encoder fails to initialize, so this never returns an error — a usable rough number is more helpful at call sites than a plumbed error path.

func CountTokensOver 

func CountTokensOver(s string, limit int64) bool

CountTokensOver reports whether counting tokens in s exceeds limit. Short-circuits using the cheap char-based upper bound before invoking the BPE encoder, so callers can gate long strings without paying the full tokenization cost when they're obviously under the limit.

func DebugPromptsEnabled 

func DebugPromptsEnabled() bool

DebugPromptsEnabled reports whether prompt capture is active. Providers check this before building sys/user prompt strings for the debug panel.

func DebugSpanIDFromCtx 

func DebugSpanIDFromCtx(ctx context.Context) string

DebugSpanIDFromCtx returns the span ID on the context, or empty.

func EstimateCostFull 

func EstimateCostFull(model string, in, out, cw, cr int) float64

func GenerateTyped 

func GenerateTyped[T any](ctx context.Context, p LLMProvider, userPrompt, sysPrompt string) (*T, error)

GenerateTyped wraps CreateStructuredOutput with Go generics so callers get back a typed pointer without pre-allocating the output struct.

func MaxInputTokensLLM 

func MaxInputTokensLLM(company, modelName string) (int64, error)

MaxInputTokensLLM returns the advertised maximum input context window in tokens for the given model.

company is the model-family owner / API namespace — one of "anthropic", "openai", or "huggingface". modelName is the exact model id as passed to the provider constructor (for HF that's the full "<org>/<model>" id). Trailing date suffixes like "-20250514" are stripped automatically.

Returns an error if the model is unknown. Callers that need a safe default should handle the error explicitly rather than rely on a fallback — silently returning 0 or a guess would mask config typos.

func MaxTokensFromCtx 

func MaxTokensFromCtx(ctx context.Context, fallback int) int

func MeterCallerIDFromCtx added in v0.0.2 

func MeterCallerIDFromCtx(ctx context.Context) uuid.UUID

func MeterOperationFromCtx 

func MeterOperationFromCtx(ctx context.Context) string

func ModelsForProvider 

func ModelsForProvider(provider string) []string

func PricingTable 

func PricingTable() map[string]modelPricing

func ReasoningEffortFromCtx 

func ReasoningEffortFromCtx(ctx context.Context) string

func SetDebugPromptsEnabled 

func SetDebugPromptsEnabled(enabled bool)

SetDebugPromptsEnabled toggles prompt capture in UsageEvents. Call once at startup based on your dev-mode flag.

func SetImageMeter 

func SetImageMeter(img ImageProvider, hook MeterHook)

SetImageMeter attaches a meter hook to any ImageProvider that satisfies ImageMeterable.

func SetImageModeration 

func SetImageModeration(img ImageProvider, m ModerationProvider)

SetImageModeration attaches a moderation provider to any ImageProvider that satisfies the moderable interface.

func SetLLMMeter 

func SetLLMMeter(llm LLMProvider, hook MeterHook)

SetLLMMeter attaches a meter hook to any LLMProvider that satisfies LLMMeterable.

func SetLLMModeration 

func SetLLMModeration(llm LLMProvider, m ModerationProvider)

SetLLMModeration attaches a moderation provider to any LLMProvider that satisfies LLMModerable.

func SetLogger 

func SetLogger(l *zap.Logger)

func SetOnModerationError 

func SetOnModerationError(fn func(ctx context.Context, err error) error)

SetOnModerationError overrides the default moderation-error behavior. Default (nil): warn and allow. When set, the function's return value determines whether the call proceeds (nil) or is rejected (non-nil error).

func SetUnknownModelHook 

func SetUnknownModelHook(hook func(model string))

func StreamWithChan added in v0.0.2 

func StreamWithChan(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (<-chan StreamChunk, func() (*Response, error))

StreamWithChan starts streaming and returns a channel of chunks plus a function that blocks until done and returns the accumulated response. The channel is closed when streaming completes. If the provider does not implement StreamingProvider, falls back to Chat.

func TransparentBGFromCtx 

func TransparentBGFromCtx(ctx context.Context) bool

func TruncatePromptForDebug 

func TruncatePromptForDebug(s string) string

TruncatePromptForDebug is kept for callers that still reference the old helper name; it now just delegates to capturePromptForDebug — prompts are no longer truncated. Dev mode only carries them at all.

func WithCacheSysPrompt 

func WithCacheSysPrompt(ctx context.Context) context.Context

WithCacheSysPrompt signals that the sys prompt sent in this call is a stable prefix worth provider-side prompt caching. Anthropic's adapter sets cache_control: ephemeral on the sys block; OpenAI/Ollama/HF are no-op (OpenAI auto-caches prompts ≥1024 tokens; Ollama uses KV-cache for matching prefixes at the inference layer; HF depends on backend).

Call this at the call site right before the LLM invocation. Marker is per-call, not global — stamp only when the caller knows the sys prompt is stable across turns (e.g. planner/router system prompts).

func WithDebugSpanID 

func WithDebugSpanID(ctx context.Context, spanID string) context.Context

WithDebugSpanID stamps the current dev-debug span ID on the context so providers can include it on UsageEvents. Caller must be in dev mode; in prod this is a no-op path (the middleware chain never calls StartSpan).

func WithMaxTokens 

func WithMaxTokens(ctx context.Context, n int) context.Context

func WithMeterCallerID added in v0.0.2 

func WithMeterCallerID(ctx context.Context, id uuid.UUID) context.Context

func WithMeterOperation 

func WithMeterOperation(ctx context.Context, op string) context.Context

func WithPromptBlocks 

func WithPromptBlocks(ctx context.Context, contents map[string]string) context.Context

WithPromptBlocks stamps a per-block breakdown on the context so providers can include it on UsageEvents. Call-site passes the raw block strings keyed by block name — this helper computes both Chars (len(s)) and Tokens (CountTokens(s)) per entry.

Tokenizer cost is paid here, but `dev-only` callers and a single tokenize per call mean it's negligible vs the LLM round-trip. If you need to skip the tokenize (e.g. hot path that doesn't care about tokens), call WithPromptBlocksRaw with a pre-built map[string]BlockSize directly.

Empty maps are intentionally not stored — PromptBlocksFromCtx returning nil is the canonical "absent" signal.

func WithPromptBlocksRaw 

func WithPromptBlocksRaw(ctx context.Context, b PromptBlocks) context.Context

WithPromptBlocksRaw is the lower-level variant — caller pre-computes Chars and Tokens per block. Useful when the same block string is reused across calls and you want to cache the token count.

func WithReasoningEffort 

func WithReasoningEffort(ctx context.Context, effort string) context.Context

WithReasoningEffort sets the reasoning effort level ("low", "medium", "high") for providers that support it. OpenAI: maps to reasoning_effort in chat completions. Anthropic: maps to extended thinking with a budget derived from effort level.

func WithTransparentBG 

func WithTransparentBG(ctx context.Context) context.Context

WithTransparentBG signals image providers to produce transparent backgrounds.

Types

type AnthropicProvider 

type AnthropicProvider struct {
	// contains filtered or unexported fields
}

func NewAnthropicProvider 

func NewAnthropicProvider(apiKey, model string) *AnthropicProvider

func (*AnthropicProvider) Chat 

func (p *AnthropicProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*AnthropicProvider) ChatStream added in v0.0.2 

func (p *AnthropicProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*AnthropicProvider) CreateStructuredOutput 

func (p *AnthropicProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*AnthropicProvider) CreateStructuredOutputBreakpointed 

func (p *AnthropicProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM by emitting two system blocks each marked with cache_control: ephemeral. The provider hashes the prefix up through each cache_control marker, so when sysPrompt + stableMid is byte-stable across turns we hit the bp2 entry (everything stable cached); when only sysPrompt is stable (e.g. the candidate list changed) we hit bp1 (sysPrompt cached, stableMid reprocessed). One marker per change-rate tier — see .wiki/context/cache-breakpoints.md.

dynamicTail is the per-call query and rides as the user message — never cached.

func (*AnthropicProvider) CreateStructuredOutputFromSchema 

func (p *AnthropicProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*AnthropicProvider) MaxInputTokens 

func (p *AnthropicProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*AnthropicProvider) Model 

func (p *AnthropicProvider) Model() string

func (*AnthropicProvider) Name 

func (p *AnthropicProvider) Name() string

func (*AnthropicProvider) SetMeter 

func (p *AnthropicProvider) SetMeter(hook MeterHook)

func (*AnthropicProvider) SetModeration 

func (p *AnthropicProvider) SetModeration(m ModerationProvider)

func (*AnthropicProvider) WithMeter 

func (p *AnthropicProvider) WithMeter(hook MeterHook) *AnthropicProvider

func (*AnthropicProvider) WithModeration 

func (p *AnthropicProvider) WithModeration(m ModerationProvider) *AnthropicProvider

type BlockSize 

type BlockSize struct {
	Chars  int `json:"chars"`
	Tokens int `json:"tokens"`
}

BlockSize carries the two attribution signals per prompt block: chars (raw length) and tokens (tiktoken o200k_base count). Tokens are what bills you; chars exist for sanity (a high tokens/chars ratio per block flags tokenizer-hostile content like code, base64, or heavy unicode — surfd as the "density" warning in PromptBlocksBar).

type CacheControl 

type CacheControl struct {
	Type string `json:"type"` // "ephemeral"
}

CacheControl tells a provider to mark this message (or its last content block) as a cache breakpoint. Only Anthropic acts on it today — other providers ignore it silently.

type Embedder 

type Embedder interface {
	EmbedText(ctx context.Context, texts []string) ([][]float32, error)
	EmbedDimensions() int
}

Embedder produces vector embeddings for text. Consumers that need semantic similarity (discovery, memory recall, search ranking) depend on this interface. Not all LLM providers support embeddings — wire a separate provider if the main LLM doesn't (e.g. Anthropic + OpenAI embedding sidecar).

func EmbedderFromLLM 

func EmbedderFromLLM(llm LLMProvider) Embedder

EmbedderFromLLM extracts an Embedder from an LLMProvider if it satisfies the Embedder interface (e.g. OpenAIProvider).

func NewEmbedder 

func NewEmbedder(providerName, apiKey string) Embedder

NewEmbedder creates an Embedder from a provider name + API key. Only OpenAI is supported for now — returns nil for other providers.

type ErrorKind 

type ErrorKind int
const (
	ErrUnknown ErrorKind = iota
	ErrBilling
	ErrAuth
	ErrRateLimit
	ErrModeration
	ErrProviderDown
)

func ClassifyError 

func ClassifyError(err error) (ErrorKind, string)

func (ErrorKind) Retryable 

func (k ErrorKind) Retryable() bool

func (ErrorKind) UserMessage 

func (k ErrorKind) UserMessage() string

type GeminiImageProvider 

type GeminiImageProvider struct {
	// contains filtered or unexported fields
}

GeminiImageProvider implements ImageProvider using Google's Gemini API.

func (*GeminiImageProvider) Edit 

func (p *GeminiImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) EditWithReference 

func (p *GeminiImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*GeminiImageProvider) Generate 

func (p *GeminiImageProvider) Generate(ctx context.Context, prompt, model, _ string) (string, error)

func (*GeminiImageProvider) SetMeter 

func (p *GeminiImageProvider) SetMeter(hook MeterHook)

func (*GeminiImageProvider) SetModeration 

func (p *GeminiImageProvider) SetModeration(m ModerationProvider)

func (*GeminiImageProvider) WithModeration 

func (p *GeminiImageProvider) WithModeration(m ModerationProvider) *GeminiImageProvider

type GeminiProvider added in v0.0.2 

type GeminiProvider struct {
	// contains filtered or unexported fields
}

func NewGeminiProvider added in v0.0.2 

func NewGeminiProvider(ctx context.Context, apiKey, model string) (*GeminiProvider, error)

func (*GeminiProvider) Chat added in v0.0.2 

func (p *GeminiProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*GeminiProvider) ChatStream added in v0.0.2 

func (p *GeminiProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*GeminiProvider) CreateStructuredOutput added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*GeminiProvider) CreateStructuredOutputFromSchema added in v0.0.2 

func (p *GeminiProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*GeminiProvider) MaxInputTokens added in v0.0.2 

func (p *GeminiProvider) MaxInputTokens() (int64, error)

func (*GeminiProvider) Model added in v0.0.2 

func (p *GeminiProvider) Model() string

func (*GeminiProvider) Name added in v0.0.2 

func (p *GeminiProvider) Name() string

func (*GeminiProvider) SetMeter added in v0.0.2 

func (p *GeminiProvider) SetMeter(hook MeterHook)

func (*GeminiProvider) SetModeration added in v0.0.2 

func (p *GeminiProvider) SetModeration(m ModerationProvider)

func (*GeminiProvider) WithMeter added in v0.0.2 

func (p *GeminiProvider) WithMeter(hook MeterHook) *GeminiProvider

func (*GeminiProvider) WithModeration added in v0.0.2 

func (p *GeminiProvider) WithModeration(m ModerationProvider) *GeminiProvider

type HTTPMeterEmitter 

type HTTPMeterEmitter struct {
	// contains filtered or unexported fields
}

HTTPMeterEmitter buffers usage events and POSTs them to a meter service.

func NewHTTPMeterEmitter 

func NewHTTPMeterEmitter(opts HTTPMeterOpts) *HTTPMeterEmitter

func (*HTTPMeterEmitter) Hook 

func (e *HTTPMeterEmitter) Hook() MeterHook

type HTTPMeterOpts 

type HTTPMeterOpts struct {
	Endpoint      string                             // full URL — no suffix appended
	AuthHeader    string                             // sent as Authorization header when non-empty
	BatchSize     int                                // default 32
	FlushInterval time.Duration                      // default 2s
	Marshal       func([]UsageEvent) ([]byte, error) // default json.Marshal
	ContentType   string                             // default "application/json"
	OnError       func(error)                        // default logger.Warn
}

HTTPMeterOpts configures an HTTPMeterEmitter.

type HuggingFaceProvider 

type HuggingFaceProvider struct {
	// contains filtered or unexported fields
}

HuggingFaceProvider speaks the HF Inference Router, which exposes an OpenAI-compatible Chat Completions API. Models are addressed by their canonical HF id (e.g. "meta-llama/Llama-3.3-70B-Instruct").

func NewHuggingFaceProvider 

func NewHuggingFaceProvider(apiKey, model, baseURL string) *HuggingFaceProvider

NewHuggingFaceProvider builds a provider against the HF router. baseURL may be empty to use DefaultHuggingFaceBaseURL.

func (*HuggingFaceProvider) Chat 

func (p *HuggingFaceProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*HuggingFaceProvider) ChatStream added in v0.0.2 

func (p *HuggingFaceProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*HuggingFaceProvider) CreateStructuredOutput 

func (p *HuggingFaceProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*HuggingFaceProvider) CreateStructuredOutputFromSchema 

func (p *HuggingFaceProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*HuggingFaceProvider) MaxInputTokens 

func (p *HuggingFaceProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*HuggingFaceProvider) Model 

func (p *HuggingFaceProvider) Model() string

func (*HuggingFaceProvider) Name 

func (p *HuggingFaceProvider) Name() string

func (*HuggingFaceProvider) RawClient 

func (p *HuggingFaceProvider) RawClient() *openai.Client

func (*HuggingFaceProvider) SetMeter 

func (p *HuggingFaceProvider) SetMeter(hook MeterHook)

func (*HuggingFaceProvider) SetModeration 

func (p *HuggingFaceProvider) SetModeration(m ModerationProvider)

func (*HuggingFaceProvider) WithMeter 

func (p *HuggingFaceProvider) WithMeter(hook MeterHook) *HuggingFaceProvider

func (*HuggingFaceProvider) WithModeration 

func (p *HuggingFaceProvider) WithModeration(m ModerationProvider) *HuggingFaceProvider

type ImageMeterable 

type ImageMeterable interface {
	SetMeter(MeterHook)
}

ImageMeterable is implemented by image providers that accept a meter hook.

type ImagePart 

type ImagePart struct {
	MediaType string // e.g. "image/png"
	Data      string // base64-encoded
}

ImagePart carries base64-encoded image data.

type ImageProvider 

type ImageProvider interface {
	Generate(ctx context.Context, prompt, model, size string) (string, error)                                 // base64
	Edit(ctx context.Context, image []byte, editPrompt string) (string, error)                                // base64
	EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error) // base64
}

ImageProvider generates and edits images via an AI model.

func NewImageProvider 

func NewImageProvider(ctx context.Context, providerName, apiKey, model string) (ImageProvider, error)

NewImageProvider creates an ImageProvider from a provider name ("openai" or "gemini").

type LLMMeterable 

type LLMMeterable interface {
	SetMeter(MeterHook)
}

LLMMeterable is implemented by providers that accept a meter hook. SetLLMMeter uses this instead of a type-switch so new providers work without updating the switch.

type LLMModerable 

type LLMModerable interface {
	SetModeration(ModerationProvider)
}

LLMModerable is implemented by providers that accept a moderation provider.

type LLMProvider 

type LLMProvider interface {
	Name() string
	Model() string
	CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error
	CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)
	Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)
}

LLMProvider is the consumer-facing contract for any LLM backend.

func NewLLMProvider 

func NewLLMProvider(providerName, apiKey, model string, opts ...ProviderOption) LLMProvider

NewLLMProvider creates an LLMProvider from a provider name ("openai", "anthropic", "gemini", "ollama", or "huggingface"). For ollama, apiKey is the base URL (e.g. "http://ollama:11434/v1"). For huggingface, apiKey is the HF token; base URL is read from HF_BASE_URL (defaults to DefaultHuggingFaceBaseURL). For gemini, apiKey is the Google AI API key.

type Message 

type Message struct {
	Role         string        `json:"role"`
	Content      string        `json:"content,omitempty"`
	Parts        []Part        `json:"-"`
	ToolCalls    []ToolCall    `json:"tool_calls,omitempty"`
	ToolCallID   string        `json:"tool_call_id,omitempty"`
	CacheControl *CacheControl `json:"cache_control,omitempty"`
}

Message is a provider-agnostic chat message.

func (Message) MarshalJSON 

func (m Message) MarshalJSON() ([]byte, error)

func (*Message) UnmarshalJSON 

func (m *Message) UnmarshalJSON(data []byte) error

type MeterHook 

type MeterHook func(UsageEvent)

MeterHook is called after each LLM/image call with usage data.

type Middleware 

type Middleware func(next ModelFunc) ModelFunc

Middleware wraps a ModelFunc with cross-cutting behaviour.

func Chain 

func Chain(mws ...Middleware) Middleware

Chain composes middleware so the first in the list is outermost.

type ModelFunc 

type ModelFunc func(ctx context.Context, msgs []Message, tools []Tool) (*Response, error)

ModelFunc is the signature of an LLM Chat call, abstracted from any concrete provider.

type ModerationError 

type ModerationError struct {
	Categories map[string]bool
}

ModerationError is returned when a prompt is blocked by moderation.

func (*ModerationError) Error 

func (e *ModerationError) Error() string

type ModerationProvider 

type ModerationProvider interface {
	Check(ctx context.Context, input string) (*ModerationResult, error)
}

ModerationProvider checks user-supplied text for policy violations.

func NewModerationProvider 

func NewModerationProvider(providerName, apiKey string) ModerationProvider

NewModerationProvider creates a ModerationProvider from a provider name.

type ModerationResult 

type ModerationResult struct {
	Flagged    bool            `json:"flagged"`
	Categories map[string]bool `json:"categories,omitempty"`
}

ModerationResult holds the outcome of a moderation check.

type OllamaProvider 

type OllamaProvider struct {
	// contains filtered or unexported fields
}

OllamaProvider wraps the OpenAI-compatible API exposed by Ollama.

func NewOllamaProvider 

func NewOllamaProvider(baseURL, model string) *OllamaProvider

NewOllamaProvider creates a provider pointing at an Ollama instance. baseURL is the Ollama server, e.g. "http://ollama:11434/v1".

func (*OllamaProvider) Chat 

func (p *OllamaProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OllamaProvider) ChatStream added in v0.0.2 

func (p *OllamaProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OllamaProvider) CreateStructuredOutput 

func (p *OllamaProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OllamaProvider) CreateStructuredOutputFromSchema 

func (p *OllamaProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OllamaProvider) Model 

func (p *OllamaProvider) Model() string

func (*OllamaProvider) Name 

func (p *OllamaProvider) Name() string

func (*OllamaProvider) SetMeter 

func (p *OllamaProvider) SetMeter(hook MeterHook)

func (*OllamaProvider) SetModeration 

func (p *OllamaProvider) SetModeration(m ModerationProvider)

func (*OllamaProvider) WithMeter 

func (p *OllamaProvider) WithMeter(hook MeterHook) *OllamaProvider

func (*OllamaProvider) WithModeration 

func (p *OllamaProvider) WithModeration(m ModerationProvider) *OllamaProvider

type OpenAIImageProvider 

type OpenAIImageProvider struct {
	// contains filtered or unexported fields
}

OpenAIImageProvider implements ImageProvider using the OpenAI images API.

func (*OpenAIImageProvider) Edit 

func (p *OpenAIImageProvider) Edit(ctx context.Context, image []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) EditWithReference 

func (p *OpenAIImageProvider) EditWithReference(ctx context.Context, image []byte, reference []byte, editPrompt string) (string, error)

func (*OpenAIImageProvider) Generate 

func (p *OpenAIImageProvider) Generate(ctx context.Context, prompt, model, size string) (string, error)

func (*OpenAIImageProvider) SetMeter 

func (p *OpenAIImageProvider) SetMeter(hook MeterHook)

func (*OpenAIImageProvider) SetModeration 

func (p *OpenAIImageProvider) SetModeration(m ModerationProvider)

func (*OpenAIImageProvider) WithModeration 

func (p *OpenAIImageProvider) WithModeration(m ModerationProvider) *OpenAIImageProvider

type OpenAIModerationProvider 

type OpenAIModerationProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIModerationProvider 

func NewOpenAIModerationProvider(apiKey string) *OpenAIModerationProvider

func (*OpenAIModerationProvider) Check 

func (p *OpenAIModerationProvider) Check(ctx context.Context, input string) (*ModerationResult, error)

type OpenAIProvider 

type OpenAIProvider struct {
	// contains filtered or unexported fields
}

func NewOpenAIProvider 

func NewOpenAIProvider(apiKey, model string) *OpenAIProvider

func (*OpenAIProvider) Chat 

func (p *OpenAIProvider) Chat(ctx context.Context, messages []Message, tools []Tool) (*Response, error)

func (*OpenAIProvider) ChatStream added in v0.0.2 

func (p *OpenAIProvider) ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)

func (*OpenAIProvider) CreateStructuredOutput 

func (p *OpenAIProvider) CreateStructuredOutput(ctx context.Context, userPrompt, sysPrompt string, structuredOutput any) error

func (*OpenAIProvider) CreateStructuredOutputBreakpointed 

func (p *OpenAIProvider) CreateStructuredOutputBreakpointed(
	ctx context.Context,
	sysPrompt, stableMid, dynamicTail string,
	structuredOutput any,
) error

CreateStructuredOutputBreakpointed satisfies router.CachedStructuredLLM. OpenAI auto-prefix-caches any stable prefix ≥1024 tokens, so the breakpointing surf is implemented by concatenating sysPrompt and stableMid into the single system message. No explicit markers needed — the byte-stable prefix is the cache key.

dynamicTail rides as the user message (uncached).

func (*OpenAIProvider) CreateStructuredOutputFromSchema 

func (p *OpenAIProvider) CreateStructuredOutputFromSchema(ctx context.Context, userPrompt, sysPrompt string, schema json.RawMessage) (map[string]any, error)

func (*OpenAIProvider) EmbedDimensions 

func (p *OpenAIProvider) EmbedDimensions() int

func (*OpenAIProvider) EmbedText 

func (p *OpenAIProvider) EmbedText(ctx context.Context, texts []string) ([][]float32, error)

func (*OpenAIProvider) MaxInputTokens 

func (p *OpenAIProvider) MaxInputTokens() (int64, error)

MaxInputTokens returns the advertised input context window for p.model.

func (*OpenAIProvider) Model 

func (p *OpenAIProvider) Model() string

func (*OpenAIProvider) Name 

func (p *OpenAIProvider) Name() string

func (*OpenAIProvider) RawClient 

func (p *OpenAIProvider) RawClient() *openai.Client

func (*OpenAIProvider) SetMeter 

func (p *OpenAIProvider) SetMeter(hook MeterHook)

func (*OpenAIProvider) SetModeration 

func (p *OpenAIProvider) SetModeration(m ModerationProvider)

func (*OpenAIProvider) WithMeter 

func (p *OpenAIProvider) WithMeter(hook MeterHook) *OpenAIProvider

func (*OpenAIProvider) WithModeration 

func (p *OpenAIProvider) WithModeration(m ModerationProvider) *OpenAIProvider

type OpenAISTTProvider 

type OpenAISTTProvider struct {
	// contains filtered or unexported fields
}

func (*OpenAISTTProvider) Transcribe 

func (p *OpenAISTTProvider) Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)

type Part 

type Part interface {
	// contains filtered or unexported methods
}

Part is a single element of a multimodal message. Sealed — only TextPart and ImagePart satisfy this interface.

type PromptBlocks 

type PromptBlocks map[string]BlockSize

PromptBlocks is a per-call breakdown of the prompt into named blocks. Each entry carries chars + tokens. The meter hook stamps this onto UsageEvent.Metadata["blocks"] so downstream recorders can see the breakdown without needing each provider to crack the prompt apart.

func PromptBlocksFromCtx 

func PromptBlocksFromCtx(ctx context.Context) PromptBlocks

PromptBlocksFromCtx returns the prompt-block breakdown stamped on ctx, or nil if none. Providers call this inside emitUsage to enrich the UsageEvent.Metadata.

type ProviderOption 

type ProviderOption func(*providerConfig)

ProviderOption configures an LLMProvider created by NewLLMProvider.

func WithMeterOption 

func WithMeterOption(hook MeterHook) ProviderOption

WithMeterOption returns a ProviderOption that attaches a meter hook.

func WithModerationOption 

func WithModerationOption(m ModerationProvider) ProviderOption

WithModerationOption returns a ProviderOption that attaches a moderation provider.

type Response 

type Response struct {
	Content   string     `json:"content,omitempty"`
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`
}

Response is the provider-agnostic result of a Chat call.

type STTProvider 

type STTProvider interface {
	Transcribe(ctx context.Context, audio io.Reader, filename string) (string, error)
}

STTProvider transcribes audio into text.

func NewSTTProvider 

func NewSTTProvider(providerName, apiKey, model string) STTProvider

NewSTTProvider creates an STTProvider from a provider name + API key + model. Only OpenAI is supported for now — returns nil for other providers.

type StreamChunk added in v0.0.2 

type StreamChunk struct {
	Text string
}

StreamChunk is a single piece of a streaming response.

type StreamResult added in v0.0.2 

type StreamResult struct {
	// contains filtered or unexported fields
}

StreamResult holds the final accumulated response from a stream.

func Stream added in v0.0.2 

func Stream(ctx context.Context, p LLMProvider, msgs []Message, tools []Tool) (iter.Seq2[StreamChunk, error], *StreamResult)

Stream returns an iter.Seq2 that yields chunks as they arrive. The returned StreamResult provides the accumulated *Response after iteration. If the provider does not implement StreamingProvider, falls back to Chat.

func (*StreamResult) Response added in v0.0.2 

func (r *StreamResult) Response() (*Response, error)

Response blocks until streaming completes and returns the accumulated response.

type StreamingProvider added in v0.0.2 

type StreamingProvider interface {
	ChatStream(ctx context.Context, messages []Message, tools []Tool, cb func(StreamChunk) error) (*Response, error)
}

StreamingProvider extends LLMProvider with streaming chat support.

type TextPart 

type TextPart struct {
	Text string
}

TextPart carries inline text.

type Tool 

type Tool struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema object
}

Tool defines a provider-agnostic tool the model can call.

type ToolCall 

type ToolCall struct {
	ID        string          `json:"id"`
	Name      string          `json:"name"`
	Arguments json.RawMessage `json:"arguments"`
}

ToolCall represents the model requesting a tool invocation.

type UsageEvent 

type UsageEvent struct {
	CallerID         uuid.UUID `json:"caller_id"`
	Provider         string    `json:"provider"`
	Model            string    `json:"model"`
	Operation        string    `json:"operation"`
	InputTokens      int       `json:"input_tokens"`
	OutputTokens     int       `json:"output_tokens"`
	TotalTokens      int       `json:"total_tokens"`
	EstimatedCostUSD float64   `json:"estimated_cost_usd"`
	// CacheCreationInputTokens counts tokens written to the provider's
	// prompt cache on this call (Anthropic only, populated when cache
	// writes occur; priced at ~1.25x base). Zero on cache hit or when
	// caching isn't active.
	CacheCreationInputTokens int `json:"cache_creation_input_tokens,omitempty"`
	// CacheReadInputTokens counts tokens served from the provider's
	// prompt cache (Anthropic only; priced at ~0.1x base). Nonzero means
	// WithCacheSysPrompt is actively paying off.
	CacheReadInputTokens int `json:"cache_read_input_tokens,omitempty"`
	// SystemPrompt/UserPrompt are captured for dev debug display only — empty
	// in production (see capturePromptForDebug).
	SystemPrompt string `json:"system_prompt,omitempty"`
	UserPrompt   string `json:"user_prompt,omitempty"`
	// DebugSpanID is the ID of the PromptMiddlewareChain span that was active
	// when this LLM call fired — used by the debug panel meter hook to attach
	// the llm_call event to the correct span in the tree. Empty in prod.
	DebugSpanID string         `json:"debug_span_id,omitempty"`
	Metadata    map[string]any `json:"metadata,omitempty"`
}

UsageEvent represents a single LLM/image-gen usage event for metering.

Source Files

View all Source files

Directories

Path Synopsis
cmd
testmatrix command
testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.
 testmatrix reads `go test -json` output from stdin, prints live progress to stderr as results arrive, then prints the final matrix to stdout.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.