llm

README

pi-llm-go

Provider-agnostic LLM abstraction for Go, extracted from the Pi agent framework.

Features

• Single interface: LLMProvider abstracts OpenAI-compatible and Gemini endpoints.
• Normalized streaming: EventStream with typed LLMEvent variants.
• Portable thinking: ThinkingLevel enum with per-provider mapping.
• First-class mock: mock.MockProvider with fluent builder for tests.
• No agent opinions: Just the LLM call layer; bring your own loop.

Providers

• openai-compat — OpenAI, Fireworks, Ollama, vLLM, llama.cpp server, LM Studio
• gemini — Google Gemini via google.golang.org/genai

Install

go get github.com/dev-resolute/resolute-llm-go

Usage

import (
    "github.com/dev-resolute/resolute-llm-go"
    "github.com/dev-resolute/resolute-llm-go/openai-compat"
)

provider, _ := openaicompat.New(openaicompat.Config{
    BaseURL: "https://api.openai.com/v1",
    APIKey:  os.Getenv("OPENAI_API_KEY"),
})

stream := provider.Stream(ctx, llm.LLMRequest{
    Model:    "gpt-4o",
    Messages: []llm.Message{{Role: "user", Content: llm.TextContent{Text: "Hello"}}},
})

for ev := range stream.Events {
    // type-switch on llm.LLMEvent
}
result := <-stream.Done

Testing

Unit tests pass without secrets:

go test ./...

Integration tests (require API keys):

go test -tags=integration ./...

License

MIT

Documentation

Overview

Package llm provides a provider-agnostic abstraction over LLM streaming APIs.

Index

• Constants
• Variables
• func AsContextOverflow(err error) error
• func ResolveStrictSampling(tool ToolDef, supported bool) (bool, error)
• func Retry(ctx context.Context, policy RetryPolicy, provider, model string, ...) error
• type ConstrainedSampling
• type Content
• type EventStream
  • func NewEventStream(events <-chan LLMEvent, done <-chan StreamResult) EventStream
  • func Run(ctx context.Context, req LLMRequest, produce ProduceFn) EventStream
• type GeminiHints
• type ImageContent
• type LLMErrorEvent
• type LLMEvent
• type LLMProvider
• type LLMRequest
• type LLMRetryEvent
• type Message
• type MessageEndEvent
• type OpenAIHints
• type ProduceFn
• type ProviderCapabilities
• type ProviderHints
• type RetryPolicy
• type StopReason
• type StreamResult
• type StrictMode
• type TextContent
• type TextDeltaEvent
• type ThinkingContent
• type ThinkingDeltaEvent
• type ThinkingLevel
• type ToolCallContent
• type ToolCallEndEvent
• type ToolCallStartEvent
• type ToolDef
• type ToolResultContent
• type TransientError
  • func (e *TransientError) Error() string
  • func (e *TransientError) Unwrap() error
• type TransportPreference
  • func (t TransportPreference) String() string
• type UsageEvent

Constants

const (
	DefaultMaxRetries    = 3
	DefaultMaxRetryDelay = 60 * time.Second
)

Defaults for RetryPolicy zero values.

const DefaultEventBufferSize = 16

DefaultEventBufferSize is the default buffer size for EventStream.Events.

Variables

var (
	ErrProviderFatal        = errors.New("provider fatal error")
	ErrInvalidModel         = errors.New("invalid model")
	ErrUnsupportedFeature   = errors.New("unsupported feature")
	ErrMalformedResponse    = errors.New("malformed provider response")
	ErrTransportUnsupported = errors.New("transport not supported by provider")
	// ErrProviderStop marks a provider-terminated message: the stream ended
	// with a terminal stop/finish reason that has no portable mapping (Gemini
	// SAFETY/RECITATION/..., OpenAI content_filter/network_error, or a
	// genuinely unknown reason). Fatal — the message did not complete.
	ErrProviderStop    = errors.New("provider stop")
	ErrContextOverflow = errors.New("context length exceeded")
)

Sentinel errors for pi-llm-go.

Functions

func AsContextOverflow

func AsContextOverflow(err error) error

AsContextOverflow classifies provider errors: when err's message reports the model's maximum context length was exceeded, it returns an error wrapping ErrContextOverflow so callers can react via errors.Is (Compact + retry, truncate, or switch models). Other errors and nil pass through unchanged.

func ResolveStrictSampling

func ResolveStrictSampling(tool ToolDef, supported bool) (bool, error)

ResolveStrictSampling reports whether tool should be sent with provider-side strict JSON-schema enforcement. Ports upstream resolveJsonSchemaStrictSampling (pi v0.82.0 constrained-sampling.ts): nil config → (false, nil); prefer+supported → (true, nil); prefer+unsupported → (false, nil); require+unsupported → error with the upstream message. A non-nil config with Strict outside {prefer, require} (including "") errors naming the invalid value.

func Retry

func Retry(ctx context.Context, policy RetryPolicy, provider, model string, emit func(LLMEvent) error, op func(ctx context.Context) error) error

Retry runs op, retrying TransientError failures per policy (upstream retryProviderRequest). It emits an LLMRetryEvent before each retry wait.

The retried boundary is the stream-open phase: op must be idempotent and must not emit events (a successful open is followed by unretried streaming, so content is never duplicated). Classification is the provider's: anything not wrapped in TransientError passes through unretried.

Failure modes: a server-requested wait above policy.MaxRetryDelay fails immediately (naming both delays, upstream's message shape); exhausted retries return the last TransientError (so the provider still classifies the failure as transient); context cancellation stops the ladder with the ctx cause.

Types

type ConstrainedSampling

type ConstrainedSampling struct {
	Strict StrictMode
}

ConstrainedSampling opts a tool into provider-side constrained sampling. Nil disables it (upstream's omitted/false).

type Content

type Content interface {
	// contains filtered or unexported methods
}

Content is a sealed interface for the content of a Message.

type EventStream

type EventStream struct {
	Events <-chan LLMEvent
	Done   <-chan StreamResult
}

EventStream is the shared return shape for any streaming LLM call. Events delivers a stream of typed events and closes when the run finishes. Done delivers exactly one terminal StreamResult.

func NewEventStream

func NewEventStream(events <-chan LLMEvent, done <-chan StreamResult) EventStream

NewEventStream creates an EventStream from the provided channels. Providers use this to package their internal goroutine outputs.

func Run

func Run(ctx context.Context, req LLMRequest, produce ProduceFn) EventStream

Run executes a streaming LLM call using the given producer callback. It allocates channels, manages goroutine lifetime, and enforces the EventStream contract: Events closes when the stream ends, Done delivers exactly one StreamResult with Messages = req.Messages ++ produced.

type GeminiHints

type GeminiHints struct {
	ThinkingBudget int
}

GeminiHints carries provider-specific overrides for the Gemini provider.

type ImageContent

type ImageContent struct {
	Data     []byte
	MimeType string // image/jpeg, image/png, image/gif, image/webp
}

ImageContent carries an inline image. Data is raw bytes; adapters base64-encode on the wire. encoding/json marshals []byte as base64, so JSON transcripts store compactly for free. Valid in user messages (attachments) and on ToolResultContent.Images. A user turn with text and an image is two adjacent user messages; no multi-content message shape exists.

type LLMErrorEvent

type LLMErrorEvent struct {
	Error     error
	Transient bool
}

LLMErrorEvent signals an error from the provider.

type LLMEvent

type LLMEvent interface {
	// contains filtered or unexported methods
}

LLMEvent is a sealed interface for every event that flows on EventStream.Events. Concrete variants are defined below; discrimination is via type switch.

type LLMProvider

type LLMProvider interface {
	// Name returns the provider's short identifier, used in model references
	// like "<name>/<model-id>".
	Name() string

	// Capabilities returns the feature set for the given model.
	Capabilities(model string) ProviderCapabilities

	// Stream initiates a streaming LLM call. The returned EventStream delivers
	// typed events on Events and a single terminal result on Done.
	// Cancellation is honored via ctx; the Events channel closes when the
	// stream completes or ctx is cancelled.
	Stream(ctx context.Context, req LLMRequest) EventStream
}

LLMProvider is the single interface implemented by every concrete provider. It abstracts differences between LLM wire protocols without imposing agent-loop opinions.

type LLMRequest

type LLMRequest struct {
	Model    string
	Messages []Message
	Tools    []ToolDef
	Thinking ThinkingLevel
	// SessionID optionally identifies the conversation this call belongs to.
	// The OpenAI-compatible adapter sends it as prompt-cache affinity headers
	// (session_id, x-client-request-id, x-session-affinity) and as the
	// prompt_cache_key body param, so repeated calls route to the same replica
	// and maximize prompt-cache hits. The Gemini adapter ignores it. Empty means
	// "no session hint".
	SessionID string
	// ThinkingBudgets optionally overrides the per-level token budget for the
	// active Thinking level. Providers whose reasoning control is token-based
	// (Gemini's thinking_budget) apply it; providers whose control is categorical
	// (OpenAI-compatible reasoning_effort) ignore it. Nil means "use provider
	// defaults". ProviderHints, when set, takes precedence over this map.
	ThinkingBudgets map[ThinkingLevel]int
	// Transport is the preferred stream transport. Providers that support only
	// HTTP/SSE honor TransportAuto and TransportSSE; TransportWebSocket returns
	// ErrTransportUnsupported until a websocket-capable provider exists.
	Transport       TransportPreference
	ProviderHints   ProviderHints
	Retry           RetryPolicy
	Headers         map[string]string
	OnBeforeRequest func(headers map[string]string) error
	OnAfterResponse func(statusCode int, headers map[string]string)
}

LLMRequest carries all inputs for a single streaming LLM call.

type LLMRetryEvent

type LLMRetryEvent struct {
	Provider   string
	Model      string
	Attempt    int
	NextDelay  time.Duration
	Reason     string
	ServerHint bool
}

LLMRetryEvent signals that a retry attempt is being made.

type Message

type Message struct {
	Role    string
	Content Content
}

Message is the LLM-side unit of transcript content. It is provider-shaped, not user-extensible; distinct from any agent-side transcript Message.

type MessageEndEvent

type MessageEndEvent struct {
	StopReason StopReason
}

MessageEndEvent signals the end of the assistant's message. It is always the last non-error event in a successful stream; a stream that terminates with a provider stop (or missing finish reason) emits a fatal LLMErrorEvent instead.

type OpenAIHints

type OpenAIHints struct {
	ReasoningEffort string
}

OpenAIHints carries provider-specific overrides for the OpenAI-compatible adapter.

type ProduceFn

type ProduceFn func(ctx context.Context, req LLMRequest, emit func(LLMEvent) error, headers map[string]string, setResponseMeta func(status int, respHeaders map[string]string)) ([]Message, error)

ProduceFn is the callback signature for provider-specific streaming logic. The producer receives an emit function that handles ctx cancellation. It returns the new messages produced during the stream and any error. The headers map is the merged result of Config.Headers + LLMRequest.Headers + any mutations made by OnBeforeRequest hooks. setResponseMeta lets the producer report HTTP status + response headers for the OnAfterResponse hook. It may be called at most once.

type ProviderCapabilities

type ProviderCapabilities struct {
	Streaming         bool
	ToolCalling       bool
	ParallelToolCalls bool
	Thinking          bool
	PromptCaching     bool
	Vision            bool
}

ProviderCapabilities describes the feature set of a specific model.

type ProviderHints

type ProviderHints struct {
	OpenAI *OpenAIHints
	Gemini *GeminiHints
}

ProviderHints is a typed escape hatch for provider-specific config. Only the field matching the active provider is consulted; others are ignored.

type RetryPolicy

type RetryPolicy struct {
	// MaxRetries bounds retry attempts after the initial call. 0 resolves to
	// DefaultMaxRetries; negative disables retries.
	MaxRetries int
	// MaxRetryDelay caps a server-requested (retry-after) wait: a hint above
	// the cap fails immediately (upstream rule). 0 resolves to
	// DefaultMaxRetryDelay; negative disables the cap. The exponential backoff
	// is never capped.
	MaxRetryDelay time.Duration
}

RetryPolicy configures the provider retry ladder (Retry). The zero value resolves to the documented defaults (DefaultMaxRetries, DefaultMaxRetryDelay) — matching upstream's agent policy — and negative fields disable: MaxRetries < 0 runs the operation once; MaxRetryDelay < 0 lifts the cap on server-requested waits.

type StopReason

type StopReason string

StopReason describes why the assistant message ended. Mirrors upstream's StopReason set (types.ts) minus error/aborted/pending/deferred: terminal provider stops are fatal errors, not stop reasons (see below), upstream's pending is an interim partial-message state this event shape does not expose, and deferred responses are out of scope.

Since 0.13.0 (upstream #7272 parity): native finish reasons without a portable mapping (Gemini SAFETY/RECITATION/..., OpenAI content_filter/network_error, or a genuinely unknown reason) surface as a fatal LLMErrorEvent wrapping ErrProviderStop — never as a successful StopReasonUnknown. A stream that ends without any finish reason is a protocol error wrapping ErrMalformedResponse, unless the provider is known to omit it (openai-compat Compat.SupportsFinishReason), in which case the reason is inferred from content.

const (
	// StopReasonUnknown is the zero value. Providers no longer emit it on a
	// successful MessageEndEvent; it remains for unmapped custom-provider use.
	StopReasonUnknown StopReason = ""
	StopReasonStop    StopReason = "stop"
	StopReasonLength  StopReason = "length"
	StopReasonToolUse StopReason = "toolUse"
)

type StreamResult

type StreamResult struct {
	Messages []Message
	Err      error
}

StreamResult is the single terminal value delivered on EventStream.Done.

type StrictMode

type StrictMode string

StrictMode selects how strongly a tool requests JSON-schema-enforced calls.

const (
	StrictPrefer  StrictMode = "prefer"  // use when supported, silently fall back
	StrictRequire StrictMode = "require" // error when the provider can't honor it
)

type TextContent

type TextContent struct {
	Text string
	// ThoughtSignature is an opaque provider token bound to this text part
	// (Gemini thought signatures). Gemini can attach the signature to a part
	// whose visible text is empty; such parts must be replayed verbatim or
	// the reasoning chain breaks (upstream #7362). Callers replaying history
	// must carry it back verbatim; empty for providers without one.
	ThoughtSignature []byte
}

TextContent carries plain text.

type TextDeltaEvent

type TextDeltaEvent struct {
	Delta string
	// ThoughtSignature is an opaque provider token bound to the text part this
	// delta belongs to (Gemini thought signatures). It typically appears on only
	// one delta of a part — possibly one with an empty Delta — so consumers
	// assembling a message retain the last non-empty value (upstream
	// retainThoughtSignature); empty for providers without one.
	ThoughtSignature []byte
}

TextDeltaEvent carries a fragment of text output from the LLM.

type ThinkingContent

type ThinkingContent struct {
	Text string
	// ThoughtSignature is an opaque provider token bound to this thinking
	// part (Gemini thought signatures). Callers replaying history must carry
	// it back verbatim; empty for providers without one.
	ThoughtSignature []byte
}

ThinkingContent carries reasoning/thinking content from the LLM.

type ThinkingDeltaEvent

type ThinkingDeltaEvent struct {
	Delta string
	// ThoughtSignature behaves as on TextDeltaEvent, for thinking parts.
	ThoughtSignature []byte
}

ThinkingDeltaEvent carries a fragment of thinking/reasoning content.

type ThinkingLevel

type ThinkingLevel int

ThinkingLevel is a portable abstraction over provider-specific reasoning controls.

const (
	ThinkingOff ThinkingLevel = iota
	ThinkingMinimal
	ThinkingLow
	ThinkingMedium
	ThinkingHigh
	ThinkingXhigh
	ThinkingMax
)

type ToolCallContent

type ToolCallContent struct {
	CallID   string
	ToolName string
	Args     json.RawMessage
	// ThoughtSignature is an opaque provider token bound to this tool call
	// (Gemini 3 thought signatures). Callers replaying history must carry it
	// back verbatim; empty for providers without one.
	ThoughtSignature []byte
}

ToolCallContent carries a tool invocation from the LLM.

type ToolCallEndEvent

type ToolCallEndEvent struct {
	CallID string
	// ToolName, Args, and ThoughtSignature carry the finalized call: for
	// providers that stream arguments incrementally (openai-compat), this
	// event — not ToolCallStartEvent — is where complete arguments appear.
	ToolName         string
	Args             json.RawMessage
	ThoughtSignature []byte
}

ToolCallEndEvent signals the end of a tool call block.

type ToolCallStartEvent

type ToolCallStartEvent struct {
	CallID   string
	ToolName string
	Args     json.RawMessage
	// ThoughtSignature is an opaque provider token bound to this tool call
	// (Gemini 3 thought signatures). Consumers persisting the transcript must
	// carry it onto the replayed ToolCallContent; empty for providers without one.
	ThoughtSignature []byte
}

ToolCallStartEvent signals the beginning of a tool call in the stream.

type ToolDef

type ToolDef struct {
	Name                string
	Description         string
	Schema              json.RawMessage
	ConstrainedSampling *ConstrainedSampling
}

ToolDef is the LLM-visible tool specification.

type ToolResultContent

type ToolResultContent struct {
	CallID   string
	ToolName string
	Content  string
	// Images carries optional image parts of the tool result (e.g. the
	// read tool returning a screenshot). Nil for text-only results.
	Images  []ImageContent
	Data    json.RawMessage
	IsError bool
}

ToolResultContent carries the result of a tool execution back to the LLM.

type TransientError

type TransientError struct {
	Err error
	// RetryAfter is the server-requested wait (retry-after / retry-after-ms
	// response headers). 0 when the server gave no hint — the ladder then uses
	// the exponential backoff.
	RetryAfter time.Duration
}

TransientError wraps a stream-open failure the provider classifies as retryable, optionally carrying the server's requested wait (retry-after). Providers return it from their Retry op; Retry retries it per policy.

func (*TransientError) Error

func (e *TransientError) Error() string

func (*TransientError) Unwrap

func (e *TransientError) Unwrap() error

type TransportPreference

type TransportPreference int

TransportPreference is a portable preference for the stream transport a provider uses. Providers that support only one transport honor TransportAuto and TransportSSE by using HTTP/SSE; a provider that does not implement the requested transport returns ErrTransportUnsupported rather than silently falling back, so callers learn the transport is unavailable.

const (
	TransportAuto TransportPreference = iota
	TransportSSE
	TransportWebSocket
)

func (TransportPreference) String

func (t TransportPreference) String() string

String returns the wire-style name of the transport preference.

type UsageEvent

type UsageEvent struct {
	InputTokens  int
	OutputTokens int
}

UsageEvent carries token-usage metadata from the provider.