llm

README

pi-llm-go

Provider-agnostic LLM abstraction for Go, extracted from the Pi agent framework.

Features

• Single interface: LLMProvider abstracts OpenAI-compatible and Gemini endpoints.
• Normalized streaming: EventStream with typed LLMEvent variants.
• Portable thinking: ThinkingLevel enum with per-provider mapping.
• First-class mock: mock.MockProvider with fluent builder for tests.
• No agent opinions: Just the LLM call layer; bring your own loop.

Providers

• openai-compat — OpenAI, Fireworks, Ollama, vLLM, llama.cpp server, LM Studio
• gemini — Google Gemini via google.golang.org/genai

Install

go get github.com/dev-resolute/resolute-llm-go

Usage

import (
    "github.com/dev-resolute/resolute-llm-go"
    "github.com/dev-resolute/resolute-llm-go/openai-compat"
)

provider, _ := openaicompat.New(openaicompat.Config{
    BaseURL: "https://api.openai.com/v1",
    APIKey:  os.Getenv("OPENAI_API_KEY"),
})

stream := provider.Stream(ctx, llm.LLMRequest{
    Model:    "gpt-4o",
    Messages: []llm.Message{{Role: "user", Content: llm.TextContent{Text: "Hello"}}},
})

for ev := range stream.Events {
    // type-switch on llm.LLMEvent
}
result := <-stream.Done

Testing

Unit tests pass without secrets:

go test ./...

Integration tests (require API keys):

go test -tags=integration ./...

License

MIT

Documentation

Overview

Package llm provides a provider-agnostic abstraction over LLM streaming APIs.

Index

• Constants
• Variables
• func AsContextOverflow(err error) error
• type Content
• type EventStream
  • func NewEventStream(events <-chan LLMEvent, done <-chan StreamResult) EventStream
  • func Run(ctx context.Context, req LLMRequest, produce ProduceFn) EventStream
• type GeminiHints
• type ImageContent
• type LLMErrorEvent
• type LLMEvent
• type LLMProvider
• type LLMRequest
• type LLMRetryEvent
• type Message
• type MessageEndEvent
• type OpenAIHints
• type ProduceFn
• type ProviderCapabilities
• type ProviderHints
• type RetryPolicy
• type StreamResult
• type TextContent
• type TextDeltaEvent
• type ThinkingContent
• type ThinkingDeltaEvent
• type ThinkingLevel
• type ToolCallContent
• type ToolCallEndEvent
• type ToolCallStartEvent
• type ToolDef
• type ToolResultContent
• type TransportPreference
  • func (t TransportPreference) String() string
• type UsageEvent

Constants

const (
	DefaultMaxRetries    = 3
	DefaultMaxRetryDelay = 60 * time.Second
)

Defaults for RetryPolicy zero values.

const DefaultEventBufferSize = 16

DefaultEventBufferSize is the default buffer size for EventStream.Events.

Variables

var (
	ErrProviderFatal        = errors.New("provider fatal error")
	ErrInvalidModel         = errors.New("invalid model")
	ErrUnsupportedFeature   = errors.New("unsupported feature")
	ErrMalformedResponse    = errors.New("malformed provider response")
	ErrTransportUnsupported = errors.New("transport not supported by provider")
	ErrContextOverflow      = errors.New("context length exceeded")
)

Sentinel errors for pi-llm-go.

Functions

func AsContextOverflow

func AsContextOverflow(err error) error

AsContextOverflow classifies provider errors: when err's message reports the model's maximum context length was exceeded, it returns an error wrapping ErrContextOverflow so callers can react via errors.Is (Compact + retry, truncate, or switch models). Other errors and nil pass through unchanged.

Types

type Content

type Content interface {
	// contains filtered or unexported methods
}

Content is a sealed interface for the content of a Message.

type EventStream

type EventStream struct {
	Events <-chan LLMEvent
	Done   <-chan StreamResult
}

EventStream is the shared return shape for any streaming LLM call. Events delivers a stream of typed events and closes when the run finishes. Done delivers exactly one terminal StreamResult.

func NewEventStream

func NewEventStream(events <-chan LLMEvent, done <-chan StreamResult) EventStream

NewEventStream creates an EventStream from the provided channels. Providers use this to package their internal goroutine outputs.

func Run

func Run(ctx context.Context, req LLMRequest, produce ProduceFn) EventStream

Run executes a streaming LLM call using the given producer callback. It allocates channels, manages goroutine lifetime, and enforces the EventStream contract: Events closes when the stream ends, Done delivers exactly one StreamResult with Messages = req.Messages ++ produced.

type GeminiHints

type GeminiHints struct {
	ThinkingBudget int
}

GeminiHints carries provider-specific overrides for the Gemini provider.

type ImageContent

type ImageContent struct {
	Data     []byte
	MimeType string // image/jpeg, image/png, image/gif, image/webp
}

ImageContent carries an inline image. Data is raw bytes; adapters base64-encode on the wire. encoding/json marshals []byte as base64, so JSON transcripts store compactly for free. Valid in user messages (attachments) and on ToolResultContent.Images. A user turn with text and an image is two adjacent user messages; no multi-content message shape exists.

type LLMErrorEvent

type LLMErrorEvent struct {
	Error     error
	Transient bool
}

LLMErrorEvent signals an error from the provider.

type LLMEvent

type LLMEvent interface {
	// contains filtered or unexported methods
}

LLMEvent is a sealed interface for every event that flows on EventStream.Events. Concrete variants are defined below; discrimination is via type switch.

type LLMProvider

type LLMProvider interface {
	// Name returns the provider's short identifier, used in model references
	// like "<name>/<model-id>".
	Name() string

	// Capabilities returns the feature set for the given model.
	Capabilities(model string) ProviderCapabilities

	// Stream initiates a streaming LLM call. The returned EventStream delivers
	// typed events on Events and a single terminal result on Done.
	// Cancellation is honored via ctx; the Events channel closes when the
	// stream completes or ctx is cancelled.
	Stream(ctx context.Context, req LLMRequest) EventStream
}

LLMProvider is the single interface implemented by every concrete provider. It abstracts differences between LLM wire protocols without imposing agent-loop opinions.

type LLMRequest

type LLMRequest struct {
	Model    string
	Messages []Message
	Tools    []ToolDef
	Thinking ThinkingLevel
	// SessionID optionally identifies the conversation this call belongs to.
	// The OpenAI-compatible adapter sends it as prompt-cache affinity headers
	// (session_id, x-client-request-id, x-session-affinity) and as the
	// prompt_cache_key body param, so repeated calls route to the same replica
	// and maximize prompt-cache hits. The Gemini adapter ignores it. Empty means
	// "no session hint".
	SessionID string
	// ThinkingBudgets optionally overrides the per-level token budget for the
	// active Thinking level. Providers whose reasoning control is token-based
	// (Gemini's thinking_budget) apply it; providers whose control is categorical
	// (OpenAI-compatible reasoning_effort) ignore it. Nil means "use provider
	// defaults". ProviderHints, when set, takes precedence over this map.
	ThinkingBudgets map[ThinkingLevel]int
	// Transport is the preferred stream transport. Providers that support only
	// HTTP/SSE honor TransportAuto and TransportSSE; TransportWebSocket returns
	// ErrTransportUnsupported until a websocket-capable provider exists.
	Transport       TransportPreference
	ProviderHints   ProviderHints
	Retry           RetryPolicy
	Headers         map[string]string
	OnBeforeRequest func(headers map[string]string) error
	OnAfterResponse func(statusCode int, headers map[string]string)
}

LLMRequest carries all inputs for a single streaming LLM call.

type LLMRetryEvent

type LLMRetryEvent struct {
	Provider   string
	Model      string
	Attempt    int
	NextDelay  time.Duration
	Reason     string
	ServerHint bool
}

LLMRetryEvent signals that a retry attempt is being made.

type Message

type Message struct {
	Role    string
	Content Content
}

Message is the LLM-side unit of transcript content. It is provider-shaped, not user-extensible; distinct from any agent-side transcript Message.

type MessageEndEvent

type MessageEndEvent struct{}

MessageEndEvent signals the end of the assistant's message. It is always the last non-error event in a successful stream.

type OpenAIHints

type OpenAIHints struct {
	ReasoningEffort string
}

OpenAIHints carries provider-specific overrides for the OpenAI-compatible adapter.

type ProduceFn

type ProduceFn func(ctx context.Context, req LLMRequest, emit func(LLMEvent) error, headers map[string]string, setResponseMeta func(status int, respHeaders map[string]string)) ([]Message, error)

ProduceFn is the callback signature for provider-specific streaming logic. The producer receives an emit function that handles ctx cancellation. It returns the new messages produced during the stream and any error. The headers map is the merged result of Config.Headers + LLMRequest.Headers + any mutations made by OnBeforeRequest hooks. setResponseMeta lets the producer report HTTP status + response headers for the OnAfterResponse hook. It may be called at most once.

type ProviderCapabilities

type ProviderCapabilities struct {
	Streaming         bool
	ToolCalling       bool
	ParallelToolCalls bool
	Thinking          bool
	PromptCaching     bool
	Vision            bool
}

ProviderCapabilities describes the feature set of a specific model.

type ProviderHints

type ProviderHints struct {
	OpenAI *OpenAIHints
	Gemini *GeminiHints
}

ProviderHints is a typed escape hatch for provider-specific config. Only the field matching the active provider is consulted; others are ignored.

type RetryPolicy

type RetryPolicy struct {
	MaxRetries    int
	MaxRetryDelay time.Duration
}

RetryPolicy configures retry behavior for a provider call. The actual retry logic is delegated to the underlying SDK; this struct is the configuration shape shared across providers.

type StreamResult

type StreamResult struct {
	Messages []Message
	Err      error
}

StreamResult is the single terminal value delivered on EventStream.Done.

type TextContent

type TextContent struct {
	Text string
}

TextContent carries plain text.

type TextDeltaEvent

type TextDeltaEvent struct {
	Delta string
}

TextDeltaEvent carries a fragment of text output from the LLM.

type ThinkingContent

type ThinkingContent struct {
	Text string
}

ThinkingContent carries reasoning/thinking content from the LLM.

type ThinkingDeltaEvent

type ThinkingDeltaEvent struct {
	Delta string
}

ThinkingDeltaEvent carries a fragment of thinking/reasoning content.

type ThinkingLevel

type ThinkingLevel int

ThinkingLevel is a portable abstraction over provider-specific reasoning controls.

const (
	ThinkingOff ThinkingLevel = iota
	ThinkingMinimal
	ThinkingLow
	ThinkingMedium
	ThinkingHigh
)

type ToolCallContent

type ToolCallContent struct {
	CallID   string
	ToolName string
	Args     json.RawMessage
	// ThoughtSignature is an opaque provider token bound to this tool call
	// (Gemini 3 thought signatures). Callers replaying history must carry it
	// back verbatim; empty for providers without one.
	ThoughtSignature []byte
}

ToolCallContent carries a tool invocation from the LLM.

type ToolCallEndEvent

type ToolCallEndEvent struct {
	CallID string
}

ToolCallEndEvent signals the end of a tool call block.

type ToolCallStartEvent

type ToolCallStartEvent struct {
	CallID   string
	ToolName string
	Args     json.RawMessage
	// ThoughtSignature is an opaque provider token bound to this tool call
	// (Gemini 3 thought signatures). Consumers persisting the transcript must
	// carry it onto the replayed ToolCallContent; empty for providers without one.
	ThoughtSignature []byte
}

ToolCallStartEvent signals the beginning of a tool call in the stream.

type ToolDef

type ToolDef struct {
	Name        string
	Description string
	Schema      json.RawMessage
}

ToolDef is the LLM-visible tool specification.

type ToolResultContent

type ToolResultContent struct {
	CallID   string
	ToolName string
	Content  string
	// Images carries optional image parts of the tool result (e.g. the
	// read tool returning a screenshot). Nil for text-only results.
	Images  []ImageContent
	Data    json.RawMessage
	IsError bool
}

ToolResultContent carries the result of a tool execution back to the LLM.

type TransportPreference

type TransportPreference int

TransportPreference is a portable preference for the stream transport a provider uses. Providers that support only one transport honor TransportAuto and TransportSSE by using HTTP/SSE; a provider that does not implement the requested transport returns ErrTransportUnsupported rather than silently falling back, so callers learn the transport is unavailable.

const (
	TransportAuto TransportPreference = iota
	TransportSSE
	TransportWebSocket
)

func (TransportPreference) String

func (t TransportPreference) String() string

String returns the wire-style name of the transport preference.

type UsageEvent

type UsageEvent struct {
	InputTokens  int
	OutputTokens int
}

UsageEvent carries token-usage metadata from the provider.