llm

README

pi-llm-go

A minimal, Go-native LLM adapter with streaming, tool calling, and extended thinking. Provider-agnostic interface with built-in support for Anthropic Messages and OpenAI-compatible Chat Completions (covering OpenAI, Groq, Together, vLLM, OpenRouter, Ollama, and similar).

Status: v0.x — pre-1.0. API may change between minor versions; see CHANGELOG.md.

Why

The Go LLM library landscape forces you to pick between heavy vendor SDKs that don't compose, code-generated client surfaces that don't track new providers, or "framework" libraries that ship more concepts than you want. pi-llm-go is the opposite: ~1.5kLoc of plain Go that gives you an interface, two providers, and a streaming model that uses iter.Seq2 like a normal iterator. No HTTP wrappers, no model registries, no provider-specific magic.

Installation

go get github.com/amit-timalsina/pi-llm-go

Requires Go 1.23 or later (for iter.Seq2).

Quickstart

package main

import (
    "context"
    "fmt"
    "os"

    llm "github.com/amit-timalsina/pi-llm-go"
    "github.com/amit-timalsina/pi-llm-go/providers/anthropic"
)

func main() {
    p, _ := anthropic.New(anthropic.Options{APIKey: os.Getenv("ANTHROPIC_API_KEY")})

    msg, err := llm.Complete(context.Background(), p, llm.Request{
        Model:     anthropic.ClaudeSonnet4_6,
        MaxTokens: 1024,
        Messages: []llm.Message{
            {Role: llm.RoleUser, Content: []llm.Block{llm.TextBlock{Text: "hello"}}},
        },
    })
    if err != nil { panic(err) }

    for _, block := range msg.Content {
        if tb, ok := block.(llm.TextBlock); ok {
            fmt.Println(tb.Text)
        }
    }
}

Streaming is iter.Seq2[llm.StreamEvent, error] — range over it:

for event, err := range p.Stream(ctx, req) {
    if err != nil { /* handle */ }
    if d, ok := event.(llm.EventTextDelta); ok {
        fmt.Print(d.Delta)
    }
}

Features

• Streaming-first. Stream() returns iter.Seq2[StreamEvent, error] — Go 1.23 iterators, no callbacks, no goroutine leaks. Complete() is the synchronous helper for one-shot use.
• Sealed sum types. Block and StreamEvent are interfaces with package-private marker methods. Type-switch exhaustively; the compiler tells you if you miss a case.
• Tool calling. Declare tools on Request.Tools; receive ToolCallBlocks on the response; send ToolResultBlocks back. Pi-llm-go does not execute tools — that's pi-agent-go's job.
• Extended thinking. ThinkingConfig{BudgetTokens: N} on requests, surfaced as ThinkingBlock content. Anthropic-only at v1.
• Open-closed providers. Implement LLM.Stream to add custom providers; no plugin registry needed.
• Errors that branch cleanly. errors.Is(err, llm.ErrRateLimit) works through *APIError wraps; errors.As(err, &apiErr) gives you status + body.
• Cancellation = context.Context. No bespoke abort signal types.

Providers

Anthropic

import "github.com/amit-timalsina/pi-llm-go/providers/anthropic"

p, _ := anthropic.New(anthropic.Options{
    APIKey: os.Getenv("ANTHROPIC_API_KEY"),
    // BaseURL: defaults to https://api.anthropic.com
    // Version: defaults to "2023-06-01"
    // Beta:    optional anthropic-beta header values
})

Honors Request.Thinking. Surfaces all content-block types (text, thinking, tool_use).

OpenAI-compatible

import "github.com/amit-timalsina/pi-llm-go/providers/openai"

p, _ := openai.New(openai.Options{
    APIKey:  os.Getenv("OPENAI_API_KEY"),
    BaseURL: "https://api.openai.com/v1",  // or "https://api.groq.com/openai/v1", etc.
})

Talks the Chat Completions wire format, so the same provider works against any compatible host. Request.Thinking is ignored at v1 — reasoning-effort dialects vary too much across compatible hosts to map portably.

Examples

Runnable examples in examples/:

• examples/streaming — basic streaming of a text response.
• examples/tool_calling — hand-rolled tool-call loop against get_current_time.

Run them with go run ./examples/streaming (set ANTHROPIC_API_KEY first).

Versioning

This package is pre-1.0. Anything can change between minor versions; refer to CHANGELOG.md for each release.

v1.0 lands when the API surface has been used in production for ≥4 weeks without churn. Post-1.0 follows semver strictly.

License

MIT. See LICENSE.

Acknowledgements

Designed and named after pi-ai by Mario Zechner (TypeScript, MIT). The wire-level vocabulary and event types follow the upstream's lead; the Go-native API surface (interface-based providers, iterator streaming, sealed sum types) is a from-scratch redesign for Go idioms.

Built and maintained by Amit Timalsina with Claude Code assistance — all design decisions and release calls are human-owned.

Documentation

Overview

Package llm is a minimal, provider-agnostic Go interface for streaming LLM completions with tool calling. Built-in providers cover the Anthropic Messages API and OpenAI-compatible Chat Completions APIs (OpenAI, Groq, Together, vLLM, OpenRouter, Ollama, and similar).

Two patterns of use:

• Streaming events: range over LLM.Stream to react to deltas in real time.
• One-shot completion: call Complete to drain the stream and receive the final assistant message.

Cancellation propagates through context.Context. Provider errors surface through the iterator's error return as *APIError wrapping a sentinel (ErrAuth, ErrRateLimit, ErrInvalidRequest, ErrProvider).

pi-llm-go does not execute tools. It declares tool schemas on requests and surfaces ToolCallBlocks on responses. The companion pi-agent-go module adds the execution loop.

Index

• Variables
• func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]
• func SentinelForStatus(status int) error
• type APIError
  • func (e *APIError) Error() string
  • func (e *APIError) Unwrap() error
• type Block
• type EventMessageEnd
• type EventMessageStart
• type EventTextDelta
• type EventTextEnd
• type EventTextStart
• type EventThinkingDelta
• type EventThinkingEnd
• type EventThinkingStart
• type EventToolCallDelta
• type EventToolCallEnd
• type EventToolCallStart
• type LLM
• type Message
  • func Complete(ctx context.Context, l LLM, req Request) (*Message, error)
• type Request
• type Role
• type StopReason
• type StreamEvent
• type TextBlock
• type ThinkingBlock
• type ThinkingConfig
• type Tool
• type ToolCallBlock
• type ToolResultBlock
• type Usage

Variables

var (
	ErrAuth           = errors.New("llm: authentication failed")
	ErrRateLimit      = errors.New("llm: rate limited")
	ErrInvalidRequest = errors.New("llm: invalid request")
	ErrProvider       = errors.New("llm: provider error")
)

Sentinel errors. Wrap via APIError or return directly. Use errors.Is to branch on them in caller retry / fallback logic.

Functions

func Accumulate

func Accumulate(events iter.Seq2[StreamEvent, error]) iter.Seq2[*Message, error]

Accumulate folds a stream of events into a stream of progressively-built messages. Each yield emits a snapshot of the assistant message as it stands after the most recent event. The final yielded value (when err is nil) is the fully-assembled assistant message.

Callers that only want the final message should prefer Complete. Use Accumulate when intermediate snapshots are useful (e.g. driving a UI that renders each delta against a complete message tree rather than against individual deltas).

Each snapshot is an independent value — internal slices and strings are copied on emit so callers can retain previous snapshots without aliasing.

func SentinelForStatus

func SentinelForStatus(status int) error

SentinelForStatus maps an HTTP status code to the matching sentinel error. Used by provider implementations when constructing APIError.

Types

type APIError

type APIError struct {
	Provider string
	Status   int
	Body     []byte
	Inner    error
}

APIError wraps a non-2xx HTTP response from a provider. The Inner field is one of the sentinel errors above so that errors.Is works through the wrapping. Status and Body let callers inspect the raw failure (e.g. to parse a structured provider error payload).

func (*APIError) Error

func (e *APIError) Error() string

func (*APIError) Unwrap

func (e *APIError) Unwrap() error

type Block

type Block interface {
	// contains filtered or unexported methods
}

Block is the sealed sum type for message content. Concrete implementations are TextBlock, ThinkingBlock, ToolCallBlock, and ToolResultBlock — all defined in this package. The unexported marker method keeps the set closed: provider converters need exhaustive type-switches to serialize content correctly, so new block types must be added inside the package.

type EventMessageEnd

type EventMessageEnd struct {
	StopReason StopReason
	Usage      Usage
}

EventMessageEnd is the terminal event. It carries the normalized stop reason and the final usage tally.

type EventMessageStart

type EventMessageStart struct {
	Model string
}

EventMessageStart is emitted once at the start of an assistant turn, before any block events.

type EventTextDelta

type EventTextDelta struct {
	BlockIndex int
	Delta      string
}

EventTextDelta appends Delta to the text in the block at BlockIndex.

type EventTextEnd

type EventTextEnd struct {
	BlockIndex int
}

EventTextEnd marks the end of the TextBlock at BlockIndex.

type EventTextStart

type EventTextStart struct {
	BlockIndex int
}

EventTextStart marks the beginning of a TextBlock.

type EventThinkingDelta

type EventThinkingDelta struct {
	BlockIndex int
	Delta      string
}

EventThinkingDelta appends Delta to the thinking block at BlockIndex.

type EventThinkingEnd

type EventThinkingEnd struct {
	BlockIndex int
	Signature  string
}

EventThinkingEnd marks the end of the ThinkingBlock. Signature is the opaque provider-supplied token to round-trip on follow-up messages.

type EventThinkingStart

type EventThinkingStart struct {
	BlockIndex int
}

EventThinkingStart marks the beginning of a ThinkingBlock.

type EventToolCallDelta

type EventToolCallDelta struct {
	BlockIndex int
	Delta      string
}

EventToolCallDelta delivers a fragment of the streaming JSON arguments. Callers that need the assembled arguments should wait for EventToolCallEnd rather than accumulate Delta bytes themselves (provider JSON delta framing is not guaranteed to be at value boundaries).

type EventToolCallEnd

type EventToolCallEnd struct {
	BlockIndex int
	Arguments  json.RawMessage
}

EventToolCallEnd marks the end of a ToolCallBlock and carries the assembled arguments.

type EventToolCallStart

type EventToolCallStart struct {
	BlockIndex int
	ID         string
	Name       string
}

EventToolCallStart marks the beginning of a ToolCallBlock. ID and Name are available immediately; arguments stream as deltas and are emitted in fully assembled form on EventToolCallEnd.

type LLM

type LLM interface {
	Stream(ctx context.Context, req Request) iter.Seq2[StreamEvent, error]
}

LLM is the provider-agnostic streaming interface. Anthropic and OpenAI providers implement this; third-party providers may do the same to plug into Complete and Accumulate.

Implementations must:

• Honor cancellation of ctx by terminating the underlying HTTP request and yielding (nil, ctx.Err()) from the iterator.
• Surface HTTP errors as *APIError values via the iterator's error half.
• Emit events in the order documented on StreamEvent.

type Message

type Message struct {
	Role    Role
	Content []Block

	Usage      Usage
	StopReason StopReason
	Model      string
}

Message is one turn in the transcript. Content holds a sequence of blocks — the model emits assistant messages with mixed text / thinking / tool-call content; the caller sends user messages with text and tool messages with tool-result content.

Usage, StopReason, and Model are populated on assistant messages produced by Complete or Accumulate; they are zero on user / tool messages and on messages sent into Stream.

func Complete

func Complete(ctx context.Context, l LLM, req Request) (*Message, error)

Complete drains a streaming completion and returns the final assistant message. It is equivalent to iterating Stream and folding each event into a Message via Accumulate.

Returns the partial message and a wrapped error if the stream terminates early; the partial may be useful for debugging or replay.

type Request

type Request struct {
	Model       string
	System      string
	Messages    []Message
	Tools       []Tool
	Temperature *float64
	MaxTokens   int
	Thinking    *ThinkingConfig
	StopReasons []string
}

Request is the common payload for a completion. Provider-specific tunables that have no portable meaning live on the provider's own Options struct (passed to its constructor), not here.

Temperature is a pointer so the zero value can be distinguished from "unset"; callers that want temperature=0 must set *Temperature to 0.

type Role

type Role string

Role enumerates message roles in a transcript. RoleTool messages carry tool results back to the model and may hold only ToolResultBlock content.

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type StopReason

type StopReason string

StopReason is the normalized reason a model stopped generating. Provider-specific stop reasons map to one of these; unmappable values surface as errors rather than leaking provider strings.

const (
	StopReasonEnd       StopReason = "end"        // natural end of turn
	StopReasonMaxTokens StopReason = "max_tokens" // hit MaxTokens cap
	StopReasonToolUse   StopReason = "tool_use"   // model requested tool calls
	StopReasonStop      StopReason = "stop"       // matched a stop sequence
)

type StreamEvent

type StreamEvent interface {
	// contains filtered or unexported methods
}

StreamEvent is the sealed sum type emitted during a streaming completion. Errors flow through the iterator's error half rather than as event values, so consumers do not need an EventError variant.

Event order for one assistant turn:

EventMessageStart
  ( EventTextStart, EventTextDelta*, EventTextEnd
  | EventThinkingStart, EventThinkingDelta*, EventThinkingEnd
  | EventToolCallStart, EventToolCallDelta*, EventToolCallEnd
  )*
EventMessageEnd

BlockIndex on per-block events is the position of the block inside the emitted message's Content slice — it lets consumers route deltas when rendering or reconstructing incrementally.

type TextBlock

type TextBlock struct {
	Text string
}

TextBlock holds plain text content.

type ThinkingBlock

type ThinkingBlock struct {
	Thinking  string
	Signature string
}

ThinkingBlock holds an extended-thinking segment emitted by reasoning models. Signature is an opaque provider-supplied token that must be preserved and replayed for multi-turn thinking continuity (Anthropic).

type ThinkingConfig

type ThinkingConfig struct {
	// BudgetTokens is the maximum number of thinking tokens the model may
	// emit before producing the final response. Required when ThinkingConfig
	// is non-nil. Provider minimums apply (Anthropic: 1024).
	//
	// IMPORTANT: Anthropic requires Request.MaxTokens > BudgetTokens because
	// thinking tokens are counted against max_tokens. A common safe choice
	// is MaxTokens == BudgetTokens * 2, giving roughly equal budget to the
	// reasoning trace and the visible answer.
	BudgetTokens int
}

ThinkingConfig enables extended thinking on supported models. Honored by the Anthropic provider. Ignored by the OpenAI-compatible provider in v1.

type Tool

type Tool struct {
	Name        string
	Description string
	InputSchema json.RawMessage
}

Tool is the wire-level declaration of a callable function exposed to the model. pi-llm-go does not execute tools — it surfaces ToolCallBlocks on the response and accepts ToolResultBlocks in follow-up messages. Execution lives in pi-agent-go.

InputSchema is a JSON Schema document describing the tool's expected input. Both Anthropic and OpenAI accept JSON Schema draft-07; the schema is forwarded to the provider as-is, so the caller is responsible for dialect choice.

type ToolCallBlock

type ToolCallBlock struct {
	ID        string
	Name      string
	Arguments json.RawMessage
}

ToolCallBlock represents a tool invocation requested by the model. Arguments is the raw JSON object the model emitted, matching the tool's declared InputSchema. The agent layer validates and dispatches.

type ToolResultBlock

type ToolResultBlock struct {
	ToolCallID string
	Content    string
	IsError    bool
}

ToolResultBlock carries the result of a tool invocation back to the model. ToolCallID matches the ID on the originating ToolCallBlock.

type Usage

type Usage struct {
	InputTokens      int
	OutputTokens     int
	CacheReadTokens  int
	CacheWriteTokens int
	TotalTokens      int
}

Usage records token accounting returned by a provider for a single completion request. Cache fields are zero when the provider doesn't bill or report cache reads/writes separately.