llm

package module
v0.2.2 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 4, 2026 License: MIT Imports: 16 Imported by: 0

README

go-llm-sdk

CI Go Reference Go

Multi-provider Go SDK for LLM inference endpoints — OpenAI, Google Gemini, DeepSeek, Z.ai, Kimi (Moonshot) and Anthropic, plus any OpenAI-compatible gateway. Stdlib only; zero external dependencies.

  • Multiple authenticated endpoints at once — auto-discovered from <PROVIDER>_API_KEY environment variables (aliases supported).
  • Dynamic model discovery — ListModels returns what the account can actually access. No static model tables.
  • One canonical API — OpenAI-shaped requests and responses; Anthropic and Gemini wire formats are translated for you.
  • Production streaming — SSE with an idle watchdog and a hard wall-clock deadline, abort-with-partial-result, retries that never duplicate partial output, premature-close detection, and learn-once fallbacks for providers that reject stream_options, streaming, or reasoning_effort+tools.
  • Predictable under load — goroutine-leak-free streaming, race-clean shared state, and a canonical-only error vocabulary (API keys never leak into error text).

Install

go get github.com/BackendStack21/go-llm-sdk@v0.2.2

Requires Go 1.25+. No dependencies beyond the standard library.

Quickstart

package main

import (
	"context"
	"fmt"
	"log"

	llm "github.com/BackendStack21/go-llm-sdk"
)

func main() {
	ctx := context.Background()
	sdk := llm.New(llm.FromEnv()) // OPENAI_API_KEY, GEMINI_API_KEY, DEEPSEEK_API_KEY,
	                             // ZAI_API_KEY, KIMI_API_KEY, ANTHROPIC_API_KEY (+ aliases)

	for _, p := range sdk.Providers() { // authenticated endpoints, registry order
		models, err := p.ListModels(ctx)
		if err != nil {
			log.Fatal(err)
		}
		fmt.Printf("%s: %d models\n", p.ID(), len(models))
		// models[i] → ID, DisplayName, CreatedAt, ContextWindow, MaxOutputTokens, Capabilities
	}

	chat, err := sdk.Chat("deepseek", "deepseek-chat")
	if err != nil {
		log.Fatal(err)
	}

	res, err := chat.Call(ctx, &llm.ChatRequest{
		System:   []llm.SystemBlock{{Text: "Be terse."}},
		Messages: []llm.Message{{Role: llm.RoleUser, Content: "Hello"}},
	})
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Content)
}

Streaming:

res, err = chat.CallStream(ctx, req, func(d llm.Delta) error {
	switch d.Kind {
	case llm.DeltaReasoning: // thinking fragment
	case llm.DeltaContent:   // text fragment
	case llm.DeltaToolArgs:  // tool-call argument fragment (d.ToolID, d.ToolName)
	}
	return nil // or an error to abort — partial result comes back with *StreamAbortedError
})

Providers

ID Format Default base URL Env var (alias) Base-URL override
openai openai https://api.openai.com/v1 OPENAI_API_KEY OPENAI_BASE_URL
gemini gemini https://generativelanguage.googleapis.com GEMINI_API_KEY (GOOGLE_API_KEY) GEMINI_BASE_URL
deepseek openai https://api.deepseek.com DEEPSEEK_API_KEY DEEPSEEK_BASE_URL
zai openai https://api.z.ai/api/paas/v4 ZAI_API_KEY ZAI_BASE_URL (e.g. coding-plan endpoint)
kimi openai https://api.moonshot.ai/v1 KIMI_API_KEY (MOONSHOT_API_KEY) KIMI_BASE_URL
anthropic anthropic https://api.anthropic.com ANTHROPIC_API_KEY ANTHROPIC_BASE_URL

Primary env var beats its alias. Explicit keys (WithAPIKey) beat env. Base-URL overrides accept any gateway speaking the provider's format. A bad URL passed to WithBaseURL is rejected at wiring time: the provider is marked invalid, Providers() omits it, and Chat returns a *ConfigError — no request is sent.

Custom gateways:

sdk := llm.New(llm.WithProvider("my-gateway",
	llm.WithFormat(llm.FormatOpenAI),
	llm.WithBaseURL("http://localhost:11434/v1"),
	llm.WithAPIKey("local"),
))

Canonical API

Requests and results are provider-neutral. Unknown message roles are rejected at the SDK boundary (never silently dropped or reinterpreted).

type ChatRequest struct {
	Model          string         // optional; ChatClient's model wins when both set
	Messages       []Message      // RoleUser | RoleAssistant | RoleSystem | RoleTool; Message.Cache → Anthropic user-block cache_control
	System         []SystemBlock  // {Text, Cache} — Cache marks Anthropic prompt-cache blocks
	Tools          []ToolDef      // {Name, Description, Parameters json.RawMessage}
	Thinking       string         // "", "enabled", "disabled", "low", "medium", "high", "max"
	ThinkingBudget int            // explicit token budget where the provider supports it
	MaxTokens      int            // routed to max_completion_tokens on o-series/gpt-5
	Temperature    float64        // 0 = provider default; negative = explicit 0
}

type ChatResult struct {
	Content           string
	ReasoningContent  string      // provider thinking text; replayed as reasoning_content on OpenAI-format assistant turns
	ThinkingSignature string      // Anthropic: replay via Message.ThinkingSignature
	ToolCalls         []ToolCall  // {ID, Name, Arguments}
	FinishReason      string      // stop | length | tool_calls | content_filter | ""
	Usage             Usage       // PromptTokens is uncached-only; cache volumes in CacheReadTokens / CacheCreationTokens / CachedTokens
}

Finish reasons are canonical: anything a provider reports outside that vocabulary maps to "" (unknown) rather than leaking provider-specific strings.

Streaming semantics

CallStream enforces four guarantees, each covered by regression tests:

  1. Idle watchdog — a stream silent longer than StreamIdleTimeout() (120s default; override with SetStreamIdleTimeout, positive values only) fails with ErrIdleTimeout. Keepalive comments reset it.
  2. Hard wall-clock deadline — the whole stream is bounded by the per-request timeout (WithRequestTimeout, default 120s; per-client via SetRequestTimeout).
  3. Retries never duplicate output — retries happen only before the first emitted delta. A failure after partial output returns the partial *ChatResult plus a wrapped error and is never retried.
  4. No silent empty successes — a provider that closes the stream before its completion signal (before [DONE] / message_stop) yields a retryable error, not an empty result. Gemini, whose streams legitimately end at EOF, is exempt.

Aborting from the delta handler returns the partial result alongside *StreamAbortedError — the parser goroutine is always released, so aborted streams leak nothing.

Tool-call loop
for {
	res, err := chat.CallStream(ctx, req, func(d llm.Delta) error { return nil })
	if err != nil {
		return err
	}
	if len(res.ToolCalls) == 0 {
		return nil
	}
	req.Messages = append(req.Messages,
		llm.Message{Role: llm.RoleAssistant, Content: res.Content,
			ReasoningContent: res.ReasoningContent, ThinkingSignature: res.ThinkingSignature,
			ToolCalls: res.ToolCalls})
	for _, tc := range res.ToolCalls {
		req.Messages = append(req.Messages,
			llm.Message{Role: llm.RoleTool, ToolCallID: tc.ID, ToolName: tc.Name, Content: execute(tc)})
	}
}

On Gemini, a tool result's ToolName may be omitted — the SDK recovers the function name from the assistant ToolCall it answers, and errors loudly if it cannot.

Extended thinking

  • Anthropic — thinking blocks are parsed in both buffered and streaming modes. ChatResult.ThinkingSignature carries the provider signature; for tool loops, replay it on the assistant message (Message.ReasoningContent + Message.ThinkingSignature) — the SDK re-serializes it as the first block, as Anthropic's API requires. Omitting it makes extended-thinking tool loops fail mid-conversation.
  • DeepSeek / GLM — reasoning streams as DeltaReasoning fragments and lands in ReasoningContent. Assistant-turn replay echoes it as reasoning_content (required for DeepSeek/GLM tool loops). GLM maps thinking medium → reasoning_effort high (no medium level) and max → max.
  • Gemini — thought: true parts map to reasoning deltas; thinkingConfig is derived from Thinking / ThinkingBudget.

Learn-once fallbacks

When a provider rejects a request pattern, the SDK learns the constraint once per provider (shared across every ChatClient you mint) and never re-pays the failed round-trip:

Trigger (provider 400) Learned fallback
Rejects stream_options omit stream_options from streaming requests
Rejects reasoning_effort + tools pin reasoning_effort: "none"
Rejects streaming itself downgrade to buffered calls permanently
Answers a streamed request with a non-SSE body downgrade to buffered calls permanently

Retry policy

8 attempts, exponential backoff capped at 30s with ±20% jitter, Retry-After (seconds or HTTP-date) honored and capped at 120s, context cancellation honored between and during attempts. Retryable statuses include 408/429/5xx plus Cloudflare 520–524 and Anthropic 529. Persistent 429s surface as *llm.RateLimitError{Attempts, RetryAfter} — including when the retry sleep is cut short by a deadline, so the caller never loses the retry signal. A 429 whose body is billing exhaustion (insufficient_quota, exceeded your current quota, insufficient balance, no resource package) is not retryable and fails on the first attempt. RateLimitError unwraps to *APIError for errors.As access to Status/Retryable.

Timeouts & cancellation

  • The pooled transport honors HTTP_PROXY / HTTPS_PROXY / NO_PROXY via http.ProxyFromEnvironment.
  • Buffered calls: per-request timeout on the HTTP client (WithRequestTimeout, per-client SetRequestTimeout — race-safe, swap is atomic).
  • Streaming: the same timeout becomes the hard wall-clock deadline via context; per-attempt SSE reads are additionally bounded by the idle watchdog.
  • Every wait (backoff, Retry-After, stream reads) selects on the caller's context — cancellation propagates everywhere, and a cancelled call never misreports as "retry exhausted".
  • Response bodies are capped (50 MB chat, 8 MB listings, 1 MiB SSE lines, 4 MiB SSE events) as an OOM bound.

Error handling

*ConfigError (unknown/unauthenticated provider, invalid wiring, unknown role, no model), *APIError{Provider, Status, Code, Message, Retryable}, *RateLimitError{Attempts, RetryAfter} (unwraps to *APIError), *StreamAbortedError (returned together with the partial *ChatResult). A stream failure after partial output returns the partial *ChatResult plus a wrapped error and is never retried; the idle watchdog surfaces as ErrIdleTimeout (retried only before the first delta); wall-clock deadlines surface as context deadline errors. Recommended classification:

var abort *llm.StreamAbortedError
var rl *llm.RateLimitError
var ae *llm.APIError
switch {
case errors.As(err, &abort):                 // consumer abort (partial result returned)
case errors.As(err, &rl):                    // back off rl.RetryAfter
case errors.As(err, &ae):                    // provider said no (ae.Status)
case errors.Is(err, llm.ErrIdleTimeout):     // stream went silent
case errors.Is(err, context.DeadlineExceeded): // wall-clock budget spent
}

API keys never appear in any error text. Provider error bodies are parsed per format (nested OpenAI envelope, Anthropic error.type/message, Gemini error.status/message) with a 512-byte raw-body fallback.

Thread safety

SDK and Provider are safe for concurrent use. ChatClient is safe for concurrent Call/CallStream; SetRequestTimeout is race-safe (atomic swap) but should still be called before the first request so in-flight calls use one timeout. Learn-once state is shared per provider via atomics — monotonic, converging, race-free.

Model discovery

ListModels hits each provider's models endpoint (Anthropic paginates with after_id, Gemini with pageToken), caches per SDK for 5 minutes (WithModelCacheTTL(0) disables, ForceRefresh() bypasses), and retries transient failures 3×. Fields the provider does not report stay zero — the SDK never guesses.

Testing

make quality    # fmt + vet + tests
make test-race  # race detector
make lint       # golangci-lint (v2 config)

Live end-to-end tests against real APIs (tag-gated, never run in CI). The suite covers every provider you have credentials for — currently DeepSeek, Z.ai, and a custom OpenRouter gateway. Each target skips when its key is absent.

go test -tags e2e -run 'TestE2E' -timeout 15m -v .

Credentials come from the environment or a repo-root .env file (KEY=VALUE); the file is gitignored and its contents are never logged. Override a target's model with <ID>_E2E_MODEL (e.g. DEEPSEEK_E2E_MODEL). Adding a provider is one e2eTarget entry in e2e_test.go.

Coverage sits at 97.7% of statements, including the streaming failure-orchestration paths (deadline, 429, premature close, partial-output) that are usually the blind spot of SDK test suites. The residual ~2% is unreachable defensive code.

Repo guidance

See AGENTS.md for the architecture map, invariants, testing conventions, and the odek migration path.

Status

v0.2.2 — API may shift until the odek integration lands, then v1.0.

License

MIT

Documentation

Overview

Package llm is a multi-provider Go SDK for LLM inference endpoints: OpenAI, Google Gemini, DeepSeek, Z.ai, Kimi (Moonshot) and Anthropic, plus any custom OpenAI-compatible gateway.

Design highlights:

  • Multiple authenticated endpoints simultaneously, discovered from the environment via <PROVIDER>_API_KEY (aliases supported).
  • Dynamic model discovery (ListModels) — no static model profile tables.
  • One canonical request/response shape (OpenAI-compatible); Anthropic and Gemini formats are translated by per-format serializers.
  • Zero external dependencies: stdlib only.
  • Streaming with idle watchdog, hard wall-clock deadline, abort-with- partial-result, and retries that never duplicate partial output.

Index

Constants

View Source 
const (
	FinishStop          = "stop"
	FinishLength        = "length"
	FinishToolCalls     = "tool_calls"
	FinishContentFilter = "content_filter"
)

Canonical finish reasons.

View Source 
const (
	DefaultTimeout = 120 * time.Second
)

Transport tuning defaults, shared by every provider client an SDK builds.

Variables

View Source 
var ErrIdleTimeout = errors.New("llm: stream idle timeout")

ErrIdleTimeout is returned (with the partial result) when a stream goes silent longer than the idle watchdog after the first delta has been emitted. Before the first delta, idle timeouts are retried instead.

Functions

func SetStreamIdleTimeout added in v0.2.1 

func SetStreamIdleTimeout(d time.Duration)

SetStreamIdleTimeout overrides the SSE idle watchdog. Call at startup, before the first request; non-positive values are ignored.

func StreamIdleTimeout added in v0.2.1 

func StreamIdleTimeout() time.Duration

StreamIdleTimeout reports the active idle watchdog (introspection/tests).

Types

type APIError 

type APIError struct {
	Provider  string
	Status    int
	Code      string
	Message   string
	Retryable bool
}

APIError is a non-2xx provider response. Status 0 with a non-nil underlying error form is not used; network failures surface as plain wrapped errors. Message is the provider's own error text when parseable. API keys are never included in any error text.

func (*APIError) Error 

func (e *APIError) Error() string

type ChatClient 

type ChatClient struct {
	// contains filtered or unexported fields
}

ChatClient runs chat completions against one provider+model pair. Each client carries its own learn-once fallbacks and request timeout; it is safe for concurrent use but SetRequestTimeout must be called before the first request.

func (*ChatClient) Call 

func (c *ChatClient) Call(ctx context.Context, req *ChatRequest) (*ChatResult, error)

Call runs a buffered chat completion.

func (*ChatClient) CallStream 

func (c *ChatClient) CallStream(ctx context.Context, req *ChatRequest, onDelta func(Delta) error) (*ChatResult, error)

CallStream runs a streaming chat completion. onDelta receives canonical fragments; returning an error from it aborts generation — CallStream then returns the partial result alongside *StreamAbortedError.

func (*ChatClient) Model 

func (c *ChatClient) Model() string

Model returns the bound model.

func (*ChatClient) ProviderID 

func (c *ChatClient) ProviderID() string

ProviderID returns the bound provider's id.

func (*ChatClient) RequestTimeout 

func (c *ChatClient) RequestTimeout() time.Duration

RequestTimeout reports the per-request timeout (streaming wall-clock deadline).

func (*ChatClient) SetRequestTimeout 

func (c *ChatClient) SetRequestTimeout(d time.Duration)

SetRequestTimeout adjusts the per-request timeout. Call before the first request.

type ChatRequest 

type ChatRequest struct {
	Model          string
	Messages       []Message
	System         []SystemBlock
	Tools          []ToolDef
	Thinking       string
	ThinkingBudget int
	MaxTokens      int
	Temperature    float64
}

ChatRequest is the canonical request. Model is filled from the ChatClient when empty. Thinking accepts "", "enabled", "disabled", "low", "medium", "high", "max" and is translated per provider format. Temperature: 0 means use the provider default (field omitted); use a negative value to explicitly send 0.

type ChatResult 

type ChatResult struct {
	Content          string
	ReasoningContent string
	// ThinkingSignature authenticates ReasoningContent (Anthropic extended
	// thinking). Consumers must carry it back on the next assistant Message
	// for tool loops to stay valid.
	ThinkingSignature string
	ToolCalls         []ToolCall
	FinishReason      string
	Usage             Usage
}

ChatResult is the canonical response for both buffered and streaming calls.

type ConfigError 

type ConfigError struct{ Msg string }

ConfigError reports SDK misuse: unknown provider id, provider without an API key, or malformed options. Never retryable.

func (*ConfigError) Error 

func (e *ConfigError) Error() string

type Delta 

type Delta struct {
	Kind      DeltaKind
	Text      string
	ToolIndex int
	ToolID    string
	ToolName  string
}

Delta is one streamed fragment. Text is the fragment for this event, not accumulated output.

type DeltaKind 

type DeltaKind int

DeltaKind discriminates streamed fragments. 

const (
	// DeltaReasoning is a thinking/reasoning fragment, usually before content.
	DeltaReasoning DeltaKind = iota
	// DeltaContent is an assistant text fragment.
	DeltaContent
	// DeltaToolArgs is a tool-call argument fragment (partial JSON). ToolIndex
	// and, on the first fragment of a call, ToolID/ToolName identify the call.
	DeltaToolArgs
)

type Format 

type Format string

Format identifies a wire protocol family. The SDK translates the single canonical request shape into exactly one of these. 

const (
	// FormatOpenAI is the OpenAI chat-completions protocol — also spoken by
	// DeepSeek, Z.ai (GLM), Kimi/Moonshot and most self-hosted gateways.
	FormatOpenAI Format = "openai"
	// FormatAnthropic is the Anthropic Messages API.
	FormatAnthropic Format = "anthropic"
	// FormatGemini is the Google Gemini generateContent API.
	FormatGemini Format = "gemini"
)

type ListOption 

type ListOption func(*listOpts)

ListOption tweaks ListModels.

func ForceRefresh 

func ForceRefresh() ListOption

ForceRefresh bypasses the model cache for this call and refreshes it.

type Message 

type Message struct {
	Role             Role
	Content          string
	ReasoningContent string
	// ThinkingSignature authenticates ReasoningContent for providers that
	// require thinking to be replayed verbatim (Anthropic signature).
	ThinkingSignature string
	ToolCalls         []ToolCall
	ToolCallID        string
	ToolName          string
	// Cache marks this user message for Anthropic prompt caching
	// (cache_control ephemeral on the text block). Ignored on other
	// formats and on non-user roles.
	Cache bool
}

Message is one canonical chat message. For RoleTool messages, ToolCallID and ToolName identify the call being answered and Content carries the tool result. ReasoningContent is provider-reported thinking text (deepseek-reasoner, anthropic thinking, gemini thoughts). The SDK replays it where a provider requires conversation continuity: OpenAI-format assistant messages echo it as reasoning_content (DeepSeek/GLM tool loops), and Anthropic re-serializes a signed thinking block as the first content block when ThinkingSignature is also set.

type Model 

type Model struct {
	ID              string
	DisplayName     string
	CreatedAt       time.Time
	ContextWindow   int // input token limit, 0 = unknown
	MaxOutputTokens int // 0 = unknown
	Capabilities    []string
}

Model is one accessible model, as reported by a provider's models endpoint. Fields the provider does not report stay zero — the SDK never guesses.

type Option 

type Option func(*SDK)

Option configures an SDK at construction time.

func FromEnv 

func FromEnv() Option

FromEnv resolves <PROVIDER>_API_KEY (aliases included, primary first) and optional <PROVIDER>_BASE_URL overrides via os.LookupEnv. Providers without a key stay registered but unauthenticated.

func WithEnv 

func WithEnv(lookup func(string) (string, bool)) Option

WithEnv resolves API keys and base-URL overrides from the environment using lookup — the testable twin of FromEnv.

func WithModelCacheTTL 

func WithModelCacheTTL(d time.Duration) Option

WithModelCacheTTL sets the ListModels cache TTL. Zero disables caching — every call hits the provider's models endpoint.

func WithProvider 

func WithProvider(id string, popts ...ProviderOption) Option

WithProvider configures one provider — a built-in id (override) or a new custom provider (requires WithFormat and WithBaseURL, plus auth).

func WithRequestTimeout 

func WithRequestTimeout(d time.Duration) Option

WithRequestTimeout sets the default per-request timeout for all chat clients (default 120s). Streaming calls use it as the hard wall-clock deadline.

func WithTransport 

func WithTransport(rt http.RoundTripper) Option

WithTransport replaces the SDK's pooled HTTP transport (tests, proxies).

type Provider 

type Provider struct {
	// contains filtered or unexported fields
}

Provider is one configured endpoint with dynamic model discovery.

func (*Provider) Authenticated 

func (p *Provider) Authenticated() bool

Authenticated reports whether an API key resolved.

func (*Provider) Config 

func (p *Provider) Config() ProviderConfig

Config returns the provider configuration. The copy carries the API key: treat it as a secret; it is never logged by the SDK itself.

func (*Provider) ID 

func (p *Provider) ID() string

ID returns the provider's registry id.

func (*Provider) ListModels 

func (p *Provider) ListModels(ctx context.Context, opts ...ListOption) ([]Model, error)

ListModels discovers the models accessible to this provider's account, on the fly — no static tables. Results are cached per SDK instance for the cache TTL (default 5 min; WithModelCacheTTL(0) disables). Fields the provider does not report stay zero.

type ProviderConfig 

type ProviderConfig struct {
	ID      string
	Format  Format
	BaseURL string
	APIKey  string
	// EnvKeys lists environment variable names to consult, primary first
	// (aliases after). Resolution stops at the first non-empty value.
	EnvKeys []string
	Quirks  Quirks
}

ProviderConfig fully describes one inference endpoint. APIKey is resolved from EnvKeys (primary first) or set explicitly; it is never included in error text, logs, or String() output.

func (ProviderConfig) String 

func (c ProviderConfig) String() string

type ProviderOption 

type ProviderOption func(*ProviderConfig)

ProviderOption configures one provider entry.

func WithAPIKey 

func WithAPIKey(key string) ProviderOption

WithAPIKey sets an explicit API key (overrides env).

func WithBaseURL 

func WithBaseURL(url string) ProviderOption

WithBaseURL overrides the provider's default base URL.

func WithEnvKeys 

func WithEnvKeys(keys ...string) ProviderOption

WithEnvKeys sets env var names consulted for this provider's key, primary first.

func WithFormat 

func WithFormat(f Format) ProviderOption

WithFormat sets the wire format (custom providers).

func WithQuirks 

func WithQuirks(q Quirks) ProviderOption

WithQuirks overrides protocol quirk flags.

type Quirks 

type Quirks struct {
	// ThinkingObject: provider accepts the Anthropic-style top-level
	// "thinking" object (Anthropic, DeepSeek, Z.ai GLM).
	ThinkingObject bool
	// ReasoningEffort: provider accepts reasoning_effort (OpenAI, GLM-5.3+).
	ReasoningEffort bool
	// ForceThinking lists model-name prefixes that reject
	// thinking.type=disabled outright (GLM-5.3 always reasons; the
	// documented migration is {type: enabled} + reasoning_effort "low").
	ForceThinking []string
	// AnthropicVersion is the anthropic-version header value required by
	// FormatAnthropic providers ("2023-06-01").
	AnthropicVersion string
}

Quirks carries per-provider protocol deviations, resolved at registration time. This replaces odek's URL-sniffing: a custom base URL gets its format's default quirks unless the caller overrides them explicitly.

type RateLimitError 

type RateLimitError struct {
	APIError
	Attempts   int
	RetryAfter time.Duration
}

RateLimitError is the final failure after persistent 429 responses. RetryAfter carries the last observed Retry-After hint (0 if none).

func (*RateLimitError) Error 

func (e *RateLimitError) Error() string

func (*RateLimitError) Unwrap 

func (e *RateLimitError) Unwrap() error

Unwrap exposes the embedded APIError so errors.As(err, *APIError) reaches Status/Retryable without a type switch.

type Role 

type Role string

Role enumerates canonical message roles. 

const (
	RoleSystem    Role = "system"
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
	RoleTool      Role = "tool"
)

type SDK 

type SDK struct {
	// contains filtered or unexported fields
}

SDK is a configured set of LLM endpoints. Build one with New and options: 

sdk := llm.New(llm.FromEnv())

Providers are authenticated when an API key resolves from the environment or is set explicitly; every authenticated provider is usable concurrently through the SDK's shared connection pool.

func New 

func New(opts ...Option) *SDK

New builds an SDK with the built-in provider registry and the given options. Options apply in order; later WithProvider calls for the same id override earlier ones.

func (*SDK) Chat 

func (s *SDK) Chat(providerID, model string) (*ChatClient, error)

Chat returns a chat client bound to a provider and model.

func (*SDK) Provider 

func (s *SDK) Provider(id string) (*Provider, error)

Provider looks a provider up by id. Unknown ids return a ConfigError; known-but-unauthenticated providers are returned with Authenticated() == false.

func (*SDK) Providers 

func (s *SDK) Providers() []*Provider

Providers returns the authenticated providers in registry order.

type StreamAbortedError 

type StreamAbortedError struct{ Reason error }

StreamAbortedError is returned by CallStream when the delta handler aborted generation. CallStream also returns the partial ChatResult assembled so far alongside this error.

func (*StreamAbortedError) Error 

func (e *StreamAbortedError) Error() string

func (*StreamAbortedError) Unwrap 

func (e *StreamAbortedError) Unwrap() error

type SystemBlock 

type SystemBlock struct {
	Text  string
	Cache bool
}

SystemBlock is one system-prompt segment. On Anthropic each block maps to a system text block (Cache marks it for prompt caching); on OpenAI-format providers blocks concatenate into a leading system message; on Gemini they become systemInstruction parts.

type ToolCall 

type ToolCall struct {
	ID        string
	Name      string
	Arguments string // JSON object as a string
}

ToolCall is a model-requested tool invocation.

type ToolDef 

type ToolDef struct {
	Name        string
	Description string
	Parameters  json.RawMessage
}

ToolDef declares a callable tool. Parameters is a JSON Schema object (json.RawMessage so callers can pass through marshaled schemas verbatim).

type Usage 

type Usage struct {
	PromptTokens        int
	CompletionTokens    int
	ReasoningTokens     int
	CacheReadTokens     int
	CacheCreationTokens int
	CachedTokens        int
	CacheReported       bool
}

Usage reports token accounting. Fields the provider does not report stay 0. PromptTokens is exclusive (uncached-only) after provider-specific normalization: OpenAI cached_tokens and DeepSeek hit/miss are subsets of prompt_tokens and are subtracted; Anthropic reports cache volumes exclusively and is left alone. Cache volumes live in the cache fields so budget enforcement can sum without double-counting.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.