llm

package
v0.14.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 11, 2026 License: MIT Imports: 27 Imported by: 0

Documentation

Index

Constants

View Source 
const (
	// AnthropicAPIVersion is the Messages API protocol version. It is
	// independent of model generation — every request, regardless of which
	// Claude (or Claude-compatible) model is targeted, sends this same value.
	AnthropicAPIVersion = "2023-06-01"

	// AnthropicDefaultMaxOutputTokens is the max_tokens fallback used when
	// neither the request nor the catalog's per-model default supplies one.
	// It is a conservative floor, not a target — NewAnthropicProviderClient
	// prefers the catalog's default_max_tokens for the resolved model.
	AnthropicDefaultMaxOutputTokens = 8192

	// AnthropicBaseURL is the real Anthropic API endpoint. The catalog's
	// "anthropic" entry names its api_endpoint as the template
	// "$ANTHROPIC_API_ENDPOINT" (an optional self-hosted-proxy override), not
	// a literal URL, so a fallback is needed whenever no override is exported.
	//
	// It aliases config.AnthropicDefaultAPIEndpoint rather than repeating the
	// URL: the config layer now substitutes that template for every provider,
	// and two independently maintained copies of the same endpoint are exactly
	// how a dialect fallback and a config default drift apart.
	AnthropicBaseURL = config.AnthropicDefaultAPIEndpoint

	// AnthropicAPIKeyEnv names the environment variable holding the official
	// Anthropic API key. Only the name is ever configured; the value is read
	// from the process environment so a secret can never land in YAML.
	AnthropicAPIKeyEnv = "ANTHROPIC_API_KEY"

	// KimiCodingBaseURL, MiniMaxBaseURL, and MiniMaxChinaBaseURL are the three
	// Anthropic-Messages-API-compatible proxies the catalog lists with wire
	// type "anthropic". Unlike Anthropic's own template, the catalog gives
	// each of these a literal endpoint; they are reproduced here as
	// doc-verified constants (internal/catalog/providers.json) rather than
	// re-reading the catalog on every client build.
	KimiCodingBaseURL   = "https://api.kimi.com/coding"
	MiniMaxBaseURL      = "https://api.minimax.io/anthropic"
	MiniMaxChinaBaseURL = "https://api.minimaxi.com/anthropic"
)
View Source 
const (
	AnthropicProviderID    = "anthropic"
	KimiCodingProviderID   = "kimi-coding"
	MiniMaxProviderID      = "minimax"
	MiniMaxChinaProviderID = "minimax-china"
)

Provider identities for the four catalog entries whose wire type is "anthropic". NewProviderClient selects the Anthropic dialect by checking a provider's identity against this set, not by reading the catalog's wire type string — see NewProviderClient's dialect-selection comment in deepseek.go for why that distinction is load-bearing.

View Source 
const (
	// DeepSeekBaseURL is the documented OpenAI-format endpoint. DeepSeek also
	// serves an Anthropic-format surface at /anthropic, which sonar does not use.
	DeepSeekBaseURL = "https://api.deepseek.com"

	// DeepSeekAPIKeyEnv names the environment variable holding the key. Only
	// the NAME is ever configured; the value is read from the process
	// environment so a secret can never land in YAML.
	DeepSeekAPIKeyEnv = "DEEPSEEK_API_KEY"

	// DeepSeekFlashModel is the model sonar pins to. Its current published
	// build is DeepSeek-V4-Flash-0731 (284B total / 13B active).
	DeepSeekFlashModel = "deepseek-v4-flash"

	// DeepSeekContextWindow is the 1M-token context DeepSeek serves by default.
	DeepSeekContextWindow = 1_000_000

	// DeepSeekMaxOutputTokens is the published generation ceiling.
	DeepSeekMaxOutputTokens = 384_000

	// DeepSeekDefaultEffort matches the API default for ordinary requests.
	// "low" and "medium" are mapped to "high" server-side; "xhigh" maps to "max".
	DeepSeekDefaultEffort = "high"

	// DeepSeekMaxEffort is the deepest setting, which DeepSeek selects
	// automatically for some agent clients.
	DeepSeekMaxEffort = "max"
)

DeepSeek is sonar's only inference backend. Every fact below is taken from the official API contract (api-docs.deepseek.com) rather than inferred from the generic OpenAI shape, because three of them differ in ways that break an agent loop:

  • Thinking is toggled by `{"thinking": {"type": ...}}`, not by reasoning_effort. reasoning_effort only grades depth once thinking is on.
  • Thinking defaults to ENABLED. A harness that never sends the toggle pays for chain-of-thought on every turn.
  • An assistant message that carries tool calls must echo its own reasoning_content back on all later requests, or the API answers 400.

See OpenAICompatibleClient's DialectDeepSeek handling for the wire details.

View Source 
const (
	DeepSeekInputCacheHitUSDPerMTok  = 0.0028
	DeepSeekInputCacheMissUSDPerMTok = 0.14
	DeepSeekOutputUSDPerMTok         = 0.28
)

DeepSeekPricing is the published per-million-token rate card, in USD. Cache hits are ~50x cheaper than misses, so prompt-prefix stability is the single biggest cost lever in a long agent session.

DeepSeek has announced peak/off-peak pricing that doubles every billing item during 09:00–12:00 and 14:00–18:00 Beijing time (UTC+8). These constants are the off-peak/regular rates; any cost surface must present them as an estimate, never as a settled charge.

View Source 
const DialectDeepSeek = "deepseek"

DialectDeepSeek marks the DeepSeek flavor of the OpenAI chat contract.

Variables

View Source 
var (
	// ErrNoModelSelected is a local preflight rejection. No provider request or
	// generation can have started when a Client returns this sentinel.
	ErrNoModelSelected = errors.New("no model selected")

	// ErrInferenceNotStarted identifies a host-side rejection that happened
	// before the provider's inference dispatch. Provider ChatStream errors are
	// intentionally not wrapped with this sentinel because dispatch may already
	// have happened by then.
	ErrInferenceNotStarted = errors.New("inference not started")
)
View Source 
var ErrStreamIdle = errors.New("provider stream produced no data within the idle timeout")

ErrStreamIdle reports that an accepted provider stream stopped producing data for longer than the idle watchdog allows. It is retryable.

Functions

func EstimateDeepSeekCostUSD 

func EstimateDeepSeekCostUSD(promptTokens, cachedPromptTokens, completionTokens int) float64

EstimateDeepSeekCostUSD returns the estimated turn cost from a token receipt. cachedPromptTokens is the cache-hit portion of promptTokens; pass 0 when the provider did not report one. The result is an estimate: it does not know whether the request landed in a peak-pricing window.

func IsAnthropicFamilyProvider 

func IsAnthropicFamilyProvider(providerType string) bool

IsAnthropicFamilyProvider reports whether providerType (already normalized by config.NormalizedProviderType) is one of the four catalog providers that speak the Anthropic Messages API.

func IsRemoteInferenceError 

func IsRemoteInferenceError(err error) bool

IsRemoteInferenceError reports whether err carries exact provenance from a dispatched inference request against a credentialed remote provider. It deliberately does not classify local Ollama or host-side preflight failures, and never decides provenance by matching error strings.

func IsRetryableProviderError 

func IsRetryableProviderError(err error) bool

IsRetryableProviderError reports whether a dispatched provider request failed in a way that retrying the same request could plausibly resolve. It is a superset of IsRetryableTransport that additionally admits 429 (rate limit / quota exhaustion).

The 429 case is deliberately not folded into IsRetryableTransport: that predicate signals "safe to resend now with the same request," which is true for a dropped connection or a provider 5xx but never true for a rate limit — an immediate resend on a 429 wastes another billable request against a quota that is, by definition, already exhausted or throttled. A caller that receives true here for a 429 must consult ProviderRetryAfter (or otherwise back off) before resending; it must not treat this the same as an IsRetryableTransport-style immediate retry.

Every other 4xx status (401, 403, 404, 400, ...) returns false: the exact same request cannot succeed without changing credentials, the endpoint, or the payload, so retrying it is never productive.

func IsRetryableTransport 

func IsRetryableTransport(err error) bool

IsRetryableTransport reports whether err looks like a transient provider transport failure (connection loss, truncated stream, server-side 5xx, a request timeout, or an idle-stream watchdog) that a caller may retry immediately with the same request. Deliberate cancellation and admission deadlines are never retryable.

A 429 (rate limit / quota exhaustion) is deliberately NOT included here, even though it is sometimes eventually retryable: unlike a 5xx or a dropped connection, resending immediately will not resolve it and only burns another billable request against an already-exhausted quota. Callers that want to retry a 429 must do so through IsRetryableProviderError, which forces them to consider ProviderRetryAfter (or their own backoff) instead of treating it the same as an ordinary transport hiccup.

func ParseBaseURL 

func ParseBaseURL(rawURL string) (*url.URL, error)

ParseBaseURL validates and normalizes an Ollama URL.

func ProviderHTTPStatus 

func ProviderHTTPStatus(err error) (int, bool)

ProviderHTTPStatus reports the HTTP status a provider returned, when the error carries one. It recognizes both HTTP-error shapes this package produces: the OpenAI/Anthropic-dialect openAIHTTPError and Ollama's own ollamaHTTPError, so callers get one status lookup regardless of which client dispatched the request.

The status code alone is a bounded, host-safe fact: unlike the response body it cannot contain provider prose, an endpoint, or a credential. That makes it the one detail a startup diagnostic can surface without weakening the transcript sanitization that ProviderFailureCopy exists to enforce.

func ProviderRetryAfter 

func ProviderRetryAfter(err error) (time.Duration, bool)

ProviderRetryAfter reports the delay a provider asked for via a Retry-After response header, when the error carries one. It is populated for both HTTP-error shapes this package produces (openAIHTTPError and ollamaHTTPError), so it works uniformly across every dialect.

A caller that wants to retry a 429 (see IsRetryableProviderError) must honor this delay — or apply its own conservative backoff when ok is false — rather than resending immediately. Resending a rate-limited or quota-exhausted request with no delay cannot succeed and only spends another billable request.

func ProviderStatusHint 

func ProviderStatusHint(status int) string

ProviderStatusHint maps a provider HTTP status to an actionable cause. The text is host-authored, never the provider's.

func ResolveProviderModel 

func ResolveProviderModel(providerType, model string) (string, error)

ResolveProviderModel fills in a provider's default model when the caller left it empty, and otherwise passes the requested model through.

sonar runs many models; DeepSeek Flash is only the default. An id the catalog does not list is accepted rather than rejected: the catalog is a pinned snapshot, so refusing unlisted models would make the harness unusable the day a provider ships a new one. The cost is that such a model has no catalog-derived context window or pricing, and surfaces that depend on those must degrade rather than assume — see catalog.FindModel's miss path.

func SanitizeImageName 

func SanitizeImageName(value string) string

SanitizeImageName removes directory components, control characters, and bidirectional-format characters from display metadata. It retains ordinary Unicode and punctuation because the bounded name is never used as a path.

Types

type AnthropicClient 

type AnthropicClient struct {
	// contains filtered or unexported fields
}

AnthropicClient is a streaming chat adapter for the Anthropic Messages API (https://api.anthropic.com/v1/messages) and its wire-compatible proxies. Four catalog providers share this exact contract — see NewAnthropicProviderClient.

The Messages API differs from the OpenAI chat-completions shape in ways that break a naive OpenAI-compatible client:

  • The system prompt is a top-level "system" field, not a message with role "system".
  • Message content is an array of typed content blocks ("text", "tool_use", "tool_result", "image"), not a plain string.
  • Tool use arrives as tool_use content blocks; results are sent back as tool_result content blocks inside a user-role message.
  • Streaming uses a distinct SSE event vocabulary (message_start, content_block_start/delta/stop, message_delta, message_stop) instead of OpenAI's choices[].delta shape.
  • Auth is the x-api-key header plus a required anthropic-version header, not "Authorization: Bearer".
  • max_tokens is required on every request; OpenAI treats it as optional.

Credentials are supplied by the process environment only — never from config files.

func NewAnthropicClient 

func NewAnthropicClient(opts AnthropicOptions) (*AnthropicClient, error)

NewAnthropicClient builds a client against baseURL (for example https://api.anthropic.com). It performs no provider-identity resolution — callers that know a catalog provider identity should prefer NewAnthropicProviderClient, which also supplies doc-verified endpoint fallbacks and a catalog-derived default max_tokens.

func NewAnthropicProviderClient 

func NewAnthropicProviderClient(providerID, baseURL, model, apiKey string) (*AnthropicClient, error)

NewAnthropicProviderClient builds the Anthropic Messages API client for one of the four anthropic-family catalog providers. model must already be resolved (see ResolveProviderModel); apiKey must already be resolved from the process environment by the caller.

baseURL is whatever the caller's configuration resolved. When it is empty, or is an unresolved catalog template (Catwalk's "anthropic" entry names its endpoint as the literal string "$ANTHROPIC_API_ENDPOINT"), this falls back to the provider's real, doc-verified endpoint instead of guessing.

func (*AnthropicClient) BaseURL 

func (c *AnthropicClient) BaseURL() string

BaseURL returns the configured Anthropic base URL for display.

func (*AnthropicClient) ChatStream 

func (c *AnthropicClient) ChatStream(ctx context.Context, opts ChatOptions, fn func(StreamChunk) error) error

ChatStream streams a chat completion via the Anthropic Messages SSE protocol.

func (*AnthropicClient) Embed 

func (c *AnthropicClient) Embed(context.Context, string, []string) ([][]float32, error)

Embed is not implemented for the Anthropic Messages API adapter: Anthropic serves no embeddings endpoint. ICE and local embeddings continue to use Ollama when enabled.

func (*AnthropicClient) Model 

func (c *AnthropicClient) Model() string

func (*AnthropicClient) Ping 

func (c *AnthropicClient) Ping() error

Ping checks that the API key and model are usable.

func (*AnthropicClient) PingContext 

func (c *AnthropicClient) PingContext(ctx context.Context) error

PingContext prefers GET /v1/models when available, then falls back to a minimal non-streaming Messages request so proxies without a models list still work.

func (*AnthropicClient) SetModel 

func (c *AnthropicClient) SetModel(model string) error

SetModel updates the model id for subsequent requests.

type AnthropicOptions 

type AnthropicOptions struct {
	BaseURL string
	Model   string
	APIKey  string
	// MaxTokens is the max_tokens value sent when ChatOptions.MaxEvalTokens is
	// zero. Defaults to AnthropicDefaultMaxOutputTokens when zero or negative.
	MaxTokens int
}

AnthropicOptions constructs a remote Anthropic Messages API client.

type ChatOptions 

type ChatOptions struct {
	Messages      []Message
	Tools         []ToolDef
	System        string
	MaxEvalTokens int // zero leaves provider generation uncapped
	// DisableReasoning asks providers with a native control to emit only the
	// visible answer. It is used for bounded child-expert reports where private
	// reasoning is neither a valid result nor safe durable content. Providers
	// without such a control may ignore it; callers must still reject a terminal
	// response that contains no visible text.
	DisableReasoning bool
	// NumThread is a host-only local inference cap. Zero leaves the provider
	// default unchanged; positive values are sent as Ollama num_thread.
	NumThread int
	// ExpectedContext pins a host-side context budget to the request. Provider
	// managers use it to reject a turn whose model policy changed after the
	// agent took its budget snapshot. Direct clients may ignore it.
	ExpectedContext int
	// ExpectedModel pins the tokenizer/model identity paired with the prompt
	// estimate. ModelManager validates it atomically before dispatch; direct
	// clients may ignore it.
	ExpectedModel string
}

ChatOptions holds parameters for a chat request.

type Client 

type Client interface {
	// ChatStream sends messages to the LLM and streams the response.
	// The callback is called for each chunk. Return a non-nil error to abort.
	ChatStream(ctx context.Context, opts ChatOptions, fn func(StreamChunk) error) error

	// Ping checks if the LLM is reachable and the model is available.
	Ping() error

	// Model returns the current model name.
	Model() string

	// Embed generates embeddings for the given texts using the specified model.
	Embed(ctx context.Context, model string, texts []string) ([][]float32, error)
}

Client is the interface for LLM providers.

type DeepSeekOptions 

type DeepSeekOptions struct {
	// APIKey is the resolved secret value, read from the environment by the
	// caller. Required: sonar has no unauthenticated mode.
	APIKey string
	// Model defaults to DeepSeekFlashModel. Any other value is rejected.
	Model string
	// BaseURL overrides the endpoint for a gateway or proxy that speaks the
	// same dialect. Defaults to DeepSeekBaseURL.
	BaseURL string
	// Thinking enables chain-of-thought. Defaults to true via NewDeepSeekClient,
	// matching the API default.
	Thinking bool
	// ReasoningEffort grades thinking depth. Defaults to DeepSeekDefaultEffort.
	ReasoningEffort string
}

DeepSeekOptions configures the pinned DeepSeek client.

type ExpertModelResource 

type ExpertModelResource struct {
	Name          string
	WeightBytes   int64
	ResidentBytes int64
	ContextLength int
	Location      OllamaModelLocation
	Active        bool
	Resident      bool
	Current       bool
	Selected      bool
	ExpertOnly    bool
}

ExpertModelResource is one live Ollama model fact used by the expert admission planner. Inventory weights come from /api/tags and residency comes from /api/ps; Active is the manager's process-local usage record.

type ExpertModelSnapshot 

type ExpertModelSnapshot struct {
	Models            []ExpertModelResource
	InventoryVerified bool
	// contains filtered or unexported fields
}

ExpertModelSnapshot is a point-in-time union of every manager-active, Ollama-resident, current, and selected model. InventoryVerified means the selected weights were resolved from the live daemon rather than startup configuration. The lease field is intentionally opaque and process-local.

type ImageData 

type ImageData struct {
	SHA256    string `json:"sha256,omitempty"`
	Name      string `json:"name,omitempty"`
	MediaType string `json:"mime_type,omitempty"`
	Size      int64  `json:"size_bytes,omitempty"`
	Width     int    `json:"width,omitempty"`
	Height    int    `json:"height,omitempty"`
	Data      []byte `json:"-"`
}

ImageData is one image input for a multimodal model. Its durable fields form a path-free content-addressed reference; Data is transient provider input. Ollama's native API has no codec allowlist in its wire contract, so this layer validates a syntactic image/* media type without inventing a narrower set.

func NewImageData 

func NewImageData(mediaType string, data []byte) (ImageData, error)

NewImageData validates and copies an image payload for provider transport.

func NewReferencedImageData 

func NewReferencedImageData(name, mediaType string, width, height int, data []byte) (ImageData, error)

NewReferencedImageData creates a complete content-addressed reference and retains a defensive copy of the bytes for the current provider request. Callers are responsible for placing the same bytes in the resolver's backing store before persisting the returned reference.

func (ImageData) Validate 

func (image ImageData) Validate() error

Validate reports whether the image is safe to send through a provider adapter. Provider adapters call this even for struct literals so callers cannot bypass NewImageData's admission checks.

func (ImageData) ValidateReference 

func (image ImageData) ValidateReference() error

ValidateReference verifies the complete durable, path-free metadata. It does not require Data, allowing a restored session to validate before resolving the payload.

func (ImageData) WithData 

func (image ImageData) WithData(data []byte) (ImageData, error)

WithData hydrates a durable reference, verifying its content address and size before returning an independent provider-owned value.

type LocalModel 

type LocalModel struct {
	Name string
	Size int64
}

LocalModel records an Ollama model identity and the byte size of its local weights. Remote/cloud entries never appear in this inventory.

type Message 

type Message struct {
	Role    string `json:"role"` // system, user, assistant, tool
	Content string `json:"content"`
	// Images carry path-free, content-addressed metadata across a durable
	// session boundary. Raw Data remains provider-only and is never serialized.
	// Persistence writers must use agent.SanitizeMessagesForPersistence, which
	// drops transient images without a complete durable reference.
	Images []ImageData `json:"images,omitempty"`
	// DurableContent is the bounded replacement for transient tool content when
	// history crosses a persistence or compaction boundary. It is host-only:
	// providers receive Content, while JSON/session/checkpoint writers must call
	// agent.SanitizeMessagesForPersistence before serialization.
	DurableContent string `json:"-"`
	// ReasoningContent is the provider's chain-of-thought for this assistant
	// message. DeepSeek requires it echoed back verbatim on every later request
	// whenever the same message also carries ToolCalls; dropping it makes the
	// API reject the next agent iteration with HTTP 400. Host-only (`json:"-"`)
	// so private reasoning never crosses a session, transcript, or checkpoint
	// boundary.
	ReasoningContent string     `json:"-"`
	ToolCalls        []ToolCall `json:"tool_calls,omitempty"`
	ToolName         string     `json:"tool_name,omitempty"`
	ToolCallID       string     `json:"tool_call_id,omitempty"`
	// HostOwned marks a message whose exact contents were validated and
	// authored by the local host. It is deliberately not persisted or sent on
	// the wire: restore code must re-derive the marker from durable state, so a
	// user-authored message cannot forge host authority through JSON history.
	HostOwned bool `json:"-"`
}

Message represents a conversation message.

type ModelContextPolicy 

type ModelContextPolicy struct {
	Native      int
	Request     int
	Effective   int
	Cloud       bool
	NativeKnown bool
}

ModelContextPolicy is the context contract for one exact Ollama model. Native is the verified model maximum reported by Ollama. Request is the num_ctx value sent on the wire; zero deliberately omits num_ctx. Effective is the window host-side budgeting may rely on. Ollama Cloud models use their verified native maximum and omit num_ctx so the service keeps its documented maximum-by-default behavior.

type ModelManager 

type ModelManager struct {
	// contains filtered or unexported fields
}

func NewModelManager 

func NewModelManager(baseURL string, numCtx int) *ModelManager

func (*ModelManager) ActiveProviderName 

func (m *ModelManager) ActiveProviderName() string

ActiveProviderName is the catalog profile currently selected.

func (*ModelManager) BaseURL 

func (m *ModelManager) BaseURL() string

func (*ModelManager) ChatStream 

func (m *ModelManager) ChatStream(ctx context.Context, opts ChatOptions, fn func(StreamChunk) error) error

func (*ModelManager) ChatStreamForModel 

func (m *ModelManager) ChatStreamForModel(ctx context.Context, model string, opts ChatOptions, fn func(StreamChunk) error) error

func (*ModelManager) ClearCurrentModel 

func (m *ModelManager) ClearCurrentModel() error

ClearCurrentModel removes inference authority from the current selection. It is deliberately fail-closed: the selection is cleared before a best- effort unload, so an Ollama inventory reclassification cannot leave a remote model usable merely because releasing its resident runner failed.

func (*ModelManager) ClearRemoteProvider 

func (m *ModelManager) ClearRemoteProvider()

ClearRemoteProvider returns chat inference to the local Ollama path.

func (*ModelManager) Close 

func (m *ModelManager) Close()

func (*ModelManager) ConfigureLocalInventory 

func (m *ModelManager) ConfigureLocalInventory(required bool, models []LocalModel, verified bool)

ConfigureLocalInventory installs a verified local-weight inventory. The legacy name-only method intentionally records unknown sizes and therefore cannot admit inference in local-only mode.

func (*ModelManager) ConfigureLocalOnly 

func (m *ModelManager) ConfigureLocalOnly(required bool, models []string, verified bool)

ConfigureLocalOnly requires all inference model names to be proven by an Ollama inventory entry with local weights. An unverified inventory keeps the UI available for diagnostics, but no client can be selected until a later operation successfully refreshes the inventory.

func (*ModelManager) ConfigureOllamaCloudInventory 

func (m *ModelManager) ConfigureOllamaCloudInventory(models []string, verified bool)

ConfigureOllamaCloudInventory records exact cloud identities reported by the configured Ollama daemon. It never grants access by itself.

func (*ModelManager) ConfigureOllamaInventory 

func (m *ModelManager) ConfigureOllamaInventory(models []OllamaModel, verified bool)

ConfigureOllamaInventory installs the context and execution-location facts from one verified Ollama inventory snapshot. Model names are canonicalized so implicit :latest aliases share a single policy. An unverified snapshot clears prior facts instead of allowing stale cloud or native-context metadata to authorize later requests.

func (*ModelManager) ConfigureOllamaRuntimeInventory 

func (m *ModelManager) ConfigureOllamaRuntimeInventory(required bool, models []OllamaModel, verified bool)

ConfigureOllamaRuntimeInventory atomically installs all privacy-admission and context facts from one Ollama snapshot. Keeping local weights, cloud identity, grants, and native limits in one inference-serialized commit prevents a refresh from reclassifying a model between admission and request creation.

func (*ModelManager) ConfigureProviderCatalog 

func (m *ModelManager) ConfigureProviderCatalog(catalog config.ProviderConfig, localOnly bool, ollamaModel string)

ConfigureProviderCatalog installs the multi-profile provider definitions used by /provider. localOnly is the host privacy gate for remote base URLs.

func (*ModelManager) ConfigureRemoteProvider 

func (m *ModelManager) ConfigureRemoteProvider(client RemoteChatClient, contextSize int, label string) error

ConfigureRemoteProvider attaches a remote chat backend (OpenAI-compatible or Anthropic Messages API). When set, ordinary chat, ping, and model selection use the remote client. Local Ollama inventory, admission guards, and embeddings remain available for ICE.

func (*ModelManager) ConfiguredNumCtx 

func (m *ModelManager) ConfiguredNumCtx() int

ConfiguredNumCtx returns the host-configured local KV allocation (the value sent as options.num_ctx for local models), independent of cloud effective windows.

func (*ModelManager) ContextPolicy 

func (m *ModelManager) ContextPolicy(model string) ModelContextPolicy

ContextPolicy returns the current model-specific request and host-budget policy. Cloud status is derived only from a verified Ollama inventory; model name suffixes never grant cloud semantics.

func (*ModelManager) CurrentModel 

func (m *ModelManager) CurrentModel() string

func (*ModelManager) CurrentModelReceipt 

func (m *ModelManager) CurrentModelReceipt(ctx context.Context) ModelReceipt

CurrentModelReceipt inspects the active model's verified identity. It is a bounded read-only probe: inventory errors degrade to a receipt without digest/residency rather than failing the caller's settlement path.

func (*ModelManager) EffectiveContext 

func (m *ModelManager) EffectiveContext(model string) (int, bool)

EffectiveContext returns the current host-side context budget and whether it is known. Local requests always have an explicit effective allocation. Cloud requests are known only when their native maximum came from verified Ollama metadata.

func (*ModelManager) Embed 

func (m *ModelManager) Embed(ctx context.Context, model string, texts []string) ([][]float32, error)

func (*ModelManager) EmbedWithCurrentModel 

func (m *ModelManager) EmbedWithCurrentModel(ctx context.Context, texts []string) ([][]float32, error)

func (*ModelManager) GetClient 

func (m *ModelManager) GetClient(modelName string) (*OllamaClient, error)

func (*ModelManager) GrantOllamaCloudModel 

func (m *ModelManager) GrantOllamaCloudModel(model string) error

GrantOllamaCloudModel grants one verified cloud model for the current conversation. The grant is exact, ephemeral, and never covers remote-host models.

func (*ModelManager) ListLocalModelInventory 

func (m *ModelManager) ListLocalModelInventory(ctx context.Context) ([]LocalModel, error)

ListLocalModelInventory returns local identities with their actual weight sizes so memory admission never has to infer safety from a tag string.

func (*ModelManager) ListLocalModels 

func (m *ModelManager) ListLocalModels(ctx context.Context) ([]string, error)

ListLocalModels returns only models with local weights. Ollama cloud entries are deliberately excluded so a "local-only" routing decision cannot silently cross the network.

func (*ModelManager) ListOllamaModels 

func (m *ModelManager) ListOllamaModels(ctx context.Context) ([]OllamaModel, error)

ListOllamaModels returns the configured Ollama host's unfiltered inventory. Privacy and routing policy must be applied by the caller using Location and Capabilities; unlike ListLocalModels, cloud and remote-host entries remain.

func (*ModelManager) ListRunningOllamaModels 

func (m *ModelManager) ListRunningOllamaModels(ctx context.Context) ([]OllamaRunningModel, error)

func (*ModelManager) LocalModelWeightBytes 

func (m *ModelManager) LocalModelWeightBytes(model string) int64

LocalModelWeightBytes returns the cached local weight size for model when known from inventory.

func (*ModelManager) Model 

func (m *ModelManager) Model() string

func (*ModelManager) NumCtx 

func (m *ModelManager) NumCtx() int

func (*ModelManager) OllamaVersion 

func (m *ModelManager) OllamaVersion(ctx context.Context) (string, error)

func (*ModelManager) Ping 

func (m *ModelManager) Ping() error

func (*ModelManager) PingContext 

func (m *ModelManager) PingContext(ctx context.Context) error

PingContext checks the currently selected provider/model while preserving the caller's cancellation and deadline across the actual network request.

func (*ModelManager) PingModel 

func (m *ModelManager) PingModel(ctx context.Context, model string) error

func (*ModelManager) PrepareExpertModels 

func (m *ModelManager) PrepareExpertModels(ctx context.Context, selected []string) (ExpertModelSnapshot, error)

PrepareExpertModels serializes expert consultations sharing one manager, refreshes live weights and residency, and returns an opaque cleanup lease. Ordinary current-model and embedding calls share a context-aware admission boundary and cannot change residency until the lease is released.

func (*ModelManager) ProviderCatalog 

func (m *ModelManager) ProviderCatalog() []ProviderDescriptor

ProviderCatalog returns descriptors for every installed profile.

func (*ModelManager) ProviderNames 

func (m *ModelManager) ProviderNames() []string

ProviderNames lists configured profile names for /provider list.

func (*ModelManager) PullOllamaModel 

func (m *ModelManager) PullOllamaModel(ctx context.Context, model string, fn func(OllamaPullProgress) error) error

func (*ModelManager) ReleaseExpertModels 

func (m *ModelManager) ReleaseExpertModels(ctx context.Context, snapshot ExpertModelSnapshot) error

ReleaseExpertModels unloads only selected models that were not protected by pre-existing non-expert residency, are still non-current, and have not gained a non-expert process-local user since the snapshot. It always releases the consultation gate, even when one unload fails.

func (*ModelManager) RemoteProvider 

func (m *ModelManager) RemoteProvider() bool

RemoteProvider reports whether chat inference uses a non-Ollama adapter.

func (*ModelManager) RemoteProviderLabel 

func (m *ModelManager) RemoteProviderLabel() string

RemoteProviderLabel is a short UI/startup label (for example "xai").

func (*ModelManager) RevokeOllamaCloudGrants 

func (m *ModelManager) RevokeOllamaCloudGrants()

func (*ModelManager) RevokeOllamaCloudModel 

func (m *ModelManager) RevokeOllamaCloudModel(model string)

func (*ModelManager) SetCurrentModel 

func (m *ModelManager) SetCurrentModel(model string) error

func (*ModelManager) SetNumCtx 

func (m *ModelManager) SetNumCtx(numCtx int) error

SetNumCtx updates the local KV-cache allocation used for subsequent local Ollama requests. Cloud profiles keep their native maximum. The current local client is rebuilt so the next turn picks up the new window; in-flight turns keep their frozen snapshot.

func (*ModelManager) ShowOllamaModel 

func (m *ModelManager) ShowOllamaModel(ctx context.Context, model string) (OllamaModelInfo, error)

func (*ModelManager) SwitchProvider 

func (m *ModelManager) SwitchProvider(name string) error

SwitchProvider activates a named profile from the installed catalog. Remote profiles resolve API keys from the process environment at switch time.

func (*ModelManager) SwitchProviderContext 

func (m *ModelManager) SwitchProviderContext(ctx context.Context, name string) error

SwitchProviderContext activates a named profile while allowing callers to cancel admission and bounded local-inventory refresh work. Once the provider mutation begins it is committed atomically under the manager locks; a cancellation that arrives during the best-effort inventory refresh only shortens that refresh and does not report a half-applied switch.

type ModelReceipt 

type ModelReceipt struct {
	Name     string
	Digest   string
	Provider string
	Remote   bool
	// VRAMBytes/TotalBytes report weights residency from the runtime's process
	// inventory. VRAMBytes < TotalBytes means part of the model runs on CPU.
	VRAMBytes    int64
	TotalBytes   int64
	OffloadKnown bool
}

ModelReceipt binds a run to the exact inference artifact it used. Digest and residency come from the live local inventory; both stay empty/unknown for remote providers or when the runtime cannot be reached, so a consumer can always distinguish "not offloaded" from "not inspected".

type OllamaClient 

type OllamaClient struct {
	// contains filtered or unexported fields
}

OllamaClient is a deliberately small HTTP adapter for the local Ollama API. Keeping the wire client here avoids linking Ollama's server/runtime graph (GPU runners, archives, SSH helpers, and registry code) into this CLI.

func NewOllamaClient 

func NewOllamaClient(baseURL, model string, numCtx int) (*OllamaClient, error)

NewOllamaClient creates a new Ollama client without mutating process-wide environment state. baseURL wins over OLLAMA_HOST; otherwise the loopback default used by Ollama is selected.

func (*OllamaClient) BaseURL 

func (o *OllamaClient) BaseURL() string

BaseURL returns the configured Ollama base URL for display.

func (*OllamaClient) ChatStream 

func (o *OllamaClient) ChatStream(ctx context.Context, opts ChatOptions, fn func(StreamChunk) error) error

ChatStream sends a chat request and streams the response via callback.

func (*OllamaClient) Embed 

func (o *OllamaClient) Embed(ctx context.Context, model string, texts []string) ([][]float32, error)

Embed generates embeddings for the given texts using the specified model.

func (*OllamaClient) ListModels 

func (o *OllamaClient) ListModels(ctx context.Context) ([]OllamaModel, error)

ListModels returns every model reported by Ollama. Unlike ListLocalModelInventory, it intentionally does not apply privacy, memory, or static-catalog policy.

func (*OllamaClient) ListRunningModels 

func (o *OllamaClient) ListRunningModels(ctx context.Context) ([]OllamaRunningModel, error)

func (*OllamaClient) Model 

func (o *OllamaClient) Model() string

func (*OllamaClient) Ping 

func (o *OllamaClient) Ping() error

Ping checks Ollama is running and the model exists.

func (*OllamaClient) PingContext 

func (o *OllamaClient) PingContext(ctx context.Context) error

PingContext checks model availability within the caller's cancellation and deadline rather than creating an unrelated background operation.

func (*OllamaClient) PullModel 

func (o *OllamaClient) PullModel(ctx context.Context, model string, fn func(OllamaPullProgress) error) error

PullModel streams bounded Ollama pull progress. Cancellation propagates through the request context and callback errors stop the stream immediately.

func (*OllamaClient) ShowModel 

func (o *OllamaClient) ShowModel(ctx context.Context, model string) (OllamaModelInfo, error)

ShowModel obtains capability and model-native metadata without verbose token tables, keeping the response within the shared metadata bound.

func (*OllamaClient) Unload 

func (o *OllamaClient) Unload(ctx context.Context) error

Unload asks Ollama to evict this model immediately. It is used before an exclusive profile switch so two chat-model weight sets do not overlap in unified memory.

func (*OllamaClient) Version 

func (o *OllamaClient) Version(ctx context.Context) (string, error)

type OllamaModel 

type OllamaModel struct {
	Name          string
	Digest        string
	ModifiedAt    time.Time
	SizeBytes     int64
	RemoteModel   string
	RemoteHost    string
	Location      OllamaModelLocation
	Format        string
	Family        string
	Families      []string
	ParameterSize string
	Quantization  string
	ContextLength int64
	Capabilities  []string
}

OllamaModel is one inventory entry returned by the configured Ollama host. It preserves cloud stubs and custom models; policy filtering belongs above the wire layer.

type OllamaModelInfo 

type OllamaModelInfo struct {
	Model         OllamaModel
	Capabilities  []string
	ModelInfo     map[string]any
	Parameters    string
	Template      string
	System        string
	License       string
	NativeContext int64
}

OllamaModelInfo contains the authoritative metadata returned by /api/show.

type OllamaModelLocation 

type OllamaModelLocation string

OllamaModelLocation describes where Ollama executes a model. LocationLocal means /api/tags proved local weights; LocationCloud is an Ollama cloud stub. Unknown is deliberately retained instead of guessing from a model name. 

const (
	OllamaModelLocationUnknown OllamaModelLocation = "unknown"
	OllamaModelLocationLocal   OllamaModelLocation = "local"
	OllamaModelLocationCloud   OllamaModelLocation = "cloud"
	OllamaModelLocationRemote  OllamaModelLocation = "remote-host"
)

type OllamaPullProgress 

type OllamaPullProgress struct {
	Status    string `json:"status"`
	Digest    string `json:"digest,omitempty"`
	Total     int64  `json:"total,omitempty"`
	Completed int64  `json:"completed,omitempty"`
}

type OllamaRunningModel 

type OllamaRunningModel struct {
	Model         OllamaModel
	ExpiresAt     time.Time
	SizeVRAM      int64
	ContextLength int64
}

OllamaRunningModel is transient residency data from /api/ps.

type OpenAICompatibleClient 

type OpenAICompatibleClient struct {
	// contains filtered or unexported fields
}

OpenAICompatibleClient is a streaming chat adapter for OpenAI-compatible HTTP APIs (xAI, OpenAI, OpenRouter, local vLLM, etc.). Credentials are supplied by the process environment only — never from config files.

func NewDeepSeekClient 

func NewDeepSeekClient(opts DeepSeekOptions) (*OpenAICompatibleClient, error)

NewDeepSeekClient builds the pinned DeepSeek chat client.

func NewOpenAICompatibleClient 

func NewOpenAICompatibleClient(opts OpenAICompatibleOptions) (*OpenAICompatibleClient, error)

NewOpenAICompatibleClient builds a client against baseURL (for example https://api.x.ai/v1). The API key may be empty only for local open servers.

func (*OpenAICompatibleClient) BaseURL 

func (c *OpenAICompatibleClient) BaseURL() string

func (*OpenAICompatibleClient) ChatStream 

func (c *OpenAICompatibleClient) ChatStream(ctx context.Context, opts ChatOptions, fn func(StreamChunk) error) error

ChatStream streams a chat completion via the OpenAI SSE protocol.

func (*OpenAICompatibleClient) Embed 

func (c *OpenAICompatibleClient) Embed(context.Context, string, []string) ([][]float32, error)

Embed is not implemented for the generic OpenAI-compatible chat adapter. ICE and local embeddings continue to use Ollama when enabled.

func (*OpenAICompatibleClient) Model 

func (c *OpenAICompatibleClient) Model() string

func (*OpenAICompatibleClient) Ping 

func (c *OpenAICompatibleClient) Ping() error

Ping checks that the API key and model are usable.

func (*OpenAICompatibleClient) PingContext 

func (c *OpenAICompatibleClient) PingContext(ctx context.Context) error

PingContext prefers GET /models when available, then falls back to a minimal non-streaming chat completion so APIs without a models list still work.

func (*OpenAICompatibleClient) SetModel 

func (c *OpenAICompatibleClient) SetModel(model string) error

SetModel updates the model id for subsequent requests.

type OpenAICompatibleOptions 

type OpenAICompatibleOptions struct {
	BaseURL string
	Model   string
	APIKey  string
	// Dialect selects provider-specific request extensions. Empty is plain
	// OpenAI. DialectDeepSeek adds the `thinking` toggle and the
	// `reasoning_content` round-trip that DeepSeek's tool-call turns require.
	Dialect string
	// Thinking requests chain-of-thought for dialects that expose a toggle.
	// Ignored by dialects without one.
	Thinking bool
	// ReasoningEffort grades thinking depth ("high", "max") when Thinking is on.
	ReasoningEffort string
}

OpenAICompatibleOptions constructs a remote OpenAI-style client.

type ProviderDescriptor 

type ProviderDescriptor struct {
	Name       string
	Type       string
	Model      string
	APIKeyEnv  string
	Remote     bool
	Active     bool
	KeyPresent bool // process env has a non-empty value for APIKeyEnv
}

ProviderDescriptor is a UI-safe view of one catalog profile. It never includes secret values — only whether the configured env var is currently set.

type ProviderTiming 

type ProviderTiming struct {
	TimeToFirstToken   time.Duration
	TotalDuration      time.Duration
	LoadDuration       time.Duration
	PromptEvalDuration time.Duration
	EvalDuration       time.Duration
}

ProviderTiming carries the provider-reported request timings plus the client-measured time to first streamed token. Durations the provider did not report stay zero; consumers must treat zero as "not reported", never as "instant".

type RemoteChatClient 

type RemoteChatClient interface {
	Client

	// SetModel updates the model id for subsequent requests.
	SetModel(model string) error

	// PingContext checks reachability like Ping but honors ctx's
	// cancellation and deadline.
	PingContext(ctx context.Context) error
}

RemoteChatClient is the superset of Client that ModelManager needs to route ordinary chat, ping, and model switching through a single active remote provider (see ModelManager.ConfigureRemoteProvider). It exists so NewProviderClient can hand back distinct dialect implementations (the OpenAI-compatible client, the Anthropic Messages API client) through one interface without widening Client itself — every existing Client implementation, including the fakes littered across the agent/ice test suites, would otherwise be forced to grow SetModel/PingContext methods they never use.

func NewProviderClient 

func NewProviderClient(providerType, baseURL, model, apiKey string) (RemoteChatClient, error)

NewProviderClient builds the chat client for a resolved provider profile.

The dialect is chosen from the provider's identity, not from its endpoint shape. The catalog calls DeepSeek "openai-compat", which is true about the URL and wrong about the request contract: it needs a thinking toggle and a reasoning_content round-trip no generic OpenAI client sends. Selecting on wire type alone would produce a client that connects, answers once, and then fails every tool-call turn with a 400.

The anthropic-family branch below is the one place selecting by wire type would actually have been safe — Catwalk's "anthropic" type genuinely means "speaks the real Anthropic Messages API" for all four providers that carry it — but identity is still used, for the same reason as DeepSeek: it keeps dialect selection independent of how Catwalk chooses to classify a provider upstream, and it is one line longer than a wire-type switch would have been.

Both the startup path and the runtime /provider switch go through here so the two can never disagree about which dialect a provider gets.

type StreamChunk 

type StreamChunk struct {
	Text            string     // incremental text content
	Reasoning       string     // provider-native thinking/reasoning delta
	ToolCalls       []ToolCall // tool calls (usually in final chunk)
	Done            bool       // true on the last chunk
	EvalCount       int        // tokens generated (only on Done)
	PromptEvalCount int        // prompt tokens evaluated (only on Done)
	// FinishReason is the provider's terminal reason (only on Done): "stop",
	// "tool_calls", or "length" for a truncated generation. Empty when the
	// provider did not report one — a truncated response must therefore be
	// detected by "length", never by the absence of "stop".
	FinishReason string
	// Timing is attached to the terminal chunk when any timing fact is known.
	Timing *ProviderTiming
}

StreamChunk is a piece of a streaming response.

type ToolBehavior 

type ToolBehavior struct {
	Declared    bool
	ReadOnly    bool
	Destructive bool
	Idempotent  bool
	OpenWorld   bool
}

ToolBehavior is the bounded presentation projection of standard MCP tool annotations. It is untrusted server metadata and must never by itself alter authorization, durable effect classification, or recovery semantics.

type ToolCall 

type ToolCall struct {
	ID        string         `json:"id"`
	Name      string         `json:"name"`
	Arguments map[string]any `json:"arguments"`
}

ToolCall represents a tool invocation requested by the LLM.

type ToolDef 

type ToolDef struct {
	Name        string         `json:"name"`
	Description string         `json:"description"`
	Parameters  map[string]any `json:"parameters"` // JSON Schema
	// DisplayName and Behavior are host-only MCP presentation metadata.
	// They must never be sent to model providers as part of a tool schema.
	DisplayName string       `json:"-"`
	Behavior    ToolBehavior `json:"-"`
}

ToolDef defines a tool the LLM can call.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.