provider

Documentation

Overview

Package provider is one client for every model runtime.

Ollama, LM Studio, Jan and Msty all speak OpenAI-compatible /v1, so the only thing that differs is the port and which models happen to be loaded. Cloud BYOK is the same surface with a base URL and a key.

Index

• Variables
• type Discovered
  • func Discover() []Discovered
• type LocalEndpoint
• type Msg
• type Provider
  • func New(name, baseURL, apiKey string) *Provider
  • func (p *Provider) Chat(model, system, user string, schema map[string]any) (string, error)
  • func (p *Provider) ChatStream(model string, messages []Msg, onToken func(string)) (string, error)
  • func (p *Provider) Embed(model string, inputs []string) ([][]float32, error)

Variables

var LocalEndpoints = []LocalEndpoint{
	{"Ollama", "http://localhost:11434/v1"},
	{"LM Studio", "http://localhost:1234/v1"},
	{"Jan", "http://localhost:1337/v1"},
	{"Msty", "http://localhost:10000/v1"},
}

Probed in this order; first match wins when nothing is configured.

Types

type Discovered

type Discovered struct {
	Provider *Provider
	Models   []string
}

func Discover

func Discover() []Discovered

Discover probes the well-known ports and reports what is actually running. This is what lets the app say "found LM Studio running Qwen3" on first launch instead of presenting an empty endpoint configuration box.

type LocalEndpoint

type LocalEndpoint struct {
	Name string
	URL  string
}

LocalEndpoint is a runtime we know how to find without configuration.

type Msg

type Msg struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Msg is one turn in a conversation.

type Provider

type Provider struct {
	Name    string
	BaseURL string
	APIKey  string
	// Think controls how much a reasoning model reasons before answering:
	// "off", "low", "medium", "high" (empty = "low"). It matters because the
	// OpenAI-compatible /v1 endpoint gives a reasoning model no way to bound its
	// thinking, so it spends the whole token budget thinking and returns an empty
	// answer. When set (Ollama only), chat routes through the native /api/chat
	// endpoint, which honours it. "default" forces the plain /v1 path.
	Think string
	// contains filtered or unexported fields
}

func New

func New(name, baseURL, apiKey string) *Provider

func (*Provider) Chat

func (p *Provider) Chat(model, system, user string, schema map[string]any) (string, error)

Chat runs a completion. A non-nil schema enables constrained decoding: small local models are unreliable at tool-calling but near-perfect when JSON is enforced at the sampler, which is why every extraction path goes through here rather than through tool definitions.

func (*Provider) ChatStream

func (p *Provider) ChatStream(model string, messages []Msg, onToken func(string)) (string, error)

ChatStream runs a multi-turn completion and streams tokens to onToken as they arrive. This is what makes the assistant feel alive: a 40-second answer that appears word by word reads as thinking, while the same answer delivered in one silent lump reads as broken.

Returns the full assembled text so callers can persist the turn.

func (*Provider) Embed

func (p *Provider) Embed(model string, inputs []string) ([][]float32, error)