llm

package
v0.1.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 13, 2026 License: Apache-2.0 Imports: 18 Imported by: 0

Documentation

Overview

Package llm is the model provider boundary.

One interface, several backends. §10.1 requires this rather than a direct Anthropic wrapper: token-mode budgeting is justified partly by users on self-hosted models, and hard-wiring one vendor would make that claim false.

Every call returns Usage, because the budget ledger settles against it. A provider that cannot report tokens cannot be used in a metered session — see §4.2 for the delegated mode where that is true and what replaces the budget there.

Index

Constants

View Source 
const (
	DefaultAnthropicStrong = "claude-opus-5"
	DefaultAnthropicCheap  = "claude-haiku-4-5"
)

Defaults for the Anthropic backend. Opus 5 for reasoning, Haiku 4.5 for the per-chunk extraction work that runs an order of magnitude more often.

View Source 
const (
	MaxReasoningAllowance = 16384
)

Reasoning allowance bounds.

MaxReasoningAllowance caps what one model may be given for thinking: a model that has not produced an answer in this many tokens is not about to, and a token budget that grows without a ceiling is not a budget.

firstReasoningAllowance is what a first empty response jumps to, and it is measured rather than chosen. qwen3:4b asked for the JSON `{"ok":true}` — the smallest useful prompt there is — spent 1,924 completion tokens on its chain in one run and more than 2,128 in the next, on the same prompt. A model that reasons at all reasons in thousands, and stepping there 512 tokens at a time means paying for four wasted calls to learn what one can.

maxReasoningAttempts bounds the paid discovery at three provider calls.

Variables

View Source 
var (
	// ErrRateLimited is transient; the executor backs off and retries (§9.5).
	ErrRateLimited = errors.New("llm: rate limited")
	// ErrUnauthorized is fatal. Retrying a bad credential burns wall-clock and
	// never succeeds.
	ErrUnauthorized = errors.New("llm: unauthorized")
	// ErrQuotaExceeded is fatal for this session.
	ErrQuotaExceeded = errors.New("llm: quota exceeded")
	// ErrOverloaded is transient.
	ErrOverloaded = errors.New("llm: provider overloaded")
	// ErrNoUsageReported means the provider returned a completion without
	// token counts. Treated as an error rather than a zero charge: a silent
	// zero would make the budget ceiling unenforceable for that provider.
	ErrNoUsageReported = errors.New("llm: provider reported no token usage")
	// ErrContextTooLong means the request exceeded the model's window. The
	// chunker's job is to prevent this; when it happens anyway the actor
	// re-splits rather than failing the lead.
	ErrContextTooLong = errors.New("llm: context too long")

	// ErrEmptyOutput means the provider returned a successful response with no
	// content.
	//
	// The case that produces it in practice is a reasoning model: qwen3, gemma4
	// and others emit a `reasoning` field that mole does not read, and it is
	// charged against the same output budget. Ask for too few tokens and the
	// whole allowance goes to reasoning, leaving content empty with
	// finish_reason "length" — a successful call that returned nothing.
	//
	// Worth its own sentinel because the symptom is otherwise a JSON parse
	// failure, which sends whoever reads the log looking at the prompt instead
	// of at MaxTokens.
	ErrEmptyOutput = errors.New("llm: provider returned empty content")
)

Functions

func EstimateTokens 

func EstimateTokens(chars int) int64

EstimateTokens approximates a token count from characters.

Deliberately rough and deliberately high: it is used to decide how many chunks fit a sub-budget BEFORE any call is made, and over-estimating stops early while under-estimating overspends. Actual charges always come from the provider's reported usage, never from this.

func IsLoopback 

func IsLoopback(baseURL string) bool

IsLoopback reports whether a base URL points at this machine.

func Reachable 

func Reachable(ctx context.Context, baseURL string, client *http.Client) bool

Reachable probes an OpenAI-compatible endpoint.

Only meaningful for local runtimes, where the check is free and instant. A hosted provider cannot be verified without spending money, so `doctor` reports those as configured-but-unverified and points at `config test-llm` rather than printing a green tick it has not earned.

func Retryable 

func Retryable(err error) bool

Retryable reports whether any error is worth another attempt.

Types

type APIError 

type APIError struct {
	Provider string
	Status   int
	Body     string
	// contains filtered or unexported fields
}

APIError carries provider detail alongside a sentinel.

func (*APIError) Error 

func (e *APIError) Error() string

func (*APIError) Retryable 

func (e *APIError) Retryable() bool

Retryable reports whether the executor should back off rather than fail.

func (*APIError) Unwrap 

func (e *APIError) Unwrap() error

type Chunk 

type Chunk struct {
	Text  string
	Start int // byte offset into the source
	End   int
	Index int
}

Chunk is one contiguous span of the source.

func Split 

func Split(text string, opts ChunkOptions) []Chunk

Split divides text into chunks.

Offsets index the ORIGINAL string, so a quote found in a chunk can be located in the source without re-searching — which is what keeps quote verification exact for documents that were never held in one context.

type ChunkOptions 

type ChunkOptions struct {
	// MaxChars is the target ceiling per chunk. Characters rather than tokens
	// because the tokenizer differs per provider and an estimate that runs
	// long produces a request rejection rather than a slightly larger bill.
	MaxChars int
	// OverlapChars repeats the tail of the previous chunk. A claim near a
	// boundary is otherwise visible in neither chunk with full context.
	OverlapChars int
	// MinChars avoids emitting a final sliver that costs a model call and
	// carries nothing.
	MinChars int
}

ChunkOptions tune splitting.

func DefaultChunkOptions 

func DefaultChunkOptions() ChunkOptions

DefaultChunkOptions targets roughly 6k tokens of input per chunk on typical English prose, leaving generous room under any current context window.

type Config 

type Config struct {
	Kind Kind

	// APIKey may be empty for Anthropic, in which case the SDK's own
	// credential chain resolves it — an env var, then an `ant auth login`
	// profile. That is what lets a machine with a profile need no
	// configuration at all (§4.2).
	APIKey string

	// BaseURL overrides the endpoint. Required for OpenAI-compatible backends.
	BaseURL string

	// StrongModel and CheapModel resolve the two tiers.
	StrongModel string
	CheapModel  string

	MaxRetries int
	Timeout    time.Duration
}

Config selects and configures a provider.

type CredentialSource 

type CredentialSource string

CredentialSource records how a provider resolved its credential, so `doctor` can answer "which one is this actually using" without guessing. 

const (
	CredentialConfig    CredentialSource = "config"
	CredentialEnv       CredentialSource = "env"
	CredentialChain     CredentialSource = "sdk credential chain"
	CredentialNotNeeded CredentialSource = "none required"
)

func SourceOf 

func SourceOf(p Provider) CredentialSource

SourceOf reports how a provider authenticated, or CredentialConfig when the provider does not track it.

type Detected 

type Detected struct {
	Config Config
	// Reason is shown to the user, so it has to name what was found rather
	// than just asserting success.
	Reason string
	// Free is true for local runtimes, where token-mode budgeting is the only
	// meaningful ceiling because no money changes hands.
	Free bool
}

Detected describes a provider found without configuration.

func Detect 

func Detect(ctx context.Context, client *http.Client) (*Detected, bool)

Detect looks for a usable provider without any configuration.

Order matters: an explicitly configured hosted key beats a local model, because someone who set a key meant to use it. Local comes last as the zero-cost fallback that makes Mole runnable with no account at all.

type Kind 

type Kind string

Kind names a backend. 

const (
	KindAnthropic Kind = "anthropic"
	// KindOpenAICompatible covers DeepSeek, Ollama, llama.cpp, vLLM, LiteLLM,
	// Together, and anything else speaking the same wire format.
	KindOpenAICompatible Kind = "openai-compatible"
)

func Kinds 

func Kinds() []Kind

func (Kind) Valid 

func (k Kind) Valid() bool

type LocalEndpoint 

type LocalEndpoint struct {
	Name    string
	BaseURL string
	// Probe is the path that lists models; a 200 means something is listening
	// and speaking the right dialect.
	Probe string
}

LocalEndpoint is a runtime worth probing on localhost.

func LocalEndpoints 

func LocalEndpoints() []LocalEndpoint

LocalEndpoints are the runtimes checked, in order.

type MapReduce 

type MapReduce struct {
	Chunks []Chunk
	// Truncated is set when the sub-budget could not cover every chunk.
	Truncated bool
	// Skipped counts chunks dropped for budget.
	Skipped int
}

MapReduce is the plan for summarizing a document that does not fit.

It is a plan rather than an executor because the actor owns the budget: the actor asks how many chunks it can afford, runs that many, and records Truncated rather than silently dropping content or blowing the reservation (§4.1).

func Plan 

func Plan(text string, maxInputTokens int64, opts ChunkOptions) MapReduce

Plan splits text and decides how much of it the sub-budget can cover.

maxInputTokens is the sub-budget derived from the lead's reservation. When the document exceeds it, the highest-value chunks are kept — which here means the earliest, since article structure front-loads the substance and a truncation that kept the middle would read as incoherent.

type MapResult 

type MapResult struct {
	Chunk Chunk
	Text  string
	Usage Usage
	Err   error
}

MapResult is one chunk's output.

type Message 

type Message struct {
	Role Role
	Text string
}

Message is one conversational turn.

func Assistant 

func Assistant(text string) Message

func User 

func User(text string) Message

type Provider 

type Provider interface {
	Complete(ctx context.Context, req Request) (*Response, error)
	// Name identifies the backend for logs and diagnostics.
	Name() string
	// ModelFor reports which model a tier resolves to, so cost estimates and
	// `doctor` can name it without making a call.
	ModelFor(tier Tier) string
}

Provider is a model backend.

func New 

func New(cfg Config, httpClient *http.Client) (Provider, error)

New builds the configured provider.

httpClient carries the record/replay cassette transport, so model calls are deterministic and cost nothing in tests and in the eval harness. Passing nil uses the default client.

type Request 

type Request struct {
	// Tier picks the model when Model is empty.
	Tier Tier
	// Model overrides the tier's default.
	Model string

	System   string
	Messages []Message

	MaxTokens int
	// Effort maps to the provider's reasoning-depth control where it has one.
	Effort string

	// Thinking asks for extended reasoning. Off for extraction work, where it
	// adds latency and tokens without improving a mechanical task.
	Thinking bool
}

Request is one completion.

type Response 

type Response struct {
	Text       string
	Model      string
	StopReason string
	Usage      Usage
	Elapsed    time.Duration

	// Reasoning is a reasoning model's chain of thought, kept separate from Text
	// and never concatenated into it: it is not the answer, it is not quotable
	// evidence, and §11.5 would let a model cite its own reasoning as a source if
	// the two were joined.
	//
	// Carried rather than discarded so a caller can log what a model spent its
	// allowance on when the answer came back thin.
	Reasoning string
	// ReasoningTokens is what the chain cost, where the provider reports it or it
	// can be attributed. NOT part of Usage: providers count reasoning inside
	// completion_tokens, so adding it to the ledger's output count would charge
	// the same tokens twice.
	ReasoningTokens int64
	// Attempts is how many provider calls produced this response. More than one
	// means the first was spent entirely on reasoning and the ceiling was raised
	// — Usage covers every attempt, because the tokens were spent either way.
	Attempts int

	// Refused is set when the provider's safety classifiers declined the
	// request. It arrives as a successful HTTP response, so code that reads
	// Text without checking this gets an empty string and no error.
	Refused bool
	// RefusalCategory carries the provider's reason where one is given.
	RefusalCategory string
}

Response is one completion result.

type Role 

type Role string

Role is a message author. 

const (
	RoleUser      Role = "user"
	RoleAssistant Role = "assistant"
)

type SourcedProvider 

type SourcedProvider interface {
	Provider
	CredentialSource() CredentialSource
}

SourcedProvider is a Provider that can report how it authenticated.

type Tier 

type Tier string

Tier selects how much capability a call needs.

Splitting the two is a real cost lever: chunk mining runs once per chunk and is mostly extraction, while planning and synthesis run once per lead or session and carry the reasoning. Using one model for both either overpays on the many calls or underperforms on the few that matter. 

const (
	// TierCheap handles chunk summarization and claim mining.
	TierCheap Tier = "cheap"
	// TierStrong handles planning, verification, and report synthesis.
	TierStrong Tier = "strong"
)

type Usage 

type Usage struct {
	InputTokens      int64
	OutputTokens     int64
	CacheReadTokens  int64
	CacheWriteTokens int64
}

Usage is the token accounting a provider reports.

This is not optional. The ledger charges against these numbers, and a provider that returns zeros silently makes the session's ceiling unenforceable — which is why Complete rejects a response with no usage on a non-empty completion.

func (Usage) Add 

func (u Usage) Add(o Usage) Usage

Add sums two usages.

Needed because one Complete can now be more than one provider call: a reasoning model whose first answer was squeezed out by its own chain of thought is retried with more room, and both calls spent tokens the ledger has to charge.

func (Usage) IsZero 

func (u Usage) IsZero() bool

func (Usage) Total 

func (u Usage) Total() int64

type Vendor 

type Vendor struct {
	Name    string
	Kind    Kind
	BaseURL string
	// Example models, so an error message can be acted on rather than
	// researched. Not defaults: picking a model for someone silently is how a
	// session runs against something they did not choose.
	Models string
}

Vendor describes where a key came from and what it needs to work.

func VendorFromKey 

func VendorFromKey(key string) (Vendor, bool)

VendorFromKey identifies the provider a key belongs to.

This exists to catch one specific false green: llm.provider defaults to Anthropic when unset, so a Groq or OpenAI key set on its own produces a doctor line reading "✓ claude-opus-5 via config" and then an auth failure at the first model call — potentially many leads into a paid session.

Prefixes are a convention, not a guarantee, so an unrecognized key is not an error. Only a key that clearly belongs to a DIFFERENT vendor than the one configured is worth refusing.

Source Files

View all Source files

Directories

Path Synopsis
Package jsonish parses the JSON that models actually return.
 Package jsonish parses the JSON that models actually return.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.