llm

package
v0.43.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 14, 2026 License: Apache-2.0 Imports: 16 Imported by: 0

Documentation

Overview

Package llm provides configuration types and public API for the thv llm command group, which bridges AI coding tools to OIDC-protected LLM gateways.

Two authentication modes are planned:

  • Proxy mode: a localhost reverse proxy that injects fresh OIDC tokens for tools that only accept static API keys (e.g. Cursor).
  • Token helper mode: thv llm token prints a fresh JWT to stdout, suitable for use as apiKeyHelper or auth.command in OIDC-capable tools (e.g. Claude Code).

Both modes are under active development; the corresponding CLI commands currently return not-implemented errors.

Configuration is persisted in ToolHive's config.yaml under the llm: key via the existing UpdateConfig() mechanism.

Index

Constants

View Source 
const (
	// DefaultProxyListenPort is the default port the localhost proxy listens on.
	DefaultProxyListenPort = 14000
)

Variables

View Source 
var ErrTokenRequired = errors.New(
	"LLM gateway authentication required: no cached credentials found; " +
		"run \"thv llm setup\" to log in",
)

ErrTokenRequired is returned when a fresh token is needed but no cached or refreshable token exists and the caller is non-interactive (browser flow disabled). The user must first complete an interactive login so that a refresh token is persisted for subsequent non-interactive calls.

Functions

func DeleteCachedTokens 

func DeleteCachedTokens(ctx context.Context, provider pkgsecrets.Provider) error

DeleteCachedTokens removes all cached OIDC tokens stored under the LLM scope via the provided secrets provider. It is a no-op if the provider does not support listing or deletion (e.g. the environment provider), since such providers cannot hold cached tokens.

func DeriveSecretKey 

func DeriveSecretKey(gatewayURL, issuer string) string

DeriveSecretKey computes the secrets-provider key for an LLM gateway refresh token. The formula is: LLM_OAUTH_<8 hex chars> where the hex is derived from sha256(gatewayURL + "\x00" + issuer)[:4].

func PurgeTokens added in v0.26.0 

func PurgeTokens(ctx context.Context, errOut io.Writer, provider pkgsecrets.Provider)

PurgeTokens deletes all cached OIDC tokens from the provided secrets provider. Errors are logged as warnings rather than returned.

func SanitizeTokenError 

func SanitizeTokenError(err error) string

SanitizeTokenError returns a log-safe string for a token-source error. If err wraps *oauth2.RetrieveError, only the error code and description are included — never the raw response body, which may contain bearer material echoed back by the IdP.

func Setup added in v0.26.0 

func Setup(
	ctx context.Context, out, errOut io.Writer,
	gm GatewayManager, provider ConfigUpdater, login LoginFunc,
	inlineOpts SetOptions, anthropicPathPrefix string, anthropicPathPrefixSet bool, targetClient string,
	lazy bool,
) error

Setup configures detected AI tools to use the LLM gateway.

When targetClient is non-empty only that client is configured; an error is returned if the client is not installed. Pass an empty string to configure all detected clients (the original behaviour).

It applies inlineOpts in-memory before login so a failed login leaves no persisted state. Tool config files are patched only after login succeeds; on any persistence failure the patches are rolled back.

When lazy is true, the interactive OIDC login (login) is skipped entirely: tool detection, config-file patching, and config persistence still run, and a message is printed telling the user that login will occur on first gateway access. lazy is intended for unattended provisioning (e.g. an MDM profile); it is opt-in and does not change behaviour for interactive users.

func Teardown added in v0.26.0 

func Teardown(
	ctx context.Context,
	out, errOut io.Writer,
	gm GatewayManager,
	targetTool string,
	purgeTokens bool,
	provider ConfigUpdater,
	secretsProvider pkgsecrets.Provider,
) error

Teardown removes LLM gateway configuration from all (or one) configured tools.

targetTool selects which tool to revert; pass an empty string to revert all configured tools. An error is returned when targetTool is non-empty but not found in the configured tool list.

Reverting the last configured tool also resets the persisted LLM config to its zero value, so settings that are deliberately sticky across "thv llm setup" re-runs (e.g. Bedrock compat) are not silently re-applied by a later setup. A targeted teardown that leaves other tools configured keeps the config, which those tools still need to reach the gateway.

If secretsProvider is non-nil and purgeTokens is true, cached OIDC tokens are deleted after the config update succeeds.

Types

type BedrockConfig added in v0.39.0 

type BedrockConfig struct {
	// Compat, when true, writes CLAUDE_CODE_DISABLE_EXPERIMENTAL_BETAS=1 and the
	// per-tier Bedrock model IDs into Claude Code's settings.json. Bedrock rejects
	// Claude Code's experimental anthropic-beta headers, so this is required for a
	// Bedrock-backed gateway.
	Compat bool `yaml:"compat,omitempty" json:"compat,omitempty"`
	// Enable1M appends the "[1m]" suffix to the opus and sonnet model IDs to opt
	// into the 1M-token context window on Bedrock (never haiku, which is 200K).
	// Off by default: 1M-on-Bedrock is a version-dependent Claude Code behavior.
	Enable1M bool `yaml:"enable_1m,omitempty" json:"enable_1m,omitempty"`
}

BedrockConfig holds settings for configuring Claude Code to reach an LLM gateway that forwards to AWS Bedrock. It is persisted so that a later plain "thv llm setup" re-applies these settings rather than silently clearing them. That stickiness ends at teardown: reverting the last configured tool resets the whole LLM config (see Teardown), so it cannot outlive the tools that used it.

type Config 

type Config struct {
	GatewayURL    string        `yaml:"gateway_url,omitempty"       json:"gateway_url,omitempty"`
	TLSSkipVerify bool          `yaml:"tls_skip_verify,omitempty"   json:"tls_skip_verify,omitempty"`
	OIDC          OIDCConfig    `yaml:"oidc,omitempty"              json:"oidc,omitempty"`
	Proxy         ProxyConfig   `yaml:"proxy,omitempty"             json:"proxy,omitempty"`
	Bedrock       BedrockConfig `yaml:"bedrock,omitempty"           json:"bedrock,omitempty"`
	// Models is the persisted, single source of truth for the model IDs applied
	// during setup. It feeds two consumers: credential-helper clients (Claude
	// Desktop) write it verbatim as inferenceModels, and — when Bedrock compat is
	// on — each entry is also mapped to a Claude Code tier (see BedrockConfig).
	// Persisting it here (rather than passing a transient flag value) keeps both
	// consumers consistent on a later plain "thv llm setup".
	Models          []string     `yaml:"models,omitempty"            json:"models,omitempty"`
	ConfiguredTools []ToolConfig `yaml:"configured_tools,omitempty"  json:"configured_tools,omitempty"`
}

Config holds all LLM gateway settings persisted under the llm: key in ToolHive's config.yaml.

func (*Config) EffectiveProxyPort 

func (c *Config) EffectiveProxyPort() int

EffectiveProxyPort returns the configured proxy listen port, or DefaultProxyListenPort if none is set.

func (*Config) IsConfigured 

func (c *Config) IsConfigured() bool

IsConfigured reports whether the minimum required fields are present for the LLM gateway to be usable: gateway URL, OIDC issuer, and OIDC client ID.

func (*Config) SetFields 

func (c *Config) SetFields(opts SetOptions) error

SetFields applies the non-zero fields from the provided options to the config and validates the result. If the mandatory trio (gateway_url, oidc.issuer, oidc.client_id) is present after the update, full validation runs; otherwise only format/range validation runs to catch bad values early while still allowing incremental configuration.

func (*Config) Show 

func (c *Config) Show(w io.Writer) error

Show writes a human-readable representation of the config to w. If the config is not yet configured it prints a hint to run "config set".

func (*Config) Validate 

func (c *Config) Validate() error

Validate performs full validation of the LLM config, including HTTPS enforcement, port range checks, and OIDC field requirements.

func (*Config) ValidatePartial 

func (c *Config) ValidatePartial() error

ValidatePartial validates any fields that are explicitly set, without requiring the mandatory trio (gateway_url, oidc.issuer, oidc.client_id). Use this to catch URL format or port range errors during incremental configuration, before all required fields have been provided.

type ConfigUpdater added in v0.26.0 

type ConfigUpdater interface {
	// GetLLMConfig returns the current LLM section of the config.
	GetLLMConfig() Config
	// UpdateLLMConfig atomically reads, applies fn, and persists the LLM config.
	UpdateLLMConfig(fn func(*Config) error) error
}

ConfigUpdater is the subset of config.Provider used by Setup and Teardown. Defined here so pkg/llm does not import pkg/config.

type GatewayManager added in v0.26.0 

type GatewayManager interface {
	// DetectedLLMGatewayClients returns tool names for all installed LLM-gateway-capable tools.
	DetectedLLMGatewayClients() []string
	// ConfigureLLMGateway patches the tool's config file and returns the config path.
	ConfigureLLMGateway(clientType string, cfg llmgateway.ApplyConfig) (string, error)
	// LLMGatewayModeFor returns "direct", "proxy", or "" for the given client.
	LLMGatewayModeFor(clientType string) string
	// IsManaged reports whether a managed-preferences profile overrides the
	// client's local config (so the config setup writes would be ignored).
	IsManaged(clientType string) bool
	// ConfigureEnvFile writes .env file entries for the client and returns the
	// env file path. Returns ("", nil) when the client has no env-file entries.
	ConfigureEnvFile(clientType string, cfg llmgateway.ApplyConfig) (string, error)
	// RevertEnvFile removes the .env file entries that setup wrote. envFilePath
	// is the value returned by ConfigureEnvFile; a no-op when empty.
	RevertEnvFile(clientType, envFilePath string) error
	// RevertLLMGateway removes the LLM gateway settings from the tool's config file.
	RevertLLMGateway(clientType, configPath string) error
}

GatewayManager is the subset of client.ClientManager used by Setup and Teardown. Defined here so pkg/llm does not import pkg/client.

type LoginFunc added in v0.26.0 

type LoginFunc func(ctx context.Context, cfg *Config) error

LoginFunc performs the interactive OIDC login during setup. It is a parameter so that tests can inject a no-op without touching the keyring.

type OIDCConfig 

type OIDCConfig = pkgoidc.ClientConfig

OIDCConfig is a type alias for oidc.ClientConfig, holding OIDC provider settings and cached token state for the LLM gateway. Using a type alias ensures this type stays in sync with pkg/config.RegistryOAuthConfig, which is also an alias for the same underlying type.

type ProxyConfig 

type ProxyConfig struct {
	ListenPort int `yaml:"listen_port,omitempty" json:"listen_port,omitempty"`
}

ProxyConfig holds configuration for the localhost reverse proxy.

type SetOptions 

type SetOptions struct {
	GatewayURL    string
	Issuer        string
	ClientID      string
	Audience      string
	ProxyPort     int
	CallbackPort  int
	TLSSkipVerify *bool // nil = not provided; &false = explicitly disable
	// BedrockCompat and Enable1M use pointers so false can be distinguished from
	// "not provided" (enabling explicit clear via config set). See BedrockConfig.
	BedrockCompat *bool
	Enable1M      *bool
	// Models sets the persisted model IDs (Config.Models). nil = not provided
	// (leave existing config unchanged); an empty non-nil slice clears them.
	Models []string
}

SetOptions carries the flag values for the "config set" command. Zero values are treated as "not provided" and leave the existing config field unchanged. TLSSkipVerify uses a pointer so that false can be distinguished from "not provided" (enabling explicit clear via config set).

type TokenRefUpdater 

type TokenRefUpdater = tokensource.ConfigPersister

TokenRefUpdater is a callback invoked when the refresh token changes — either after a successful browser flow (initial login) or when the OIDC provider rotates the refresh token during a refresh. It persists the secret key and the new token expiry into the application config so future CLI invocations can restore the session. It is NOT called on routine access-token refreshes where the refresh token is unchanged. Callers typically wire this to config.UpdateConfig.

type TokenSource 

type TokenSource = tokensource.OAuthTokenSource

TokenSource provides fresh LLM gateway access tokens.

func NewTokenSource 

func NewTokenSource(
	cfg *Config, secretsProvider secrets.Provider, interactive, skipBrowser bool, tokenRefUpdater TokenRefUpdater,
) *TokenSource

NewTokenSource creates a TokenSource for the LLM gateway. secretsProvider may be nil if the secrets store is unavailable. tokenRefUpdater is called after login/refresh to persist the token reference into config — pass nil to skip config persistence (useful in tests). interactive controls whether a genuine cache miss may launch the browser OIDC flow. thv llm token passes true so a prior "thv llm setup --lazy" signs the user in transparently on first use; cached tokens are served without a browser prompt either way. When skipBrowser is true, an interactive login prints the authorization URL instead of opening a browser (headless/SSH/CI use); it has no effect unless interactive is also true.

type ToolConfig 

type ToolConfig struct {
	// Tool is the canonical tool identifier (e.g. "claude-code", "cursor").
	Tool string `yaml:"tool" json:"tool"`
	// Mode is the authentication mode: one of the llmgateway.Mode* values
	// ("direct", "proxy", "credential-helper", "codex-auth").
	Mode string `yaml:"mode" json:"mode"`
	// ConfigPath is the absolute path to the tool's config file that was patched.
	ConfigPath string `yaml:"config_path" json:"config_path"`
	// EnvFilePath is the absolute path to the .env file written during setup,
	// or empty if no .env file was managed for this tool.
	EnvFilePath string `yaml:"env_file_path,omitempty" json:"env_file_path,omitempty"`
}

ToolConfig records a tool that setup has configured, so teardown knows exactly what to reverse.

Source Files

View all Source files

Directories

Path Synopsis
Package proxy implements the LLM gateway localhost reverse proxy.
 Package proxy implements the LLM gateway localhost reverse proxy.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.