llamaclient

Documentation

Overview

Package llamaclient calls a llama.cpp server to generate grammar-constrained output. It uses /v1/chat/completions (so the Gemma chat template is applied by the server via --jinja) plus a raw "grammar" field — which avoids the json_schema crash path (#22396) while still constraining structure.

Index

• type AltToken
• type Client
  • func New(base, path, model string, timeout time.Duration) *Client
  • func (c *Client) Generate(ctx context.Context, model, system, user, grammar string, maxTokens int, ...) (GenResult, error)
  • func (c *Client) GenerateVision(ctx context.Context, model, system, user string, imageDataURIs []string, ...) (GenResult, error)
  • func (c *Client) GenerateVisionInterleaved(ctx context.Context, model, system string, frameLabels, imageDataURIs []string, ...) (GenResult, error)
  • func (c *Client) Health(ctx context.Context) error
• type GenResult
• type TokenLogprob

Types

type AltToken

type AltToken struct {
	Token   string
	Logprob float64
}

AltToken is one candidate at a generated position (raw, pre-grammar-mask).

type Client

type Client struct {
	// contains filtered or unexported fields
}

func New

func New(base, path, model string, timeout time.Duration) *Client

New builds a client. path is the generation route (default /v1/chat/completions); model is the llama-swap alias ("" = dedicated server).

func (*Client) Generate

func (c *Client) Generate(ctx context.Context, model, system, user, grammar string, maxTokens int, temperature float64, topLogprobs int) (GenResult, error)

Generate sends system+user as a chat request constrained by grammar (may be empty) and returns the content plus telemetry. model overrides the client's default (empty = use the default); this is how the family cascade routes to different tiers (e2b / e4b / 26b-a4b) per call. When topLogprobs > 0 the server returns per-token raw (pre-grammar-mask) logprobs in GenResult.Logprobs — used by the confidence gate to detect a genuinely uncertain decision.

func (*Client) GenerateVision

func (c *Client) GenerateVision(ctx context.Context, model, system, user string, imageDataURIs []string, grammar string, maxTokens int, temperature float64, topLogprobs int) (GenResult, error)

GenerateVision sends a multimodal chat request: the user message carries the prompt text plus one image_url part per data URI in imageDataURIs (each a full data:image/...;base64,... URI). model overrides the client default ("" = use it); grammar may be empty. It shares decodeGenResult with Generate, so telemetry/logprob handling is identical. CachePrompt is forced OFF on the vision path (llama.cpp #17200: consecutive-image KV reuse can corrupt output).

func (*Client) GenerateVisionInterleaved

func (c *Client) GenerateVisionInterleaved(ctx context.Context, model, system string, frameLabels, imageDataURIs []string, trailingUser, grammar string, maxTokens int, temperature float64, topLogprobs int) (GenResult, error)

GenerateVisionInterleaved sends a multimodal chat request whose user content INTERLEAVES a text label before each image (frameLabels[i] then image[i]), then appends trailingUser as the final text part. This matches Qwen3-VL's trained "<timestamp> frame" interleaved format (its MRoPE uses the timestamp tokens for temporal localization). frameLabels and imageDataURIs pair by index; a missing/empty label is skipped for that image. Shares decodeGenResult; like GenerateVision, CachePrompt is forced OFF (llama.cpp #17200).

func (*Client) Health

func (c *Client) Health(ctx context.Context) error

Health reports whether the server answers /health with 200.

type GenResult

type GenResult struct {
	Content   string
	TokensIn  int
	TokensOut int
	TokPerSec float64
	Truncated bool           // hit max_tokens before finishing (finish_reason == "length")
	Logprobs  []TokenLogprob // per-output-token, only when top_logprobs was requested
}

GenResult holds the model output and per-call telemetry.

type TokenLogprob

type TokenLogprob struct {
	Token string
	Top   []AltToken
}

TokenLogprob is the chosen token at one output position plus the top alternatives. NOTE: with a grammar active, these are the model's RAW distribution (pre-mask) — grammar-illegal tokens can appear, and the chosen token's logprob may be low if the grammar forced a non-preferred spelling. Confidence metrics must aggregate by legal class, not trust raw logprobs.