veloxquant

README

VeloxQuant Go

Memory intelligence and optimization for local AI, in Go.

VeloxQuant Go is not a wrapper around MLX. It's a Go-native toolkit for building local AI infrastructure: hardware detection, model and KV-cache memory estimation, VeloxQuant compression recommendations, and a client for talking to a local VeloxQuant runtime — all without needing to understand MLX or manually calculate memory requirements.

Go Application
      │
      ▼
VeloxQuant Go SDK
      │
      ├── Hardware Intelligence
      ├── Memory Estimation
      ├── KV Cache Optimization
      ├── AutoPilot
      └── Runtime Client
               │
               ▼
      VeloxQuant Runtime / MLX
               │
               ▼
        Apple Silicon

Part of the VeloxQuant ecosystem: VeloxQuant-MLX (Python optimization engine), VeloxQuant Studio (macOS app), VeloxQuant VS Code, and the VeloxQuant npm SDK.

Installation

go get github.com/rajveer43/veloxquant-go

Quick Start

package main

import (
	"context"
	"fmt"

	veloxquant "github.com/rajveer43/veloxquant-go"
)

func main() {
	client, err := veloxquant.NewClient()
	if err != nil {
		panic(err)
	}

	response, err := client.Chat(context.Background(), veloxquant.ChatRequest{
		Model: "mlx-community/Qwen3-8B-4bit",
		Messages: []veloxquant.Message{
			{Role: "user", Content: "Hello!"},
		},
	})
	if err != nil {
		panic(err)
	}

	fmt.Println(response.Text)
}

Streaming

stream, err := client.ChatStream(ctx, veloxquant.ChatRequest{
	Model: "mlx-community/Qwen3-8B-4bit",
	Messages: []veloxquant.Message{
		{Role: "user", Content: "Write a Go HTTP server."},
	},
})
if err != nil {
	panic(err)
}
defer stream.Close()

for stream.Next() {
	fmt.Print(stream.Chunk().Text)
}
if err := stream.Err(); err != nil {
	panic(err)
}

Memory Estimation

Estimate model and KV-cache memory before you load anything:

estimate, err := client.Memory.Estimate(ctx, veloxquant.MemoryRequest{
	Model: veloxquant.ModelArchitecture{
		NumLayers:      36,
		NumKVHeads:     8,
		HeadDim:        128,
		HiddenSize:     4096,
		ParameterCount: 8_000_000_000,
	},
	ContextLength: 32768,
	Precision:     veloxquant.Int4,
})

fmt.Println(veloxquant.FormatBytes(estimate.TotalMemoryBytes))
fmt.Println(veloxquant.FormatBytes(estimate.OptimizedTotalBytes))
fmt.Printf("%.1f%% saved\n", estimate.SavedPercent)

KV-cache memory is computed as:

KV Cache Memory = Layers × Tokens × KV Heads × Head Dimension × 2 × Bytes Per Element

Supported precisions: FP16, FP8, Int8, Int4.

Optimization Profiles

rec, err := client.Optimize.Recommend(ctx, veloxquant.OptimizationRequest{
	Model:         "Qwen3-8B",
	Architecture:  arch,
	ContextLength: 32768,
})

fmt.Println(rec.Profile)             // speed | balanced | memory | maximum-context
fmt.Println(rec.CompressionBits)     // e.g. 4
fmt.Println(rec.Reason)

Local Model Cache

client.Models exposes three operations against the local Hugging Face model cache ($HF_HOME/hub, or ~/.cache/huggingface/hub by default) — Local (list), Pull (download), and Delete (remove). They have different dependency requirements:

localModels, err := client.Models.Local(ctx) // dependency-free: pure filesystem scan

result, err := client.Models.Pull(ctx, "", "mlx-community/Qwen3-8B-4bit") // shells out to Python
freed, err := client.Models.Delete(ctx, "", "mlx-community/Qwen3-8B-4bit") // shells out to Python

Local (backed by models.ScanLocal) walks the cache directory directly with os.ReadDir/filepath.WalkDir — no Python dependency at all.

Pull and Delete (backed by models.Pull/models.Delete) instead shell out to a Python interpreter with huggingface_hub importable, running a short snippet that calls snapshot_download() / the scan_cache_dir().delete_revisions() eviction API. This is a deliberate, asymmetric design, not an oversight: downloading requires resolving a model id to its file manifest and content-addressing new blobs against the existing cache, and deleting requires safely removing only the blobs a revision uniquely owns without corrupting a different cached model's shared blobs (the cache's on-disk layout is content-addressed via symlinks). Reimplementing that logic natively in Go would mean chasing a cache format Go doesn't own; shelling out to huggingface_hub's own battle-tested implementation is the same choice the TS SDK makes. Pass an interpreter path as Pull/Delete's second argument, or "" to use the default resolution ($VELOXQUANT_PYTHON, then python3). A model id is always passed as its own subprocess argument, never interpolated into the Python source, so it can't be used to inject shell or Python syntax.

errors.Is(err, veloxquant.ErrHuggingFaceHubUnavailable) distinguishes "no working Python/huggingface_hub" from other pull/delete failures.

AutoPilot

AutoPilot inspects your hardware, picks a compatible model, chooses a safe context length and compression strategy, and returns a ready-to-use session:

session, err := client.AutoPilot(ctx, veloxquant.AutoPilotConfig{
	Task:  "coding",
	Model: "auto",
})
if err != nil {
	panic(err)
}

plan := session.Plan() // fully transparent decision trail
fmt.Println(plan.SelectedModel, plan.ContextLength, plan.Profile)

response, err := session.Chat(ctx, "Build a REST API in Go")

System Detection

info, err := client.System.Info(ctx)

fmt.Println(info.Platform, info.Architecture)
fmt.Println(info.AppleSilicon)
fmt.Println(veloxquant.FormatBytes(info.TotalMemory))
fmt.Println(info.RecommendedProfile)

Apple Silicon-specific detection degrades gracefully on Linux and Windows — AppleSilicon is simply false, and the SDK never panics on unsupported platforms.

Structured Output

ChatRequest.ResponseFormat mirrors OpenAI's response_format, built with JSONMode() or JSONSchema(name, schema, strict):

schema := map[string]any{
	"type":       "object",
	"properties": map[string]any{"name": map[string]any{"type": "string"}},
	"required":   []string{"name"},
}

response, err := client.Chat(ctx, veloxquant.ChatRequest{
	Model:          "mlx-community/Qwen3-8B-4bit",
	Messages:       messages,
	ResponseFormat: veloxquant.JSONSchema("person", schema, true),
})

The field is always sent on the wire, but the VeloxQuant runtime serves completions via mlx_lm.server, which does not read or enforce response_format as of this writing — it won't constrain decoding to match your schema. Prompt the model for the shape explicitly and validate its output, as shown in examples/structured. Setting ResponseFormat is still worthwhile: it's forward-compatible with runtime versions or OpenAI-compatible backends that do enforce it.

Multi-turn Conversations

Client.NewConversation returns a Conversation that accumulates message history automatically, so you don't have to rebuild []Message on every turn:

conv := client.NewConversation("mlx-community/Qwen3-8B-4bit", "You are a helpful assistant.")

resp, err := conv.Send(ctx, "What's a KV cache?")
// ...
resp, err = conv.Send(ctx, "How does that relate to memory usage?")
// conv.History() now holds all four messages (system, user, assistant, user, assistant)

If a turn fails, the conversation's history is left exactly as it was before that call, so a retried Send starts from the same state. Use conv.SendStream(ctx, prompt) for a streaming reply; history is updated once the stream finishes without error. An AutoPilot Session also exposes session.Conversation(system).

Embeddings

response, err := client.Embed(ctx, veloxquant.EmbedRequest{
	Model: "mlx-community/all-MiniLM-L6-v2-4bit",
	Input: []string{"first string", "second string"},
})
// response.Data[i].Vector holds the embedding for Input[i]

Input accepts either a single string or a []string to embed a batch in one call, mirroring OpenAI's /v1/embeddings request shape. See examples/embeddings.

Agent: Tool-Calling Loop

The agent package implements a single-turn tool-calling loop over a *veloxquant.Client: send a prompt, execute any tools the model calls, feed the results back, and repeat until the model stops calling tools or a maximum number of round trips is used up. It reuses the OpenAI tools/tool_calls wire shape end to end, since the underlying mlx_lm server (wrapped by vq serve) parses tool calls natively against this exact shape.

import "github.com/rajveer43/veloxquant-go/agent"

a := agent.New(client, "mlx-community/Qwen3-4B-4bit")

a.RegisterTool(myWeatherTool) // implements agent.Tool

result, err := a.Run(ctx, "What's the weather in Tokyo?", agent.RunOptions{})
fmt.Println(result.Text)
fmt.Println(result.Steps) // every tool call executed, in order

Tool is a plain interface (Name() string, Description() string, Parameters() any, Execute(ctx, args json.RawMessage) (any, error)) so any type can implement it — including tools sourced from an MCP server via UseMcpServer (see the separate mcp module below). RunOptions.MaxSteps defaults to 8 when left unset; exceeding it returns an error wrapping agent.ErrAgentMaxStepsExceeded. A malformed tool-call-arguments payload, an unknown tool name, or a tool's Execute returning an error do not abort the run — each is fed back to the model as a structured {"error": "..."} tool result, and the loop continues. agent has no third-party dependency and lives in the root module. See examples/agent.

MCP Tool Sources

The mcp/ directory is a separate Go module (github.com/rajveer43/veloxquant-go/mcp) that lets an agent.Agent pull tools from a Model Context Protocol server, backed by the official github.com/modelcontextprotocol/go-sdk. It's a separate module for the same reason langchain/ is: so the MCP SDK is not a dependency of the core SDK, or even of the dependency-free agent package, unless you opt in:

go get github.com/rajveer43/veloxquant-go/mcp

import (
	"github.com/rajveer43/veloxquant-go/agent"
	vqmcp "github.com/rajveer43/veloxquant-go/mcp"
)

source, err := vqmcp.Connect(ctx, vqmcp.ServerConfig{
	Name:      "my-server",
	Transport: vqmcp.TransportStdio,
	Command:   "my-mcp-server",
})
if err != nil {
	panic(err)
}

a := agent.New(client, "mlx-community/Qwen3-4B-4bit")
if err := a.UseMcpServer(ctx, source); err != nil {
	panic(err)
}

API-shape divergence from the TS SDK: TS's Agent.useMcpServer(config) connects to the MCP server itself, via a dynamic import('./mcp.js') so that importing agent.ts doesn't force every caller to depend on the MCP SDK. Go has no equivalent runtime-lazy import, so agent.Agent.UseMcpServer instead takes an already-constructed mcp.ToolSource (built via vqmcp.Connect, from the separate module above) rather than a config struct the agent package would need to know how to connect itself. This is the Go-idiomatic way to preserve the same dependency-isolation property TS's dynamic import achieves — a deliberate difference in API shape, not an incomplete port.

unwrapMcpToolResult's content-handling rules match mcp.ts exactly: structuredContent is preferred when present; a single text content block is tried as JSON, falling back to the raw string; any other content type (image/audio/resource/resource_link) returns an explicit, actionable error rather than silently dropping it. A tool-name collision between an MCP server's tools and an already-registered tool closes the newly-opened MCP connection before UseMcpServer returns its error.

LangChain Go Adapter

The langchain/ directory is a separate Go module (github.com/rajveer43/veloxquant-go/langchain) implementing langchaingo's llms.Model interface, so a local VeloxQuant runtime can be used as the model backend in a langchaingo chain. It's a separate module so that langchaingo is not a dependency of the core SDK unless you opt in:

go get github.com/rajveer43/veloxquant-go/langchain

import (
	veloxquant "github.com/rajveer43/veloxquant-go"
	vqlangchain "github.com/rajveer43/veloxquant-go/langchain"
	"github.com/tmc/langchaingo/llms"
)

client, _ := veloxquant.NewClient(veloxquant.WithAutoDetect())
model := vqlangchain.New(client, "mlx-community/Qwen3-8B-4bit")

completion, err := llms.GenerateFromSinglePrompt(ctx, model, "Explain KV cache in simple terms.")

See examples/langchain. The adapter's chat API is text-only: message parts other than text (images, tool/function calls) are rejected with an error, since the VeloxQuant runtime's chat completion endpoint doesn't support them.

Monitoring

mon := client.Monitor(veloxquant.WithMonitorInterval(2 * time.Second))
mon.Start(ctx)

mon.Subscribe(func(m monitor.Metrics) {
	fmt.Println(veloxquant.FormatBytes(m.MemoryUsedBytes))
	fmt.Printf("%.1f tok/s\n", m.TokensPerSecond)
})

The Monitor samples system memory on WithMonitorInterval's schedule (5s by default). Between samples, every Chat/ChatStream call made through the same Client also pushes a live update carrying that request's TokensPerSecond and TimeToFirstToken, so subscribers see inference performance as it happens rather than waiting for the next tick.

CLI

go install github.com/rajveer43/veloxquant-go/cmd/vq@latest

vq doctor              # check system readiness
vq analyze Qwen3-8B    # memory breakdown for a model
vq models --local      # list downloaded models and disk usage
vq models pull <id>    # download a model's weights into the local cache
vq models delete <id>  # remove a model's weights from the local cache
vq recommend           # recommended models + profile for this hardware
vq benchmark Qwen3-8B  # tokens/sec, TTFT, resident memory, default vs. optimized
vq serve               # connect to a local VeloxQuant runtime
vq serve --model mlx-community/Qwen3-8B-4bit   # launch a runtime for this model

vq models pull/vq models delete shell out to a Python interpreter with huggingface_hub importable (--python overrides the interpreter used, default $VELOXQUANT_PYTHON, then python3) — unlike vq models --local, which is a dependency-free filesystem scan. See Local Model Cache below.

vq serve --model launches the veloxquant CLI (from the VeloxQuant-MLX Python package) as a subprocess, waits for it to report readiness, and prints its URL. Press Ctrl+C to stop it. Optional flags: --method (KV-cache compression method), --host, --port.

Benchmark

veloxquant.Benchmark(ctx, client, input) is a library function measuring tokens/sec, time-to-first-token, and measured resident memory (RSS) for a model on this machine, comparing the default (unoptimized) serve method against an optimize()-picked (or explicitly named) compression method:

result, err := veloxquant.Benchmark(ctx, client, veloxquant.BenchmarkInput{
	Model:           "mlx-community/Qwen3-8B-4bit",
	OptimizedMethod: "kivi", // empty lets the runtime pick automatically
})
fmt.Println(result.ToMarkdown())

It launches two full runtime processes sequentially — the default method, then the optimized one — each fully stopped before the next starts, so resource contention between them never skews either measurement. DefaultMethodResidentBytes/OptimizedResidentBytes are *uint64 (nil when unmeasurable, e.g. the process already exited or ps failed), sampled via ps -o rss= -p <pid> right after each model finishes loading — this is real, measured memory, not the accounting-only byte counts Memory.Estimate reports, and compression is not guaranteed to reduce it: it can measure smaller in accounting terms while resident memory stays flat or even increases. When ToMarkdown() detects exactly that (optimized RSS measured higher than default), it says so explicitly with an "accounting-only" caveat rather than silently reporting the delta.

vq benchmark <model> [--method NAME] [--max-tokens N] is a thin CLI wrapper around this function, printing its ToMarkdown() output. This is a CLI-output-shape change from earlier versions, which printed a single-shot wall-clock timing with no TTFT/RSS/comparison — the new output is strictly more informative but not byte-identical to the old format.

Requires real Apple Silicon hardware and a downloaded model; not unit-testable in CI (see benchmark_manual_test.go, build-tagged manual).

Architecture

veloxquant-go/
├── client.go, config.go, types.go, errors.go, autopilot.go,
│   conversation.go                                            Top-level API
├── system/       Hardware & platform detection (build-tagged per OS)
├── memory/       Model + KV-cache memory estimation
├── optimize/     Optimization profile recommendations
├── models/       Curated model registry + task-based recommendations
├── runtime/      HTTP client for the local VeloxQuant runtime
├── openai/       OpenAI-compatible chat completions, streaming, embeddings
├── monitor/      Thread-safe memory/inference metrics monitoring
├── agent/        Tool-calling loop over a Client (no third-party dependency)
├── mcp/          MCP tool sources for agent.Agent (separate Go module)
├── langchain/    langchaingo llms.Model adapter (separate Go module)
├── cmd/vq/       CLI
└── examples/     Runnable examples

Every subsystem is exposed as an interface (system.Detector, memory.Estimator, optimize.Optimizer, models.Registry) so it can be mocked in tests without touching real hardware or a live runtime.

Examples

See examples/ for runnable programs: chat, streaming, autopilot, server, structured, conversation, embeddings, agent, mcp, and langchain.

Testing

go test ./...
go test -race ./...
go vet ./...

Package Stats

Go modules have no central download counter (unlike npm/PyPI), so adoption is tracked with the closest available public signals:

• Imported by on pkg.go.dev — count of public modules that import this package.
• Clone/view traffic — go get and git clone both register as repo clones. GitHub exposes 14 days of this under Insights → Traffic (maintainer access required). A scheduled workflow snapshots these counts weekly into traffic-history.json (committed on first run) so history survives past GitHub's 14-day retention window.

To refresh the history immediately: gh workflow run traffic.yml.

Releasing

Releases are cut from tags:

1. Update CHANGELOG.md, moving the relevant [Unreleased] entries under a new ## [x.y.z] - YYYY-MM-DD heading.
2. Commit, then tag: git tag vX.Y.Z && git push origin vX.Y.Z.
3. The release workflow runs CI against the tag and publishes a GitHub release with auto-generated notes.

pkg.go.dev picks up new tags automatically via the Go module proxy — no separate publish step is needed there.

License

MIT

Documentation

Overview

Package veloxquant is the VeloxQuant Go SDK: memory intelligence and optimization for local AI on Apple Silicon and beyond. It hides MLX and Python runtime implementation details behind an idiomatic Go API for hardware detection, memory/KV-cache estimation, optimization profile selection, and communication with a local VeloxQuant runtime.

Index

• Constants
• Variables
• func FormatBytes(b uint64) string
• type AutoPilotConfig
• type AutoPilotPlan
• type BenchmarkInput
• type BenchmarkResult
  • func Benchmark(ctx context.Context, client *Client, input BenchmarkInput) (BenchmarkResult, error)
  • func (r BenchmarkResult) ToMarkdown() string
• type ChatChunk
• type ChatRequest
• type ChatResponse
• type ChatStream
  • func (s *ChatStream) Chunk() ChatChunk
  • func (s *ChatStream) Close() error
  • func (s *ChatStream) Err() error
  • func (s *ChatStream) Metrics() InferenceMetrics
  • func (s *ChatStream) Next() bool
• type Client
  • func NewClient(opts ...Option) (*Client, error)
  • func (c *Client) AutoPilot(ctx context.Context, cfg AutoPilotConfig) (*Session, error)
  • func (c *Client) Chat(ctx context.Context, req ChatRequest) (ChatResponse, error)
  • func (c *Client) ChatStream(ctx context.Context, req ChatRequest) (*ChatStream, error)
  • func (c *Client) Embed(ctx context.Context, req EmbedRequest) (EmbedResponse, error)
  • func (c *Client) Monitor(opts ...MonitorOption) *monitor.Monitor
  • func (c *Client) NewConversation(model string, system string) *Conversation
• type Conversation
  • func (conv *Conversation) History() []Message
  • func (conv *Conversation) Send(ctx context.Context, prompt string) (ChatResponse, error)
  • func (conv *Conversation) SendStream(ctx context.Context, prompt string) (*ConversationStream, error)
• type ConversationStream
  • func (cs *ConversationStream) Chunk() ChatChunk
  • func (cs *ConversationStream) Close() error
  • func (cs *ConversationStream) Err() error
  • func (cs *ConversationStream) Next() bool
• type DeleteResult
• type EmbedRequest
• type EmbedResponse
• type Embedding
• type InferenceMetrics
• type JSONSchemaFormat
• type LocalModelInfo
• type MemoryEstimate
• type MemoryRequest
• type MemoryService
  • func (m *MemoryService) Estimate(ctx context.Context, req MemoryRequest) (MemoryEstimate, error)
• type Message
• type ModelArchitecture
• type ModelRecommendationRequest
• type ModelsService
  • func (m *ModelsService) Delete(ctx context.Context, python string, modelID string) (DeleteResult, error)
  • func (m *ModelsService) List() []models.Info
  • func (m *ModelsService) Local(ctx context.Context) ([]LocalModelInfo, error)
  • func (m *ModelsService) Pull(ctx context.Context, python string, modelID string) (PullResult, error)
  • func (m *ModelsService) Recommend(ctx context.Context, req ModelRecommendationRequest) ([]models.Info, error)
  • func (m *ModelsService) RecommendScored(ctx context.Context, req ModelRecommendationRequest) ([]models.Scored, error)
• type MonitorConfig
• type MonitorOption
  • func WithMonitorInterval(interval time.Duration) MonitorOption
• type OptimizationRecommendation
• type OptimizationRequest
• type OptimizeService
  • func (o *OptimizeService) Recommend(ctx context.Context, req OptimizationRequest) (OptimizationRecommendation, error)
• type Option
  • func WithAutoDetect() Option
  • func WithHTTPTimeout(timeout time.Duration) Option
  • func WithOpenAICompatibleRuntime(baseURL string) Option
  • func WithProfile(profile string) Option
  • func WithRuntimeURL(url string) Option
• type Precision
• type PullResult
• type ResponseFormat
  • func JSONMode() *ResponseFormat
  • func JSONSchema(name string, schema any, strict bool) *ResponseFormat
• type RuntimeService
  • func (r *RuntimeService) Health(ctx context.Context) (RuntimeStatus, error)
• type RuntimeStatus
• type Session
  • func (s *Session) Chat(ctx context.Context, prompt string) (ChatResponse, error)
  • func (s *Session) Conversation(system string) *Conversation
  • func (s *Session) Plan() AutoPilotPlan
• type SystemInfo
• type SystemService
  • func (s *SystemService) Info(ctx context.Context) (SystemInfo, error)
• type ToolCall
• type ToolCallFunction
• type ToolDefinition
• type ToolFunction
• type Usage

Constants

const (
	FP16 = memory.FP16
	FP8  = memory.FP8
	Int8 = memory.Int8
	Int4 = memory.Int4
)

Variables

var (
	ErrRuntimeUnavailable  = errors.New("veloxquant runtime unavailable")
	ErrUnsupportedPlatform = errors.New("unsupported platform")
	ErrInsufficientMemory  = errors.New("insufficient memory")
	ErrModelNotFound       = errors.New("model not found")
	ErrInvalidConfig       = errors.New("invalid configuration")

	// ErrHuggingFaceHubUnavailable re-exports models.ErrHuggingFaceHubUnavailable
	// at the top level: it's the error models.Pull/models.Delete report when
	// the configured Python interpreter can't import huggingface_hub. Defined
	// in the models package (which is what actually detects it) and aliased
	// here, alongside this SDK's other sentinel errors, so callers can
	// errors.Is against either veloxquant.ErrHuggingFaceHubUnavailable or
	// models.ErrHuggingFaceHubUnavailable interchangeably.
	ErrHuggingFaceHubUnavailable = models.ErrHuggingFaceHubUnavailable
)

Sentinel errors returned by the SDK. Use errors.Is to check for these after wrapping with fmt.Errorf("...: %w", err).

Functions

func FormatBytes

func FormatBytes(b uint64) string

FormatBytes renders a byte count as a human-readable string, e.g. "24.0 GB".

Types

type AutoPilotConfig

type AutoPilotConfig struct {
	// Task is used to select a suitable model, e.g. "coding", "chat",
	// "reasoning", "vision", "agent", "translation".
	Task string

	// Model may be a specific model name, or "auto" (or empty) to let
	// AutoPilot choose one based on Task and available hardware.
	Model string

	// ContextLength, if set, overrides AutoPilot's automatic context
	// length selection.
	ContextLength int
}

AutoPilotConfig describes the intent behind an AutoPilot session: what task the caller wants to accomplish, and optionally which model to use.

type AutoPilotPlan

type AutoPilotPlan struct {
	Hardware SystemInfo

	SelectedModel string

	ContextLength int

	CompressionBits int

	EstimatedMemoryBytes uint64
	SafetyMarginBytes    uint64

	Profile optimize.Profile

	Reason string
}

AutoPilotPlan documents every decision AutoPilot made when constructing a Session, so the process is transparent and debuggable.

type BenchmarkInput

type BenchmarkInput struct {
	// Model is the Hugging Face model id (or local path) to benchmark.
	Model string
	// OptimizedMethod names the KV-cache compression method to compare
	// against the default (unoptimized) method, e.g. "kivi". Empty lets
	// the runtime pick automatically (optimize: "auto").
	OptimizedMethod string
	// MaxTokens caps each benchmarked generation. Defaults to 128.
	MaxTokens int
}

BenchmarkInput configures a Benchmark run.

type BenchmarkResult

type BenchmarkResult struct {
	Model              string
	Chip               string
	UnifiedMemoryBytes uint64

	TokensPerSecond    float64
	TimeToFirstTokenMs float64

	// DefaultMethodResidentBytes and OptimizedResidentBytes are measured
	// resident memory (RSS) of the runtime subprocess for each method,
	// sampled once right after each model finishes loading. nil means the
	// measurement couldn't be taken (e.g. the process had already exited,
	// or `ps` failed) — matching TS's `| null` for the same case, not a
	// zero value that could be mistaken for "measured zero bytes".
	//
	// This is real, measured memory, but it reflects idle model-load RSS,
	// not KV-cache growth under load — and compression is not guaranteed
	// to lower it: it can measure smaller in accounting terms (see
	// MemoryEstimate) while resident memory stays flat or even increases.
	DefaultMethodResidentBytes *uint64
	OptimizedResidentBytes     *uint64

	Method              string
	OptimizedMethodUsed string
}

BenchmarkResult reports the outcome of a Benchmark run. Call ToMarkdown to render it as a human-readable report.

func Benchmark

func Benchmark(ctx context.Context, client *Client, input BenchmarkInput) (BenchmarkResult, error)

Benchmark measures tokens/sec, time-to-first-token, and resident memory for input.Model on this machine, comparing the default (unoptimized) serve method against an optimized one (input.OptimizedMethod, or the runtime's own automatic choice when empty). It launches two separate runtime processes sequentially — never concurrently, so resource contention doesn't skew either measurement — and fully stops each one before starting the next.

Requires real Apple Silicon hardware and a downloaded model; this is not unit-testable in CI. See runtime/benchmark_manual_test.go (build tag "manual") for a by-hand verification harness.

func (BenchmarkResult) ToMarkdown

func (r BenchmarkResult) ToMarkdown() string

ToMarkdown renders r as a human-readable Markdown report, matching the TS SDK's benchmark().toMarkdown() format. When both resident-memory measurements are available and the optimized method measured *higher* resident memory than the default, the report includes an explicit accounting-only caveat rather than silently reporting an increase as if it were a simple regression — compression byte counts (see MemoryEstimate) are not the same thing as measured RSS, and the two can diverge in either direction.

type ChatChunk

type ChatChunk struct {
	Text string
	Done bool
}

ChatChunk is a single incremental piece of a streamed chat response.

type ChatRequest

type ChatRequest struct {
	Model    string    `json:"model"`
	Messages []Message `json:"messages"`

	Temperature float64 `json:"temperature,omitempty"`
	MaxTokens   int     `json:"max_tokens,omitempty"`

	Stream bool `json:"stream,omitempty"`

	// ResponseFormat requests a specific output shape from the model, in
	// OpenAI's response_format wire format. Build it with JSONMode or
	// JSONSchema.
	//
	// The VeloxQuant runtime serves chat completions via mlx_lm.server,
	// which as of this writing does not read or enforce response_format:
	// it is forwarded on the wire for forward-compatibility and for
	// OpenAI-compatible backends that do honor it, but the local runtime
	// will not constrain decoding to match it. Prompt the model
	// explicitly to return the desired shape, and validate its output;
	// do not rely on ResponseFormat alone for structured extraction
	// against the VeloxQuant runtime today.
	ResponseFormat *ResponseFormat `json:"response_format,omitempty"`

	// Tools declares the functions the model may call. When the model
	// responds by calling one or more of them, ChatResponse.ToolCalls is
	// populated instead of (or alongside) Text, and FinishReason is
	// "tool_calls".
	Tools []ToolDefinition `json:"tools,omitempty"`
}

ChatRequest describes a chat completion request.

type ChatResponse

type ChatResponse struct {
	ID    string `json:"id"`
	Model string `json:"model"`
	Text  string `json:"text"`

	// FinishReason is the reason generation stopped, e.g. "stop", "length",
	// or "tool_calls" when the model called one or more tools (see
	// ToolCalls).
	FinishReason string `json:"finish_reason,omitempty"`

	// ToolCalls holds the tool calls the model requested, when
	// FinishReason is "tool_calls". Empty otherwise.
	ToolCalls []ToolCall `json:"tool_calls,omitempty"`

	Usage Usage `json:"usage"`

	Metrics InferenceMetrics `json:"metrics"`
}

ChatResponse is the result of a chat completion request.

type ChatStream

type ChatStream struct {
	// contains filtered or unexported fields
}

ChatStream is a handle to a streaming chat completion. Call Next to advance, Chunk to read the current piece of text, and Err to check for errors after iteration ends. Always call Close when done. Once the stream is finished (Next returns false with a nil Err), Metrics reports the completed request's tokens/sec and time-to-first-token.

func (*ChatStream) Chunk

func (s *ChatStream) Chunk() ChatChunk

Chunk returns the most recently read chunk.

func (*ChatStream) Close

func (s *ChatStream) Close() error

Close releases the underlying connection.

func (*ChatStream) Err

func (s *ChatStream) Err() error

Err returns the first error encountered while streaming, if any.

func (*ChatStream) Metrics

func (s *ChatStream) Metrics() InferenceMetrics

Metrics reports performance characteristics of the stream so far. TokensPerSecond and TimeToFirstToken are approximate: they're derived from the number of non-empty content chunks and wall-clock time, since OpenAI-compatible streaming responses don't report per-chunk token counts.

func (*ChatStream) Next

func (s *ChatStream) Next() bool

Next advances the stream. It returns false when the stream ends (check Err for failures).

type Client

type Client struct {
	System   *SystemService
	Memory   *MemoryService
	Optimize *OptimizeService
	Runtime  *RuntimeService
	Models   *ModelsService
	// contains filtered or unexported fields
}

Client is the main entry point to the VeloxQuant Go SDK. Construct one with NewClient. Client is safe for concurrent use.

func NewClient

func NewClient(opts ...Option) (*Client, error)

NewClient constructs a VeloxQuant Client. By default it connects to a runtime at http://localhost:8765 with a 60s HTTP timeout; use the With* options to customize behavior.

func (*Client) AutoPilot

func (c *Client) AutoPilot(ctx context.Context, cfg AutoPilotConfig) (*Session, error)

AutoPilot inspects the host system, selects a compatible model and context length for the given task, chooses a VeloxQuant compression strategy, and returns a ready-to-use Session.

func (*Client) Chat

func (c *Client) Chat(ctx context.Context, req ChatRequest) (ChatResponse, error)

Chat sends a chat completion request to the configured runtime and returns the full response.

func (*Client) ChatStream

func (c *Client) ChatStream(ctx context.Context, req ChatRequest) (*ChatStream, error)

ChatStream starts a streaming chat completion request.

func (*Client) Embed

func (c *Client) Embed(ctx context.Context, req EmbedRequest) (EmbedResponse, error)

Embed sends an embeddings request to the configured runtime and returns the resulting vectors. req.Input may be a single string or a []string to embed a batch in one call.

func (*Client) Monitor

func (c *Client) Monitor(opts ...MonitorOption) *monitor.Monitor

Monitor returns a Monitor sampling memory from this client's system detector at a periodic interval (5s by default; override with WithMonitorInterval). Between samples, any Chat or ChatStream call made through this Client also pushes a live update carrying that request's TokensPerSecond and TimeToFirstToken, merged onto the most recent memory sample — so subscribers see inference performance as it happens rather than waiting for the next tick.

func (*Client) NewConversation

func (c *Client) NewConversation(model string, system string) *Conversation

NewConversation returns a Conversation bound to model. Use system to seed an initial system prompt, or leave it empty to start with no history.

type Conversation

type Conversation struct {
	// contains filtered or unexported fields
}

Conversation accumulates chat history across turns so callers don't have to build []Message by hand on every call. It is not safe for concurrent use by multiple goroutines.

func (*Conversation) History

func (conv *Conversation) History() []Message

History returns the accumulated messages so far, oldest first. The returned slice is a copy; mutating it does not affect the Conversation.

func (*Conversation) Send

func (conv *Conversation) Send(ctx context.Context, prompt string) (ChatResponse, error)

Send appends prompt as a user message, sends the full history so far to the model, and appends the assistant's reply to the history before returning it. If the request fails, the history is left exactly as it was before Send was called (the failed turn is not recorded), so a retried Send starts from the same state.

func (*Conversation) SendStream

func (conv *Conversation) SendStream(ctx context.Context, prompt string) (*ConversationStream, error)

SendStream appends prompt as a user message and starts a streaming reply. Call Drain (or otherwise fully consume the stream and check its Err) before the returned history reflects the assistant's reply — the assistant message is appended to history only once the stream finishes without error. If the stream errors, the history is left as it was before SendStream was called.

type ConversationStream

type ConversationStream struct {
	// contains filtered or unexported fields
}

ConversationStream is a streaming reply within a Conversation. It wraps ChatStream and, once the stream finishes successfully, commits the assistant's full reply to the owning Conversation's history.

func (*ConversationStream) Chunk

func (cs *ConversationStream) Chunk() ChatChunk

Chunk returns the most recently read chunk.

func (*ConversationStream) Close

func (cs *ConversationStream) Close() error

Close releases the underlying connection.

func (*ConversationStream) Err

func (cs *ConversationStream) Err() error

Err returns the first error encountered while streaming, if any.

func (*ConversationStream) Next

func (cs *ConversationStream) Next() bool

Next advances the stream. It returns false when the stream ends (check Err for failures).

type DeleteResult

type DeleteResult = models.DeleteResult

DeleteResult is the result of a successful ModelsService.Delete.

type EmbedRequest

type EmbedRequest struct {
	Model string `json:"model"`
	Input any    `json:"input"`
}

EmbedRequest describes an embeddings request. Input is either a single string or a []string to embed a batch in one call.

type EmbedResponse

type EmbedResponse struct {
	Model string      `json:"model"`
	Data  []Embedding `json:"data"`
	Usage Usage       `json:"usage"`
}

EmbedResponse is the result of an embeddings request.

type Embedding

type Embedding struct {
	Index  int       `json:"index"`
	Vector []float64 `json:"vector"`
}

Embedding is a single embedding vector within an EmbedResponse, at the same index as its corresponding entry in EmbedRequest.Input.

type InferenceMetrics

type InferenceMetrics struct {
	TokensPerSecond  float64       `json:"tokens_per_second"`
	TimeToFirstToken time.Duration `json:"time_to_first_token"`
	TotalDuration    time.Duration `json:"total_duration"`
}

InferenceMetrics reports performance characteristics of a completed inference request.

type JSONSchemaFormat

type JSONSchemaFormat = openai.JSONSchema

JSONSchemaFormat re-exports openai.JSONSchema at the top level.

type LocalModelInfo

type LocalModelInfo struct {
	Name         string     `json:"name"`
	Path         string     `json:"path"`
	SizeBytes    uint64     `json:"size_bytes"`
	LastModified *time.Time `json:"last_modified,omitempty"`
}

LocalModelInfo describes a model found in the local model cache, as returned by ModelsService.Local.

type MemoryEstimate

type MemoryEstimate struct {
	ModelMemoryBytes     uint64
	KVCacheMemoryBytes   uint64
	RuntimeOverheadBytes uint64
	TotalMemoryBytes     uint64

	OptimizedKVBytes    uint64
	OptimizedTotalBytes uint64
	SavedBytes          uint64
	SavedPercent        float64

	RecommendedStrategy string
}

MemoryEstimate is the result of a memory estimation.

type MemoryRequest

type MemoryRequest struct {
	Model         ModelArchitecture
	ContextLength int
	Precision     Precision
}

MemoryRequest describes a memory estimation query.

type MemoryService

type MemoryService struct {
	// contains filtered or unexported fields
}

MemoryService exposes model and KV-cache memory estimation.

func (*MemoryService) Estimate

func (m *MemoryService) Estimate(ctx context.Context, req MemoryRequest) (MemoryEstimate, error)

Estimate computes memory requirements for a model at a given context length and precision.

type Message

type Message struct {
	Role       string     `json:"role"`
	Content    string     `json:"content"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
}

Message is a single chat message. ToolCalls is populated on an assistant message that called one or more tools; ToolCallID is set on a "tool" role message reporting a tool's result back to the model, identifying which call it answers. See the agent package for a tool-calling loop built on these fields.

type ModelArchitecture

type ModelArchitecture = memory.Architecture

ModelArchitecture re-exports memory.Architecture at the top level.

type ModelRecommendationRequest

type ModelRecommendationRequest struct {
	Task                 string
	AvailableMemoryBytes uint64
	ContextLength        int
}

ModelRecommendationRequest describes a model recommendation query.

type ModelsService

type ModelsService struct {
	// contains filtered or unexported fields
}

ModelsService exposes the VeloxQuant model registry.

func (*ModelsService) Delete

func (m *ModelsService) Delete(ctx context.Context, python string, modelID string) (DeleteResult, error)

Delete removes modelID's weights from the local model cache (see ModelsService.Local) via huggingface_hub's cache-eviction API. python selects the interpreter to use; pass "" to use models.ResolvePythonInterpreter's default resolution.

func (*ModelsService) List

func (m *ModelsService) List() []models.Info

List returns all known models.

func (*ModelsService) Local

func (m *ModelsService) Local(ctx context.Context) ([]LocalModelInfo, error)

Local scans the local model cache directory (e.g. the MLX/Hugging Face hub cache) and reports the models found on disk, their size, and when they were last modified. It returns an empty slice, not an error, if the cache directory doesn't exist or can't be read.

func (*ModelsService) Pull

func (m *ModelsService) Pull(ctx context.Context, python string, modelID string) (PullResult, error)

Pull downloads modelID's weights into the local model cache (see ModelsService.Local) via huggingface_hub's snapshot_download(), shelling out to a Python interpreter — see models.Pull's doc comment for why this (unlike Local/ScanLocal) can't be a dependency-free operation. python selects the interpreter to use; pass "" to use models.ResolvePythonInterpreter's default resolution (VELOXQUANT_PYTHON, then "python3").

func (*ModelsService) Recommend

func (m *ModelsService) Recommend(ctx context.Context, req ModelRecommendationRequest) ([]models.Info, error)

Recommend returns models suited to the requested task that fit within AvailableMemoryBytes, ranked best first.

func (*ModelsService) RecommendScored

func (m *ModelsService) RecommendScored(ctx context.Context, req ModelRecommendationRequest) ([]models.Scored, error)

RecommendScored behaves like Recommend but also returns the score and human-readable reasoning behind each candidate's ranking.

type MonitorConfig

type MonitorConfig struct {
	// contains filtered or unexported fields
}

MonitorConfig configures a Monitor returned by Client.Monitor.

type MonitorOption

type MonitorOption func(*MonitorConfig)

MonitorOption configures a Monitor. Use the WithMonitor* functions to build options.

func WithMonitorInterval

func WithMonitorInterval(interval time.Duration) MonitorOption

WithMonitorInterval sets how often the Monitor samples system memory. Defaults to 5 seconds.

type OptimizationRecommendation

type OptimizationRecommendation struct {
	Profile optimize.Profile

	CompressionMethod string
	CompressionBits   int

	EstimatedMemoryBefore uint64
	EstimatedMemoryAfter  uint64

	ContextLength int

	Reason string
}

OptimizationRecommendation is VeloxQuant's suggested optimization strategy for a model/context combination.

type OptimizationRequest

type OptimizationRequest struct {
	Model         string
	Architecture  ModelArchitecture
	ContextLength int
}

OptimizationRequest describes an optimization recommendation query.

type OptimizeService

type OptimizeService struct {
	// contains filtered or unexported fields
}

OptimizeService exposes VeloxQuant optimization profile recommendations.

func (*OptimizeService) Recommend

func (o *OptimizeService) Recommend(ctx context.Context, req OptimizationRequest) (OptimizationRecommendation, error)

Recommend returns VeloxQuant's recommended optimization strategy for the given model and context length.

type Option

type Option func(*config)

Option configures a Client. Use the With* functions to build options.

func WithAutoDetect

func WithAutoDetect() Option

WithAutoDetect enables automatic hardware detection and profile selection when the Client is constructed.

func WithHTTPTimeout

func WithHTTPTimeout(timeout time.Duration) Option

WithHTTPTimeout sets the timeout used for HTTP requests to the runtime.

func WithOpenAICompatibleRuntime

func WithOpenAICompatibleRuntime(baseURL string) Option

WithOpenAICompatibleRuntime configures the client to send chat requests to an OpenAI-compatible endpoint (e.g. "http://localhost:8765/v1") instead of the native VeloxQuant runtime API.

func WithProfile

func WithProfile(profile string) Option

WithProfile forces a specific VeloxQuant optimization profile rather than letting the SDK choose one automatically.

func WithRuntimeURL

func WithRuntimeURL(url string) Option

WithRuntimeURL sets the base URL of the VeloxQuant runtime. Defaults to http://localhost:8765.

type Precision

type Precision = memory.Precision

Precision re-exports memory.Precision at the top level so callers don't need to import the memory subpackage for common usage.

type PullResult

type PullResult = models.PullResult

PullResult is the result of a successful ModelsService.Pull.

type ResponseFormat

type ResponseFormat = openai.ResponseFormat

ResponseFormat requests a specific chat completion output format.

func JSONMode

func JSONMode() *ResponseFormat

JSONMode returns a ResponseFormat requesting a JSON object response (OpenAI's response_format: {"type": "json_object"}), without constraining it to a specific schema.

func JSONSchema

func JSONSchema(name string, schema any, strict bool) *ResponseFormat

JSONSchema returns a ResponseFormat requesting a response constrained to the given schema (OpenAI's response_format: {"type": "json_schema", ...}). name identifies the schema; schema is typically a map[string]any describing a JSON Schema object, or a value that marshals to one. Set strict to true to request exact schema adherence, for backends that support it.

type RuntimeService

type RuntimeService struct {
	// contains filtered or unexported fields
}

RuntimeService exposes communication with a local VeloxQuant runtime.

func (*RuntimeService) Health

func (r *RuntimeService) Health(ctx context.Context) (RuntimeStatus, error)

Health checks whether the VeloxQuant runtime is reachable and healthy.

type RuntimeStatus

type RuntimeStatus struct {
	Healthy bool
	Version string
	Engine  string
}

RuntimeStatus describes the health of a VeloxQuant runtime instance.

type Session

type Session struct {
	// contains filtered or unexported fields
}

Session is a ready-to-use AI session produced by AutoPilot, bound to a specific model and optimization plan.

func (*Session) Chat

func (s *Session) Chat(ctx context.Context, prompt string) (ChatResponse, error)

Chat sends a message using the model AutoPilot selected for this session.

func (*Session) Conversation

func (s *Session) Conversation(system string) *Conversation

Conversation returns a history-tracking Conversation bound to the model AutoPilot selected for this Session.

func (*Session) Plan

func (s *Session) Plan() AutoPilotPlan

Plan returns the decisions AutoPilot made to construct this Session.

type SystemInfo

type SystemInfo struct {
	Platform     string
	Architecture string
	CPUModel     string
	AppleSilicon bool

	TotalMemory     uint64
	AvailableMemory uint64

	RecommendedProfile string
}

SystemInfo describes the host system relevant to running local LLM inference.

type SystemService

type SystemService struct {
	// contains filtered or unexported fields
}

SystemService exposes hardware and platform detection.

func (*SystemService) Info

func (s *SystemService) Info(ctx context.Context) (SystemInfo, error)

Info returns details about the host system.

type ToolCall

type ToolCall = openai.ToolCall

ToolCall re-exports openai.ToolCall at the top level: a single tool invocation the model requested.

type ToolCallFunction

type ToolCallFunction = openai.ToolCallFunction

ToolCallFunction re-exports openai.ToolCallFunction at the top level.

type ToolDefinition

type ToolDefinition = openai.ToolDefinition

ToolDefinition re-exports openai.ToolDefinition at the top level: a callable tool description in OpenAI's function-calling wire format.

type ToolFunction

type ToolFunction = openai.ToolFunction

ToolFunction re-exports openai.ToolFunction at the top level.

type Usage

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

Usage reports token accounting for a chat completion.