veloxquant

package module
v0.3.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 3, 2026 License: MIT Imports: 12 Imported by: 0

README

VeloxQuant Go

CI Go Reference Go Report Card

Memory intelligence and optimization for local AI, in Go.

VeloxQuant Go is not a wrapper around MLX. It's a Go-native toolkit for building local AI infrastructure: hardware detection, model and KV-cache memory estimation, VeloxQuant compression recommendations, and a client for talking to a local VeloxQuant runtime — all without needing to understand MLX or manually calculate memory requirements.

Go Application
      │
      ▼
VeloxQuant Go SDK
      │
      ├── Hardware Intelligence
      ├── Memory Estimation
      ├── KV Cache Optimization
      ├── AutoPilot
      └── Runtime Client
               │
               ▼
      VeloxQuant Runtime / MLX
               │
               ▼
        Apple Silicon

Part of the VeloxQuant ecosystem: VeloxQuant-MLX (Python optimization engine), VeloxQuant Studio (macOS app), VeloxQuant VS Code, and the VeloxQuant npm SDK.

Installation

go get github.com/rajveer43/veloxquant-go

Quick Start

package main

import (
	"context"
	"fmt"

	veloxquant "github.com/rajveer43/veloxquant-go"
)

func main() {
	client, err := veloxquant.NewClient()
	if err != nil {
		panic(err)
	}

	response, err := client.Chat(context.Background(), veloxquant.ChatRequest{
		Model: "mlx-community/Qwen3-8B-4bit",
		Messages: []veloxquant.Message{
			{Role: "user", Content: "Hello!"},
		},
	})
	if err != nil {
		panic(err)
	}

	fmt.Println(response.Text)
}

Streaming

stream, err := client.ChatStream(ctx, veloxquant.ChatRequest{
	Model: "mlx-community/Qwen3-8B-4bit",
	Messages: []veloxquant.Message{
		{Role: "user", Content: "Write a Go HTTP server."},
	},
})
if err != nil {
	panic(err)
}
defer stream.Close()

for stream.Next() {
	fmt.Print(stream.Chunk().Text)
}
if err := stream.Err(); err != nil {
	panic(err)
}

Memory Estimation

Estimate model and KV-cache memory before you load anything:

estimate, err := client.Memory.Estimate(ctx, veloxquant.MemoryRequest{
	Model: veloxquant.ModelArchitecture{
		NumLayers:      36,
		NumKVHeads:     8,
		HeadDim:        128,
		HiddenSize:     4096,
		ParameterCount: 8_000_000_000,
	},
	ContextLength: 32768,
	Precision:     veloxquant.Int4,
})

fmt.Println(veloxquant.FormatBytes(estimate.TotalMemoryBytes))
fmt.Println(veloxquant.FormatBytes(estimate.OptimizedTotalBytes))
fmt.Printf("%.1f%% saved\n", estimate.SavedPercent)

KV-cache memory is computed as:

KV Cache Memory = Layers × Tokens × KV Heads × Head Dimension × 2 × Bytes Per Element

Supported precisions: FP16, FP8, Int8, Int4.

Optimization Profiles

rec, err := client.Optimize.Recommend(ctx, veloxquant.OptimizationRequest{
	Model:         "Qwen3-8B",
	Architecture:  arch,
	ContextLength: 32768,
})

fmt.Println(rec.Profile)             // speed | balanced | memory | maximum-context
fmt.Println(rec.CompressionBits)     // e.g. 4
fmt.Println(rec.Reason)

AutoPilot

AutoPilot inspects your hardware, picks a compatible model, chooses a safe context length and compression strategy, and returns a ready-to-use session:

session, err := client.AutoPilot(ctx, veloxquant.AutoPilotConfig{
	Task:  "coding",
	Model: "auto",
})
if err != nil {
	panic(err)
}

plan := session.Plan() // fully transparent decision trail
fmt.Println(plan.SelectedModel, plan.ContextLength, plan.Profile)

response, err := session.Chat(ctx, "Build a REST API in Go")

System Detection

info, err := client.System.Info(ctx)

fmt.Println(info.Platform, info.Architecture)
fmt.Println(info.AppleSilicon)
fmt.Println(veloxquant.FormatBytes(info.TotalMemory))
fmt.Println(info.RecommendedProfile)

Apple Silicon-specific detection degrades gracefully on Linux and Windows — AppleSilicon is simply false, and the SDK never panics on unsupported platforms.

Monitoring

mon := client.Monitor(veloxquant.WithMonitorInterval(2 * time.Second))
mon.Start(ctx)

mon.Subscribe(func(m monitor.Metrics) {
	fmt.Println(veloxquant.FormatBytes(m.MemoryUsedBytes))
	fmt.Printf("%.1f tok/s\n", m.TokensPerSecond)
})

The Monitor samples system memory on WithMonitorInterval's schedule (5s by default). Between samples, every Chat/ChatStream call made through the same Client also pushes a live update carrying that request's TokensPerSecond and TimeToFirstToken, so subscribers see inference performance as it happens rather than waiting for the next tick.

CLI

go install github.com/rajveer43/veloxquant-go/cmd/vq@latest

vq doctor              # check system readiness
vq analyze Qwen3-8B    # memory breakdown for a model
vq recommend           # recommended models + profile for this hardware
vq benchmark Qwen3-8B  # tokens/sec, TTFT, memory (requires a running runtime)
vq serve               # connect to a local VeloxQuant runtime
vq serve --model mlx-community/Qwen3-8B-4bit   # launch a runtime for this model

vq serve --model launches the veloxquant CLI (from the VeloxQuant-MLX Python package) as a subprocess, waits for it to report readiness, and prints its URL. Press Ctrl+C to stop it. Optional flags: --method (KV-cache compression method), --host, --port.

Architecture

veloxquant-go/
├── client.go, config.go, types.go, errors.go, autopilot.go   Top-level API
├── system/       Hardware & platform detection (build-tagged per OS)
├── memory/       Model + KV-cache memory estimation
├── optimize/     Optimization profile recommendations
├── models/       Curated model registry + task-based recommendations
├── runtime/      HTTP client for the local VeloxQuant runtime
├── openai/       OpenAI-compatible chat completions + streaming
├── monitor/      Thread-safe memory/inference metrics monitoring
├── cmd/vq/       CLI
└── examples/     Runnable examples

Every subsystem is exposed as an interface (system.Detector, memory.Estimator, optimize.Optimizer, models.Registry) so it can be mocked in tests without touching real hardware or a live runtime.

Examples

See examples/ for runnable programs: chat, streaming, autopilot, and server.

Testing

go test ./...
go test -race ./...
go vet ./...

Package Stats

Go modules have no central download counter (unlike npm/PyPI), so adoption is tracked with the closest available public signals:

  • Imported by on pkg.go.dev — count of public modules that import this package.
  • Clone/view traffic — go get and git clone both register as repo clones. GitHub exposes 14 days of this under Insights → Traffic (maintainer access required). A scheduled workflow snapshots these counts weekly into traffic-history.json (committed on first run) so history survives past GitHub's 14-day retention window.

To refresh the history immediately: gh workflow run traffic.yml.

Releasing

Releases are cut from tags:

  1. Update CHANGELOG.md, moving the relevant [Unreleased] entries under a new ## [x.y.z] - YYYY-MM-DD heading.
  2. Commit, then tag: git tag vX.Y.Z && git push origin vX.Y.Z.
  3. The release workflow runs CI against the tag and publishes a GitHub release with auto-generated notes.

pkg.go.dev picks up new tags automatically via the Go module proxy — no separate publish step is needed there.

License

MIT

Documentation

Overview

Package veloxquant is the VeloxQuant Go SDK: memory intelligence and optimization for local AI on Apple Silicon and beyond. It hides MLX and Python runtime implementation details behind an idiomatic Go API for hardware detection, memory/KV-cache estimation, optimization profile selection, and communication with a local VeloxQuant runtime.

Index

Constants

View Source 
const (
	FP16 = memory.FP16
	FP8  = memory.FP8
	Int8 = memory.Int8
	Int4 = memory.Int4
)

Variables

View Source 
var (
	ErrRuntimeUnavailable  = errors.New("veloxquant runtime unavailable")
	ErrUnsupportedPlatform = errors.New("unsupported platform")
	ErrInsufficientMemory  = errors.New("insufficient memory")
	ErrModelNotFound       = errors.New("model not found")
	ErrInvalidConfig       = errors.New("invalid configuration")
)

Sentinel errors returned by the SDK. Use errors.Is to check for these after wrapping with fmt.Errorf("...: %w", err).

Functions

func FormatBytes 

func FormatBytes(b uint64) string

FormatBytes renders a byte count as a human-readable string, e.g. "24.0 GB".

Types

type AutoPilotConfig 

type AutoPilotConfig struct {
	// Task is used to select a suitable model, e.g. "coding", "chat",
	// "reasoning", "vision", "agent", "translation".
	Task string

	// Model may be a specific model name, or "auto" (or empty) to let
	// AutoPilot choose one based on Task and available hardware.
	Model string

	// ContextLength, if set, overrides AutoPilot's automatic context
	// length selection.
	ContextLength int
}

AutoPilotConfig describes the intent behind an AutoPilot session: what task the caller wants to accomplish, and optionally which model to use.

type AutoPilotPlan 

type AutoPilotPlan struct {
	Hardware SystemInfo

	SelectedModel string

	ContextLength int

	CompressionBits int

	EstimatedMemoryBytes uint64
	SafetyMarginBytes    uint64

	Profile optimize.Profile

	Reason string
}

AutoPilotPlan documents every decision AutoPilot made when constructing a Session, so the process is transparent and debuggable.

type ChatChunk 

type ChatChunk struct {
	Text string
	Done bool
}

ChatChunk is a single incremental piece of a streamed chat response.

type ChatRequest 

type ChatRequest struct {
	Model    string    `json:"model"`
	Messages []Message `json:"messages"`

	Temperature float64 `json:"temperature,omitempty"`
	MaxTokens   int     `json:"max_tokens,omitempty"`

	Stream bool `json:"stream,omitempty"`
}

ChatRequest describes a chat completion request.

type ChatResponse 

type ChatResponse struct {
	ID    string `json:"id"`
	Model string `json:"model"`
	Text  string `json:"text"`

	Usage Usage `json:"usage"`

	Metrics InferenceMetrics `json:"metrics"`
}

ChatResponse is the result of a chat completion request.

type ChatStream 

type ChatStream struct {
	// contains filtered or unexported fields
}

ChatStream is a handle to a streaming chat completion. Call Next to advance, Chunk to read the current piece of text, and Err to check for errors after iteration ends. Always call Close when done. Once the stream is finished (Next returns false with a nil Err), Metrics reports the completed request's tokens/sec and time-to-first-token.

func (*ChatStream) Chunk 

func (s *ChatStream) Chunk() ChatChunk

Chunk returns the most recently read chunk.

func (*ChatStream) Close 

func (s *ChatStream) Close() error

Close releases the underlying connection.

func (*ChatStream) Err 

func (s *ChatStream) Err() error

Err returns the first error encountered while streaming, if any.

func (*ChatStream) Metrics added in v0.3.0 

func (s *ChatStream) Metrics() InferenceMetrics

Metrics reports performance characteristics of the stream so far. TokensPerSecond and TimeToFirstToken are approximate: they're derived from the number of non-empty content chunks and wall-clock time, since OpenAI-compatible streaming responses don't report per-chunk token counts.

func (*ChatStream) Next 

func (s *ChatStream) Next() bool

Next advances the stream. It returns false when the stream ends (check Err for failures).

type Client 

type Client struct {
	System   *SystemService
	Memory   *MemoryService
	Optimize *OptimizeService
	Runtime  *RuntimeService
	Models   *ModelsService
	// contains filtered or unexported fields
}

Client is the main entry point to the VeloxQuant Go SDK. Construct one with NewClient. Client is safe for concurrent use.

func NewClient 

func NewClient(opts ...Option) (*Client, error)

NewClient constructs a VeloxQuant Client. By default it connects to a runtime at http://localhost:8765 with a 60s HTTP timeout; use the With* options to customize behavior.

func (*Client) AutoPilot 

func (c *Client) AutoPilot(ctx context.Context, cfg AutoPilotConfig) (*Session, error)

AutoPilot inspects the host system, selects a compatible model and context length for the given task, chooses a VeloxQuant compression strategy, and returns a ready-to-use Session.

func (*Client) Chat 

func (c *Client) Chat(ctx context.Context, req ChatRequest) (ChatResponse, error)

Chat sends a chat completion request to the configured runtime and returns the full response.

func (*Client) ChatStream 

func (c *Client) ChatStream(ctx context.Context, req ChatRequest) (*ChatStream, error)

ChatStream starts a streaming chat completion request.

func (*Client) Monitor 

func (c *Client) Monitor(opts ...MonitorOption) *monitor.Monitor

Monitor returns a Monitor sampling memory from this client's system detector at a periodic interval (5s by default; override with WithMonitorInterval). Between samples, any Chat or ChatStream call made through this Client also pushes a live update carrying that request's TokensPerSecond and TimeToFirstToken, merged onto the most recent memory sample — so subscribers see inference performance as it happens rather than waiting for the next tick.

type InferenceMetrics 

type InferenceMetrics struct {
	TokensPerSecond  float64       `json:"tokens_per_second"`
	TimeToFirstToken time.Duration `json:"time_to_first_token"`
	TotalDuration    time.Duration `json:"total_duration"`
}

InferenceMetrics reports performance characteristics of a completed inference request.

type MemoryEstimate 

type MemoryEstimate struct {
	ModelMemoryBytes     uint64
	KVCacheMemoryBytes   uint64
	RuntimeOverheadBytes uint64
	TotalMemoryBytes     uint64

	OptimizedKVBytes    uint64
	OptimizedTotalBytes uint64
	SavedBytes          uint64
	SavedPercent        float64

	RecommendedStrategy string
}

MemoryEstimate is the result of a memory estimation.

type MemoryRequest 

type MemoryRequest struct {
	Model         ModelArchitecture
	ContextLength int
	Precision     Precision
}

MemoryRequest describes a memory estimation query.

type MemoryService 

type MemoryService struct {
	// contains filtered or unexported fields
}

MemoryService exposes model and KV-cache memory estimation.

func (*MemoryService) Estimate 

func (m *MemoryService) Estimate(ctx context.Context, req MemoryRequest) (MemoryEstimate, error)

Estimate computes memory requirements for a model at a given context length and precision.

type Message 

type Message struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

Message is a single chat message.

type ModelArchitecture 

type ModelArchitecture = memory.Architecture

ModelArchitecture re-exports memory.Architecture at the top level.

type ModelRecommendationRequest 

type ModelRecommendationRequest struct {
	Task                 string
	AvailableMemoryBytes uint64
	ContextLength        int
}

ModelRecommendationRequest describes a model recommendation query.

type ModelsService 

type ModelsService struct {
	// contains filtered or unexported fields
}

ModelsService exposes the VeloxQuant model registry.

func (*ModelsService) List 

func (m *ModelsService) List() []models.Info

List returns all known models.

func (*ModelsService) Recommend 

func (m *ModelsService) Recommend(ctx context.Context, req ModelRecommendationRequest) ([]models.Info, error)

Recommend returns models suited to the requested task that fit within AvailableMemoryBytes, ranked best first.

func (*ModelsService) RecommendScored added in v0.3.0 

func (m *ModelsService) RecommendScored(ctx context.Context, req ModelRecommendationRequest) ([]models.Scored, error)

RecommendScored behaves like Recommend but also returns the score and human-readable reasoning behind each candidate's ranking.

type MonitorConfig added in v0.3.0 

type MonitorConfig struct {
	// contains filtered or unexported fields
}

MonitorConfig configures a Monitor returned by Client.Monitor.

type MonitorOption added in v0.3.0 

type MonitorOption func(*MonitorConfig)

MonitorOption configures a Monitor. Use the WithMonitor* functions to build options.

func WithMonitorInterval added in v0.3.0 

func WithMonitorInterval(interval time.Duration) MonitorOption

WithMonitorInterval sets how often the Monitor samples system memory. Defaults to 5 seconds.

type OptimizationRecommendation 

type OptimizationRecommendation struct {
	Profile optimize.Profile

	CompressionMethod string
	CompressionBits   int

	EstimatedMemoryBefore uint64
	EstimatedMemoryAfter  uint64

	ContextLength int

	Reason string
}

OptimizationRecommendation is VeloxQuant's suggested optimization strategy for a model/context combination.

type OptimizationRequest 

type OptimizationRequest struct {
	Model         string
	Architecture  ModelArchitecture
	ContextLength int
}

OptimizationRequest describes an optimization recommendation query.

type OptimizeService 

type OptimizeService struct {
	// contains filtered or unexported fields
}

OptimizeService exposes VeloxQuant optimization profile recommendations.

func (*OptimizeService) Recommend 

func (o *OptimizeService) Recommend(ctx context.Context, req OptimizationRequest) (OptimizationRecommendation, error)

Recommend returns VeloxQuant's recommended optimization strategy for the given model and context length.

type Option 

type Option func(*config)

Option configures a Client. Use the With* functions to build options.

func WithAutoDetect 

func WithAutoDetect() Option

WithAutoDetect enables automatic hardware detection and profile selection when the Client is constructed.

func WithHTTPTimeout 

func WithHTTPTimeout(timeout time.Duration) Option

WithHTTPTimeout sets the timeout used for HTTP requests to the runtime.

func WithOpenAICompatibleRuntime 

func WithOpenAICompatibleRuntime(baseURL string) Option

WithOpenAICompatibleRuntime configures the client to send chat requests to an OpenAI-compatible endpoint (e.g. "http://localhost:8765/v1") instead of the native VeloxQuant runtime API.

func WithProfile 

func WithProfile(profile string) Option

WithProfile forces a specific VeloxQuant optimization profile rather than letting the SDK choose one automatically.

func WithRuntimeURL 

func WithRuntimeURL(url string) Option

WithRuntimeURL sets the base URL of the VeloxQuant runtime. Defaults to http://localhost:8765.

type Precision 

type Precision = memory.Precision

Precision re-exports memory.Precision at the top level so callers don't need to import the memory subpackage for common usage.

type RuntimeService 

type RuntimeService struct {
	// contains filtered or unexported fields
}

RuntimeService exposes communication with a local VeloxQuant runtime.

func (*RuntimeService) Health 

func (r *RuntimeService) Health(ctx context.Context) (RuntimeStatus, error)

Health checks whether the VeloxQuant runtime is reachable and healthy.

type RuntimeStatus 

type RuntimeStatus struct {
	Healthy bool
	Version string
	Engine  string
}

RuntimeStatus describes the health of a VeloxQuant runtime instance.

type Session 

type Session struct {
	// contains filtered or unexported fields
}

Session is a ready-to-use AI session produced by AutoPilot, bound to a specific model and optimization plan.

func (*Session) Chat 

func (s *Session) Chat(ctx context.Context, prompt string) (ChatResponse, error)

Chat sends a message using the model AutoPilot selected for this session.

func (*Session) Plan 

func (s *Session) Plan() AutoPilotPlan

Plan returns the decisions AutoPilot made to construct this Session.

type SystemInfo 

type SystemInfo struct {
	Platform     string
	Architecture string
	CPUModel     string
	AppleSilicon bool

	TotalMemory     uint64
	AvailableMemory uint64

	RecommendedProfile string
}

SystemInfo describes the host system relevant to running local LLM inference.

type SystemService 

type SystemService struct {
	// contains filtered or unexported fields
}

SystemService exposes hardware and platform detection.

func (*SystemService) Info 

func (s *SystemService) Info(ctx context.Context) (SystemInfo, error)

Info returns details about the host system.

type Usage 

type Usage struct {
	PromptTokens     int `json:"prompt_tokens"`
	CompletionTokens int `json:"completion_tokens"`
	TotalTokens      int `json:"total_tokens"`
}

Usage reports token accounting for a chat completion.

Source Files

View all Source files

Directories

Path Synopsis
cmd
vq command
Command vq is the VeloxQuant CLI: hardware diagnostics, model memory analysis, optimization recommendations, benchmarking, and a local runtime bridge.
 Command vq is the VeloxQuant CLI: hardware diagnostics, model memory analysis, optimization recommendations, benchmarking, and a local runtime bridge.
examples
autopilot command
Example: letting AutoPilot select a model, context length, and compression strategy based on detected hardware.
 Example: letting AutoPilot select a model, context length, and compression strategy based on detected hardware.
chat command
Example: a single chat completion request against a local VeloxQuant runtime.
 Example: a single chat completion request against a local VeloxQuant runtime.
server command
Example: building a small HTTP service on top of the VeloxQuant Go SDK, exposing memory estimation and chat as JSON endpoints.
 Example: building a small HTTP service on top of the VeloxQuant Go SDK, exposing memory estimation and chat as JSON endpoints.
streaming command
Example: streaming a chat completion token-by-token.
 Example: streaming a chat completion token-by-token.
internal
httpclient
Package httpclient provides a small, shared HTTP client wrapper used by the runtime and openai packages: context-aware requests, JSON encoding helpers, and typed error responses.
 Package httpclient provides a small, shared HTTP client wrapper used by the runtime and openai packages: context-aware requests, JSON encoding helpers, and typed error responses.
langchain module
mcp module
Package memory implements VeloxQuant's memory intelligence: estimating how much RAM a model and its KV cache will need, and how much VeloxQuant compression can save.
 Package memory implements VeloxQuant's memory intelligence: estimating how much RAM a model and its KV cache will need, and how much VeloxQuant compression can save.
Package models provides a curated registry of known local LLMs and task-based recommendations.
 Package models provides a curated registry of known local LLMs and task-based recommendations.
Package monitor provides thread-safe, subscribable monitoring of memory and inference metrics.
 Package monitor provides thread-safe, subscribable monitoring of memory and inference metrics.
Package openai implements a minimal client for OpenAI-compatible chat completion APIs, used to talk to the VeloxQuant runtime (or any other OpenAI-compatible local server).
 Package openai implements a minimal client for OpenAI-compatible chat completion APIs, used to talk to the VeloxQuant runtime (or any other OpenAI-compatible local server).
Package optimize provides VeloxQuant optimization profile selection and compression recommendations for a given model and context length.
 Package optimize provides VeloxQuant optimization profile selection and compression recommendations for a given model and context length.
Package runtime implements the HTTP client used to communicate with a local VeloxQuant runtime process (typically at http://localhost:8765).
 Package runtime implements the HTTP client used to communicate with a local VeloxQuant runtime process (typically at http://localhost:8765).
Package system provides hardware and platform detection for VeloxQuant, including Apple Silicon detection and system memory inspection.
 Package system provides hardware and platform detection for VeloxQuant, including Apple Silicon detection and system memory inspection.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.