offload-harness

README

local-offload

Delegate the grunt work to a free local model — keep your cloud tokens for judgment.

A local-first harness that offloads short-context, low-judgment work — summarize · classify · extract · triage (plus vision, OCR, transcription, and image/SVG generation) — to a free Gemma-family cascade served by llama.cpp. It runs as a Go CLI and as an MCP server for AI coding agents. It never calls a cloud model: when it can't do a task confidently, it returns a structured defer so your agent handles it.

---

What & why

When an agent (or you) needs to summarize a log, label a ticket, pull fields out of a document, or answer a yes/no question about some text, that work is mechanical and low-judgment — but it still costs context window and cloud tokens. local-offload runs those tasks on a free local model so the bulk, low-value tokens never enter your expensive context. A built-in ledger reports exactly how many tokens you saved.

The design rule is simple: the local model does grunt work; your agent keeps all judgment. Every task either returns a verified, schema-valid result or defers — a structured {"deferred": true, ...} that tells the caller "do this one yourself." There is no cloud fallback inside the harness, and it holds no cloud credentials.

It's for anyone running an AI coding agent or pipeline who wants to cut token spend on bulk text work while keeping data on the box.

Features

• Free & fully local — all inference runs on your GPU via llama.cpp; no API keys, no metering, nothing leaves the machine.
• Never calls cloud, always defers — low confidence returns a structured defer instead of guessing. The core offload path has no cloud credentials by design (the opt-in nim remote tool below is the one explicit, deliberate exception).
• Self-learning cascade — fast tasks enter at a small tier and escalate to a larger model only when genuinely uncertain (logprob decision margin + self-reported confidence).
• Reliable structured output — enforces a generated GBNF grammar + Go schema validation, working around the model's JSON-schema crashes.
• Single static binary — one Go executable; CLI and MCP server in the same build.
• MCP-native — exposes 15 tools over stdio for any MCP client (Claude Code and friends).
• Beyond text — local vision (VQA / OCR / image-field-extract / render-QA), speech-to-text (whisper.cpp), image/audio/video generation (SDXL · Chatterbox TTS · ACE-Step · Hunyuan via ComfyUI), and a dependency-free SVG data-viz kit.
• Optional remote escalation — an explicit, opt-in nim tool reaches any OpenAI-compatible NVIDIA NIM endpoint (NVIDIA's hosted build.nvidia.com free-model catalog, or a self-hosted NIM) for the rare task that needs a frontier model the local GPU can't run. Key from env only; never ledgered; the local cascade is untouched.
• Token ledger — append-only JSONL accounting of every offloaded call and the cloud tokens it saved.

Quickstart

# 1. Build (Go 1.26+)
go build -o local-offload .

# 2. Point at your local llama.cpp endpoint (defaults assume http://127.0.0.1:11436)
#    ./config.json is picked up automatically when run from this directory;
#    for a global install put it at ~/.local-offload/config.json instead.
cp config.example.json config.json

# 3. Run a task — input is a file path or "-" for stdin
./local-offload summarize notes.md --max-points 5 --json
./local-offload triage log.txt --question "Does this contain an error?" --json

# 4. See what you saved
./local-offload ledger

First output in under five commands. If the local model is unreachable or unsure, you'll get {"deferred": true, "reason": "..."} — that's expected, not a failure.

Installation

Build from source (recommended):

git clone https://github.com/dmmdea/offload-harness.git
cd local-offload
go build -o local-offload .       # or: go build -o local-offload.exe . on Windows

Go install:

go install github.com/dmmdea/offload-harness@latest

Requires Go 1.26+ and a running llama.cpp server (see Serving the models).

Usage (CLI)

# Text — the four core tasks
local-offload summarize <file|-> [--max-points N] [--json]
local-offload classify  <file|-> --labels bug,feature,question [--json]
local-offload extract   <file|-> --schema fields.json [--json]
local-offload triage    <file|-> --question "Is this a refund request?" [--json]

# Vision (image understanding)
local-offload vqa           image.png --question "What number is shown?" --json
local-offload ocr           scan.png --json
local-offload extract-image invoice.png --schema fields.json --json
local-offload assess-image  render.png --brief "a red sports car at sunset" --json

# Speech-to-text (audio or video)
local-offload transcribe    clip.mp4 --language es --json
local-offload transcribe    noisy.m4a --language es --hq        # high-quality model for hard audio

# Video understanding (samples frames)
local-offload video-describe clip.mp4 --question "What happens here?" --json

# Generate (free, local GPU)
local-offload generate-image "a product photo of a coffee mug on white" --negative "people, text, watermark"
local-offload generate-svg gauge '{"value":72,"max":100,"label":"Score","unit":"%"}'

# Remote escalation (explicit, opt-in — NVIDIA NIM; needs NVIDIA_API_KEY for the free hosted catalog)
local-offload nim --list-models                                            # browse available model ids
local-offload nim "Explain MoE routing in 3 bullets" --model nvidia/nemotron-3-ultra-550b-a55b --max-tokens 600
local-offload nim "Summarize this" --model meta/llama-3.3-70b-instruct --base http://127.0.0.1:8000/v1  # self-hosted NIM (keyless)

# Operate & inspect
local-offload mcp                      # run as an MCP server (stdio)
local-offload ledger [--since DAYS]    # token-savings report
local-offload doctor                   # endpoint health + config check
local-offload models                   # show configured models + serving flags
local-offload eval [--dir DIR]         # code-based quality eval (AURC, deferral-curve AUDC/QNC)
local-offload stats                    # per-task ledger telemetry

Input is a file path or - for stdin. Add --json for the full result object, --select a,b,c to keep only certain top-level fields, and --compact to minify. Configuration is read from --config <path> or $LOCAL_OFFLOAD_CONFIG.

Example: extract structured fields

fields.json is a JSON Schema with a properties map:

{ "properties": { "name": { "type": "string" }, "amount": { "type": "number" }, "date": { "type": "string" } } }

local-offload extract invoice.txt --schema fields.json --json
# -> {"name":"...","amount":1240.50,"date":"2026-01-15"}   (values grounded in the input text)

A bare {"field":"string"} map has no usable properties and is deferred.

Self-learning jobs (offline, inference-free)

These run as a nightly batch over the ledger — pure Go statistics, zero cloud tokens:

local-offload calibrate           # per-task conformal escalation thresholds  -> thresholds.json
local-offload health              # per-tier EWMA/Page-Hinkley/CUSUM + P95 timeouts -> tier_overrides.json
local-offload train-router        # logistic entry-tier router from input features -> router-weights.json
local-offload optimize            # mine verified-good calls into few-shot exemplar pools
local-offload audit-sample --hard # surface the hardest cases for human/agent review

See How the cascade learns.

Use as an MCP server

Register the binary with your MCP client. The built-in defaults already encode the full cascade, so --config is only needed for non-default endpoints or paths.

claude mcp add local-offload --scope user -- /path/to/local-offload mcp

Or add it to your client's MCP config directly:

{
  "mcpServers": {
    "local-offload": {
      "command": "/path/to/local-offload",
      "args": ["mcp"]
    }
  }
}

Transport is stdio. Every tool returns the full result JSON — and a {"deferred": true, ...} defer is a valid result, signalling the agent to do that task itself.

Exposed MCP tools

Tool	Arguments	What it does
offload_summarize	text, max_points?	Summarize text → {summary, bullets}, or defer.
offload_classify	text, labels[]	Classify into one of the labels → {label, confidence}, or defer.
offload_extract	text, schema	Extract schema-constrained fields → object, or defer. Values grounded in the input.
offload_triage	text, question	Yes/no/unsure check → {decision, reason}, or defer.
offload_vqa	image, question	Visual Q&A on a local image → {answer}, or defer.
offload_ocr	image	Transcribe all text in an image → {text}, or defer.
offload_extract_image	image, schema	OCR then extract grounded fields from the image → object, or defer.
offload_assess_image	image, brief?	QA a render against exclusions → {has_people, has_text, matches_brief, notes}.
offload_video_describe	video, question	Sample frames from a local video and answer → {answer}, or defer.
offload_transcribe	audio, language?, hq?, select?	Transcribe local audio/video → {gist, segments[], srt_path, ...}, or defer.
offload_generate_image	prompt, negative?, width?, height?, steps?, seed?, out?	Generate an image on the local GPU (SDXL/ComfyUI) → {image_path, ...}, or defer.
offload_generate_svg	kind, spec, out?	Render a crisp data-viz SVG (gauge · comparison-bar · chromatogram · icon) — no model, no GPU.
offload_generate_audio	text, kind?, clone?, lang?, seconds?, seed?, out?	Synthesize voice (Chatterbox TTS) or music (ACE-Step) on the local GPU → {audio_path, ...}, or defer.
offload_generate_video	prompt, still?, model?, frames?, seed?, out?	Animate a still into a short clip (Hunyuan I2V) on the local GPU → {video_path, seed}, or defer.
offload_nim	prompt, model?, system?, base?, max_tokens?, temperature?, list_models?	Opt-in remote. Call an NVIDIA NIM endpoint (hosted free catalog or self-hosted) → {model, content, ...}, or defer. Key from $NVIDIA_API_KEY (sent only to NVIDIA hosts); never ledgered.

Inputs stay local. Images, audio, and video are accepted as a local file path or a data: URI — never a remote URL, so there is no network egress for media.

How it works

The pipeline is a confidence-gated cascade. A request enters at a small tier and only climbs when the result is genuinely uncertain; if every local tier is exhausted, it defers to the caller.

flowchart LR
    A[Request] --> B[Cache + context-budget trim]
    B --> C[Build prompt + GBNF grammar]
    C --> D[Small tier<br/>llama.cpp + grammar]
    D --> E[Parse → verify → schema-validate]
    E --> F{Confidence gate}
    F -->|pass| G[Accept ✓]
    F -->|low margin /<br/>validation fail| H[Escalate to larger tier]
    H --> E
    F -->|all tiers fail /<br/>truncated input| I[Defer to caller ↩]
    G --> J[Append to token ledger JSONL]

The cascade. Tasks enter at the tier sized to the job and escalate only on a validation failure or a low decision-confidence signal:

• triage / classify → small fast tier (entry)
• summarize / extract → mid workhorse tier
• on failure or uncertainty → escalate to a larger near-frontier MoE tier
• all local tiers fail → defer to the caller

Confidence-based escalation. For triage/classify the harness requests per-token logprobs and computes a class-mass margin at the decision token (the raw pre-grammar distribution, aggregated by legal class so Yes/yes don't split). A margin below the threshold means the model was torn → escalate instead of accepting a coin-flip. Classify also keeps a self-reported confidence gate (defense in depth).

Reliable structured output. The target model crashes on llama.cpp's --json-schema / response_format, so the harness instead enforces a GBNF grammar (generated per request, no external dependency) via the chat-completions grammar field, then forgivingly parses and schema-validates the result in Go. Extracted values must appear verbatim in the source text (grounding).

State. The cache is bbolt (single-writer); the token ledger is append-only JSONL, so a CLI run and the long-running MCP server can both append concurrently. If the MCP server holds the cache lock, a concurrent CLI run degrades to cache-less automatically rather than failing.

How the cascade learns

The harness improves itself offline and inference-free — pure Go statistics over the ledger, spending zero cloud tokens:

• Conformal thresholds (calibrate) — replaces a guessed margin gate with a per-task threshold that holds a chosen error rate.
• Entry-tier router (train-router) — a tiny logistic model on cheap input features bumps the entry tier up when the small tier is predicted to fail, cutting wasted escalation.
• Health monitoring + circuit breakers (health) — flags degrading tiers (EWMA / Page-Hinkley / CUSUM), sets P95 timeouts, and routes around a tier that is OOMing or timing out (infra only — never on a quality defer).
• Few-shot exemplars (optimize) — harvests verified-good (input, output) pairs and BM25-selects them into the prompt (opt-in via exemplar_shots).
• Shadow-labeling flywheel — optionally captures a fraction of live calls, replays them counterfactually through other tiers to generate training labels for the router and confidence head, then trains and calibrates them behind an adoption gate that only promotes a change when it provably lowers error.

Configuration

Copy config.example.json and edit. Config is resolved in precedence order: --config <path> > $LOCAL_OFFLOAD_CONFIG > ./config.json > ~/.local-offload/config.json > built-in defaults. A leading ~/ in any path-typed value expands to your home directory; an unknown key warns to stderr rather than being silently dropped.

Key	Default	Purpose
endpoint	http://127.0.0.1:11436	Base URL of the local llama.cpp server.
completion_path	/v1/chat/completions	Chat-completions path.
model	offload-e4b	Workhorse text tier (summarize / extract).
triage_model	gemma4-e2b	Fast entry tier (triage / classify); empty = use model.
escalation_model	gemma4-26b-a4b	Larger tier tried before deferring; empty = no escalation.
vision_model	qwen3vl-4b	Local vision tier (VQA / OCR / image extract / assess).
stt_model / stt_model_hq	whisper-stt / whisper-stt-hq	Speech-to-text upstreams (turbo / high-quality).
classify_min_confidence	0.45	Self-reported confidence floor for classify.
confidence_margin_threshold	0.35	Logprob decision margin gate (0 disables).
max_input_chars	24000	Inputs above this are trimmed; over-long inputs defer.
max_retries	1	Retries before escalating.
cache_path	~/.local-offload/cache.db	bbolt cache file.
ledger_path	~/.local-offload/ledger.jsonl	Append-only token-savings ledger.
exemplar_shots	0	Few-shot exemplars to inject (0 = off).
auto_heal	false	Auto-warmup a tripped tier's circuit breaker.
opus_input_price_per_mtok	15.0	Price used to value tokens saved in the ledger.
request_timeout_sec	120	Per-request timeout.

State (cache, ledger, learned weights, exemplars) defaults to ~/.local-offload/.

Serving the models

All tiers are served by a local llama.cpp server (multiplexed with a model-swapper such as llama-swap so only one model occupies the GPU at a time; the harness's default endpoint is http://127.0.0.1:11436). The harness talks to it over the standard chat-completions API. The text cascade fits comfortably on an 8 GB GPU.

Automated setup (Linux + NVIDIA): skill/scripts/detect.sh checks your hardware/toolchain (read-only), and skill/scripts/setup.sh is an idempotent installer — it builds a recent llama.cpp, pulls the Gemma-4 QAT model family, writes the grammar-reliable serving config, and builds the harness. Both are driven entirely by env-var overrides (MODELS_ROOT, LLAMACPP_DIR, LLAMASWAP_CONFIG, LLAMASWAP_PORT, …) — see the headers of each script. skill/ also contains an agent-installable skill for MCP clients like Claude Code.

Verified grammar-reliable serving flags (per tier):

# common
--ctx-size 8192 --flash-attn on --cache-type-k f16 --cache-type-v f16 --jinja --reasoning off

# small entry tier:     --n-gpu-layers 99
# workhorse tier:       --n-gpu-layers 99 --parallel 1
# large MoE escalation: --cpu-moe --n-gpu-layers 999 --parallel 1   (env GGML_CUDA_DISABLE_GRAPHS=1)

Serving gotchas (load-bearing)

• --reasoning off is mandatory — the model's thinking mode otherwise eats the short output budget and returns empty replies.
• No speculative draft (MTP) — it 500-errors on the grammar field. Serve with flash-attention on, f16 KV, reasoning off.
• Never use --json-schema / response_format — they crash the model; the harness passes a raw GBNF grammar field instead.
• Vision: use the Instruct build of the VLM (the Thinking variant silently bypasses GBNF) and keep the multimodal projector at F16 for OCR (Q8 hallucinates text).
• Speech-to-text: flash-attention OFF server-side — it degrades non-English / noisy transcription; turbo is still 5–8× realtime.

Requirements

OS	Linux, macOS, Windows
Go	1.26+ (to build)
GPU	NVIDIA, ~8 GB VRAM for the text + vision cascade
RAM / disk	32 GB+ system RAM and a fast SSD recommended (model weights, MoE CPU offload)
External	a running llama.cpp server; ffmpeg on PATH for audio/video; a whisper.cpp server for STT; ComfyUI for image generation (all optional per feature)

Troubleshooting

Every call returns deferred: true

Check the endpoint with local-offload doctor. The most common cause is the llama.cpp server not running or not reachable at endpoint. A defer is also normal for low-confidence, truncated, or over-long inputs — those are meant to go back to the caller.

Empty or truncated model output

You're almost certainly serving with reasoning mode on. Add --reasoning off to the llama.cpp server. Also confirm you are not passing --json-schema / response_format — both crash the model and break the grammar path.

OCR or image fields look wrong

Use the Instruct vision build (not Thinking) and keep the multimodal projector at F16. Validate dense documents before trusting them — fine-detail OCR on the server has a known accuracy regression for very small text.

"cache unavailable (held by the MCP server?)"

Expected. The bbolt cache is single-writer; when the long-running MCP server holds it, a concurrent CLI run continues cache-less. The JSONL ledger is unaffected — both can append.

Contributing

Contributions welcome. Run go test ./... and go vet ./... before opening a PR, and keep changes scoped. See CONTRIBUTING.md for build/test details.

Security

Everything runs locally — no cloud calls, no credentials, no media egress (inputs are local paths or data: URIs only). To report a vulnerability, please see SECURITY.md rather than opening a public issue.

License

Apache 2.0.

Acknowledgments

Built on llama.cpp and whisper.cpp, the Gemma and Qwen-VL model families, ComfyUI, the Model Context Protocol Go SDK, and bbolt.

Documentation

Overview

Command local-offload delegates short-context grunt work (summarize, classify, extract, triage) to a free local model, exposed as both a CLI and an MCP server. It never calls a cloud model: on failure it returns a structured defer so the caller (Claude) handles the task itself.