gopherllm

package module
v0.3.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 19, 2026 License: MIT Imports: 27 Imported by: 0

README

GopherLLM

GopherLLM is a local GGUF inference tool written in Go. It can run one-shot prompts, interactive REPL sessions, embeddings, model inspection, benchmark runs, and an HTTP server with OpenAI-compatible, Ollama-compatible, and built-in endpoints.

Contents

Features

  • Pure Go runtime with optional ARM64 (NEON) and x86-64 (AVX2 + FMA) assembly kernels.
  • Memory-mapped GGUF loading for fast startup and lower copy pressure, on every platform (Unix mmap, Windows CreateFileMapping/MapViewOfFile): weights page in on demand and quantized tensors borrow the mapping zero-copy.
  • Split/sharded GGUF loading: point at any one shard of a <name>-00001-of-00005.gguf-style download and every sibling is discovered and merged automatically (see Performance Notes).
  • Quantized matrix kernels for Q2_K, Q3_K, Q4_K, Q5_K, Q6_K, Q4_0, Q4_1, Q5_0, Q5_1, Q8_0, Q8_1, and MXFP4 tensors; F32/F16/BF16 loaded directly (BF16 covers QAT-derived and modern full-precision GGUFs).
  • Temperature, top-k, top-p, and min-p sampling with a repetition penalty.
  • OpenAI-compatible tool/function calling, with a native prompt format for Mistral-family models and a generic convention for everything else.
  • Chain-of-thought extraction (<think> blocks, gpt-oss channels) into a separate reasoning_content field instead of leaving it in the answer text.
  • Skills: point --skills-dir at a folder of SKILL.md files and the server resolves the model's load_skill calls itself, agentically, before replying.
  • CLI generation, REPL mode, embeddings, metadata inspection, and tensor listing.
  • HTTP API with /generate, /v1/chat/completions, /v1/completions, /v1/embeddings, /v1/skills, /api/generate, /api/chat, and /api/embeddings.
  • Optional browser chat UI served from the embedded web_ui assets.
  • Model discovery for LM Studio community model directories.

Requirements

  • Go 1.25 or newer.
  • A GGUF text model. By default the tool scans:
~/.cache/lm-studio/models/lmstudio-community

That default is resolved in this order: the --model-dir <path> flag (highest priority), then the GOPHERLLM_MODEL_DIR environment variable (with RUSTY_LLM_MODEL_DIR, the project's pre-rename spelling, still honored as a deprecated fallback), then the built-in default above. MODEL_DIR is a separate thing: it's a Makefile variable (see Make Targets) that make targets use to fill in --model-dir for you — it isn't read by the gopherllm binary itself, so MODEL_DIR=... bin/gopherllm ... (without make) has no effect.

Quickstart

make build                                    # -> bin/gopherllm
bin/gopherllm --model-dir /path/to/models --list-models
bin/gopherllm --model-dir /path/to/models --model "some-model" \
  --prompt "Explain local LLM inference in three sentences." --max-tokens 128

You can also pass an absolute .gguf path directly:

bin/gopherllm /path/to/model.gguf \
  --prompt "Explain local LLM inference in three sentences." \
  --max-tokens 128

Or, with make filling in the CLI flags for you:

make build
make list-models MODEL_DIR=/path/to/models
make run MODEL_DIR=/path/to/models MODEL="some-model" PROMPT="Explain local LLM inference in three sentences."

Use as a Go Library

GopherLLM is an importable module — inference runs in-process, with no child process and no HTTP round-trips:

go get github.com/SimonWaldherr/GopherLLM

import gopherllm "github.com/SimonWaldherr/GopherLLM"

model, err := gopherllm.Open(ctx, "model.gguf")
if err != nil { ... }
defer model.Close()

// One-shot generation with functional options.
res, err := model.Generate(ctx, "Explain GGUF in one sentence.",
    gopherllm.WithMaxTokens(128), gopherllm.WithTemperature(0.7))
fmt.Println(res.Text)

// Streaming (ctx cancels cleanly between tokens).
model.Stream(ctx, []gopherllm.ChatMessage{gopherllm.UserMessage("hi")},
    func(delta string) error { fmt.Print(delta); return nil })

// Embeddings, tokenization, GGUF analysis:
emb, _ := model.Embed(ctx, "semantic search query")
ids := model.Tokenize("hello")
gopherllm.AnalyzeGGUF(model.GGUF(), model.Tokenizer()).WriteText(os.Stdout)

For applications that expose the model over HTTP themselves, the entire OpenAI-/Ollama-compatible API mounts as a plain http.Handler — under any router, prefix, or middleware stack:

mux.Handle("/llm/", http.StripPrefix("/llm",
    model.HTTPHandler(gopherllm.HandlerOptions{Defaults: gopherllm.DefaultGenerationOptions()})))

The library never writes to stdout/stderr on its own; pass gopherllm.WithLogWriter(os.Stderr) (or HandlerOptions.LogWriter) to opt into diagnostics. Tool calling, reasoning extraction, and skills are available via WithTools, Result.ReasoningText, and RunAgenticChat — see the godoc and the runnable examples in example_test.go; testdata/consumer is a complete external application using the API.

Build

make build

The binary is written to bin/gopherllm.

To run formatting, tests, vet, and the release build:

make all

To verify release builds for macOS, Linux, and Windows on amd64 and arm64:

make cross-build

On sandboxed macOS shells, /usr/bin/make may print xcrun_db-* cache warnings before the Makefile can set its build environment. Use the Command Line Tools make directly if that happens:

/Library/Developer/CommandLineTools/usr/bin/make build-metal

CLI Usage

List discovered GGUF models:

bin/gopherllm --model-dir "$HOME/.cache/lm-studio/models/lmstudio-community" --list-models

Run a prompt against a selected model:

bin/gopherllm --model-dir "$HOME/.cache/lm-studio/models/lmstudio-community" \
  --model "model-name-or-file-fragment" \
  --prompt "Explain local LLM inference in three sentences." \
  --max-tokens 128

Run a prompt against an exact GGUF file:

bin/gopherllm /path/to/model.gguf \
  --prompt "Explain local LLM inference in three sentences." \
  --max-tokens 128 \
  --temp 0.7

Start an interactive REPL:

bin/gopherllm --model-dir "$HOME/.cache/lm-studio/models/lmstudio-community" \
  --model "model-name-or-file-fragment" \
  --repl

Run with a skill available (one-shot or REPL alike):

bin/gopherllm --model-dir "$HOME/.cache/lm-studio/models/lmstudio-community" \
  --model "model-name-or-file-fragment" \
  --skills-dir ./skills \
  --prompt "How do I fill out a PDF form on the command line?"

Inspect metadata without loading all weights:

bin/gopherllm /path/to/model.gguf --inspect --list-metadata

Create an embedding:

bin/gopherllm /path/to/model.gguf --embed --prompt "semantic search query"

GGUF Analyzer

Inspect any GGUF's structure without loading weights (instant, even on multi-gigabyte files):

bin/gopherllm /path/to/model.gguf --analyze

Reports architecture/geometry, parameter count, effective bits per weight, the quantization mix per tensor type, rope/sliding-window configuration, tokenizer + detected chat-template family, KV-cache size estimates, and the largest tensors.

Search the vocabulary:

bin/gopherllm /path/to/model.gguf --find-token "weather"

Explore embedding space — which tokens the model treats as related (this loads the weights and scans the embedding table):

bin/gopherllm /path/to/model.gguf --token-neighbors king --neighbors 8
#  34567  "King"      cos=0.5807
#  12566  " king"     cos=0.5079
#  108083 "キング"     cos=0.3692
#  25776  "王"         cos=0.3416

The same features are available in the library as AnalyzeGGUF, SearchTokens, and Model.NearestTokens.

Server

Start the API server with the embedded chat UI:

bin/gopherllm --model-dir "$HOME/.cache/lm-studio/models/lmstudio-community" \
  --model "model-name-or-file-fragment" \
  --serve 127.0.0.1:8080 \
  --chat

Open http://127.0.0.1:8080/chat for the browser UI.

Minimal OpenAI-compatible chat request:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "Write a haiku about Go."}],
    "max_tokens": 64,
    "temperature": 0.7
  }'

Streaming is supported on /v1/chat/completions by setting "stream": true.

Endpoints
Method Path Purpose
GET /health Liveness + loaded model id
POST /generate Native generation API (prompt or messages; accepts tools)
POST /v1/chat/completions OpenAI-compatible chat (streaming, tools, reasoning)
POST /v1/completions OpenAI-compatible text completion
POST /v1/embeddings OpenAI-compatible embeddings
GET /v1/models OpenAI-compatible model listing (the loaded model)
GET /v1/skills Names + descriptions of configured skills
POST /api/generate Ollama-compatible generation
POST /api/chat Ollama-compatible chat (accepts tools)
POST /api/embeddings Ollama-compatible embeddings
GET /models Scan --model-dir and list all discovered GGUFs
POST /models/load Hot-swap the loaded model ({"path": "..."})
GET /chat, /style.css, /script.js Embedded browser chat UI (with --chat)

Tool Use / Agentic

/v1/chat/completions (and the native /generate and Ollama-compatible /api/chat endpoints) accept an OpenAI-shaped tools array. /api/generate and /v1/completions don't (matching the real OpenAI/Ollama APIs, where tools are chat-only), but skills (below) still apply there since those are a server-side capability independent of any client-supplied tools:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{
    "messages": [{"role": "user", "content": "What is the weather in Berlin?"}],
    "tools": [{"type": "function", "function": {
      "name": "get_weather",
      "description": "Get the current weather for a city",
      "parameters": {"type": "object", "properties": {"city": {"type": "string"}}, "required": ["city"]}
    }}]
  }'

A model that decides to call the tool returns finish_reason: "tool_calls" and a message.tool_calls array (content is null when the turn is only a tool call). Continue the conversation by appending the assistant's tool-call message and a role: "tool" message with the result:

{"role": "assistant", "tool_calls": [{"id": "…", "type": "function", "function": {"name": "get_weather", "arguments": "{\"city\": \"Berlin\"}"}}]},
{"role": "tool", "tool_call_id": "…", "content": "{\"temperature_c\": 18, \"conditions\": \"sunny\"}"}

Rendering is native ([AVAILABLE_TOOLS]/[TOOL_CALLS]/[TOOL_RESULTS], verified directly against a real Ministral GGUF's chat_template) for Mistral-family models, and a generic <tool_call>{"name":...,"arguments":...}</tool_call> JSON convention for every other supported chat template. gpt-oss tool calling is not yet implemented (only its reasoning channels are, see below).

Set "tool_choice": "none" to suppress tool offering (and skills, see below) for a single request.

Reasoning

Models that emit <think>...</think> chain-of-thought (DeepSeek-R1, QwQ, etc.) have it split out of the answer and returned separately as reasoning_content on the message (and as delta.reasoning_content when streaming), rather than left mixed into the visible text. gpt-oss's analysis/final channels are parsed the same way, though gpt-oss generation currently still forces the final channel directly in the prompt — see the comment on renderGptOssMessages for how to unlock full channel-based reasoning once validated against a real gpt-oss GGUF.

Skills

Point --skills-dir at a directory of skills, Claude-Agent-Skills style — a name and one-line description are always visible to the model (via a load_skill tool), and the full body is only loaded into context once the model actually asks for it:

skills/
  pdf-fill/SKILL.md
  git-review/SKILL.md

---
name: pdf-fill
description: Fill out a PDF form given field values.
---
Full instructions the model receives once it loads this skill...

When skills are configured, every generation endpoint runs an agentic loop server-side: if the model calls load_skill, the server resolves it internally (feeding the skill body back as a tool result and letting the model continue) before ever returning a response — the client never sees the internal load_skill call. A GET /v1/skills endpoint lists the configured skills' names and descriptions. Tool calls for anything else (i.e. tools the caller supplied) are returned to the caller as usual, even with skills configured. --skills-dir works the same way in one-shot/--repl CLI mode.

Benchmarking and Profiling

Run synthetic Go microbenchmarks:

go test -run '^$' -bench=. -benchmem .

Run an end-to-end generation benchmark against a real GGUF:

bin/gopherllm /path/to/model.gguf \
  --prompt "Wer war Albert Einstein?" \
  --max-tokens 128 \
  --temp 0 \
  --bench --bench-json --bench-runs 3

Time individual model kernels for one transformer layer:

bin/gopherllm /path/to/model.gguf \
  --kernel-bench-json \
  --kernel-bench-runs 25 \
  --kernel-bench-layer 0

Capture a CPU profile during a real generation benchmark:

bin/gopherllm /path/to/model.gguf \
  --prompt "Wer war Albert Einstein?" \
  --max-tokens 128 \
  --temp 0 \
  --bench --bench-json --bench-runs 1 \
  --cpuprofile /tmp/gopherllm.prof

If your Go toolchain includes pprof, inspect it with:

go tool pprof -top bin/gopherllm /tmp/gopherllm.prof

For repeatable comparisons, keep the prompt, token count, sampler settings, thread count, and model path fixed. The first run may include cache and warmup effects, so prefer --bench-runs 3 or more when comparing changes.

Make Targets

  • make run MODEL=... PROMPT='...' builds and runs one prompt.
  • make run-prep MODEL=... runs the prompt with --prepare-quant.
  • make build-metal builds bin/gopherllm-metal with CGO and the metal tag.
  • make run-metal MODEL=... runs with experimental --metal enabled.
  • make run-full MODEL=... and make run-full-prep MODEL=... run 256-token prompt checks without and with --prepare-quant.
  • make run-full-metal MODEL=... and make run-full-metal-prep MODEL=... run 256-token prompt checks with Metal enabled.
  • make run ARGS='...' runs the CLI with a fully custom argument list instead (bypasses MODEL/PROMPT/sampler variables entirely).
  • make repl MODEL=... starts the REPL.
  • make serve MODEL=... CHAT=1 starts the HTTP server and chat UI.
  • make serve-metal MODEL=... CHAT=1 THREADS=8 starts the Metal server with prepared CPU fallback kernels enabled by default (PREPARE_QUANT=0 disables preparation).
  • make list-models scans MODEL_DIR.
  • make inspect MODEL=... prints model metadata summary.
  • make list-tensors MODEL=... prints the tensor inventory.
  • make bench runs Go microbenchmarks.
  • make bench-model MODEL=... runs generation benchmark JSON.
  • make bench-model-prep MODEL=... and make compare-bench MODEL=... benchmark the prepared quant path.
  • make bench-model-metal MODEL=... benchmarks the experimental Metal path.
  • make synonym-bench MODEL=... / make nato-bench MODEL=... run fixed benchmark prompts useful for spotting output-quality regressions.
  • make kernel-bench MODEL=... benchmarks isolated model kernels.
  • make kernel-bench-prep MODEL=... and make compare-kernel-bench MODEL=... benchmark isolated kernels with prepared quant enabled.
  • make kernel-bench-metal MODEL=... benchmarks isolated kernels with Metal enabled.
  • make test, make vet, and make check verify the codebase.
  • make coverage runs the test suite and prints per-function coverage; make coverage-html does the same and opens an HTML report.
  • make cross-build compiles release binaries for macOS, Linux, and Windows on amd64 and arm64.
  • run, repl, and serve all accept SKILLS_DIR=path/to/skills to enable skills; run and repl also accept MIN_P, REPEAT_PENALTY, and SEED alongside the existing TEMP/TOP_P/TOP_K.
  • Run make help for the full target and variable list.

Performance Notes

  • Use --threads <N> to set both GopherLLM worker threads and GOMAXPROCS. Make targets expose the same setting as THREADS=<N>; 8 was fastest in the measured M2 Max setup, but should be re-benchmarked on each target Mac.
  • Use --prepare-quant when slower startup is acceptable; it precomputes Q4_K scale/min data plus selected Q6_K scale data, then switches supported rows to prepared kernels.
  • Use --temp 0 --top-k 1 for deterministic greedy output.
  • Use --min-p <F> (e.g. 0.05) for min-p nucleus sampling; 0 disables it.
  • --bench-json and --kernel-bench-json are intended for repeatable performance comparisons.
  • Metal is available only in bin/gopherllm-metal builds made with CGO_ENABLED=1 -tags metal, and must be enabled with --metal. The selective path fuses mixed Q4_K/Q4_K/Q6_K Q/K/V projections into one command buffer and offloads Q4_K attention-output, Q4_K gate/up + SiLU + Q6_K FFN-down in one command buffer, and Q6_K vocabulary-output projections. GGUF files opened through mmap are exposed to Metal as shared no-copy weight buffers; byte-backed models retain the copying path for cgo safety. Prepared ARM64 kernels remain as the fallback for small projections and Metal failures. The path remains experimental; use --kernel-bench-json and --bench-json on the target Mac before deployment.
  • On x86-64 (AVX2 + FMA + F16C, auto-detected via CPUID), Q4_K, Q5_K, Q6_K, Q8_0, Q4_0, Q4_1, and MXFP4 matvecs default to int8-activation full-row kernels: the activation vector is quantized once per matvec to int8 with one scale per 256-element block (llama.cpp's Q8_K convention, q8kQuantize), and each weight row is processed by a single assembly call (q4kDotQ8KRow / q5kDotQ8KRow / q6kDotQ8KRow / q8_0DotQ8KRow) that decodes block scales in-register, dots 32 weights per VPMADDUBSW (Q8_0's own signed weights use the abs/sign-restore identity so the same unsigned-operand instruction applies), applies scales via VPMADDWD, and reduces horizontally once per row. Versus the previous per-block float kernels this is ~2.5x (Q4_K) to ~6x (Q6_K and Q8_0) per-row — and >20x for Q5_K, which previously had no SIMD fast path at all — and roughly 4x end-to-end decode on a Ministral 3B Q4_K_M. Set GOPHERLLM_Q8_ACTIVATIONS=0 to force the exact float kernels (bit-reproducible against the scalar reference; the int8 path stays within cosine 0.999 of it — the same accuracy tradeoff llama.cpp makes by default). GOPHERLLM_DISABLE_SIMD=1 still forces portable scalar kernels everywhere.
  • Prompt processing (prefill) is batched. With the int8 path active, each raw quantized weight row is streamed from memory exactly once per prompt chunk and dotted against all prompt tokens' pre-quantized int8 activations in L2-resident row tiles (matvecBatchQ8) — no f32 dequantization pass at all. With GOPHERLLM_Q8_ACTIVATIONS=0 the older dequantize-once-per-chunk f32 path runs instead. ARM64 reuses per-worker dequantization rows and dispatches one coarse batch range per worker to avoid allocation and scheduling overhead. Set GOPHERLLM_NO_BATCH_PREFILL=1 to fall back to the per-token path (A/B benchmarking / debugging), or GOPHERLLM_PREFILL_CHUNK=<N> to tune the chunk size on the deployment machine.
  • SwiGLU's x*sigmoid(x)*up runs through an AVX2 kernel with a Cephes-style expf polynomial (~1e-7 relative error) instead of per-element math.Exp.
  • On ARM64, Q4_K and Q6_K matvecs use NEON block kernels, attention heads are spread across the worker pool at longer contexts, and single-token matvec work is over-chunked so performance cores absorb efficiency-core stragglers.
  • Set GOPHERLLM_DISABLE_YARN=1 to skip YaRN RoPE scaling for models that declare it.
  • Split GGUFs (llama.cpp's gguf-split naming convention, <name>-00001-of-00005.gguf) are detected from any one shard's split.count metadata; every sibling is located next to it, and their tensor data is merged into one in-memory buffer before loading. This costs one full copy of the model's weights at load time — true zero-copy mmap borrowing only applies to single-file GGUFs — but needs no other opt-in.
  • On x86-64 (F16C) the KV cache stores K/V rows as f16 by default: half the cache memory (double the context fits the reusable-workspace cap) and half the bytes attention streams per generated token, with rows converted in-register (VCVTPH2PS) inside the attention kernels. Greedy decode on the test model is bit-identical to the f32 cache; set GOPHERLLM_KV_F16=0 to force the exact f32 cache. Attention itself is two-pass (independent score dots, then max-stabilized softmax weights and the weighted V accumulation), which measured ~1.15x over the previous online-softmax loop at 4k-16k context and uses the true score maximum for stability.
  • After mmap'ing a single-file GGUF, every page is touched once up front across all worker threads (prefaultPages) before the model is reported loaded. A memory-mapped file only pages in on first touch, and a forward pass touches essentially every weight byte — without this, the first request after startup silently inherited that page-in cost (disk I/O, or on Windows, real-time antivirus scanning of each mapped page) inside its own TTFT instead of load time. For a one-shot CLI run this doesn't change total wall-clock; for the HTTP server and REPL cases it means every request, including the first, sees consistent latency instead of one random request eating a multi-second page-in tax. Set GOPHERLLM_NO_PREFAULT=1 to restore pure lazy paging.
Environment variables

Quick reference for the runtime toggles described above (unset by default; details in the bullets they annotate):

Variable Effect
GOPHERLLM_MODEL_DIR Default model directory when --model-dir is not given (RUSTY_LLM_MODEL_DIR remains a deprecated fallback)
GOPHERLLM_DISABLE_SIMD Force portable scalar kernels (skip AVX2 detection)
GOPHERLLM_NO_BATCH_PREFILL Per-token prefill instead of batched
GOPHERLLM_PREFILL_CHUNK Override batched-prefill chunk size (1-256)
GOPHERLLM_Q8_ACTIVATIONS 0 disables the default int8-activation Q4_K/Q5_K/Q6_K/Q8_0/Q4_0/Q4_1/MXFP4 matvecs (x86-64)
GOPHERLLM_NO_PREFAULT Skip the post-mmap page warm-up; restores pure lazy paging
GOPHERLLM_KV_F16 0 stores the KV cache as exact f32 instead of f16 (x86-64)
GOPHERLLM_METAL_ROWS_PER_GROUP Override Metal rows per threadgroup (2, 4, 6, or 8; default 4)
GOPHERLLM_METAL_FUSED_FFN 0 disables Metal Gate/Up + SiLU + Down fusion
GOPHERLLM_DISABLE_YARN Ignore declared YaRN RoPE scaling

Supported Architectures

The loader currently accepts GGUF files whose general.architecture is one of:

llama, llama2, llama3, mistral, mistral3, ministral, mixtral, qwen2, qwen3,
gpt-oss, gemma, gemma2, gemma4, nemotron_h, nemotron_h_moe

qwen3 (including the DeepSeek-R1 Qwen3 distills) adds per-head QK-norm on top of the qwen2 graph; DeepSeek-R1 reasoning output is separated into reasoning_content in both template conventions (self-opened <think> blocks and the newer forced-open templates whose output begins mid-reasoning). deepseek2 (MLA attention) is not supported. Mistral-family models support assistant-message prefill: a conversation ending in an assistant message leaves the turn open so generation continues it.

Mistral-family instruct models (including Ministral) use the [INST]…[/INST] chat format, the Tekken byte-level BPE pre-tokenizer, and YaRN RoPE context scaling when the GGUF declares it.

nemotron_h_moe is the native hybrid Mamba-2 / attention / sparse-MoE graph used by Soofi S Isar. It retains Mamba convolution and SSM state locally and does not rely on a llama.cpp process. Prompt prefill is deliberately per-token for this architecture because its recurrent state makes the regular batched transformer prefill invalid.

Gemma-family support (gemma/gemma2/gemma4, including the Gemma QAT GGUFs) is experimental: the dense Gemma graph is implemented — hardcoded sqrt(dim) embedding scaling, GELU FFN, QK-norm, post-attention/post-FFN norms, attention/final logit softcapping, the per-layer sliding-window map (explicit sliding_window_pattern bool-array metadata or the known Gemma 2/4 interleave defaults), the <start_of_turn> chat template, and <end_of_turn> as a stop token — but it has not been validated against real Gemma weights yet, and the Gemma 4-specific mechanisms (p-RoPE frequency factors, per-layer RoPE bases, cross-layer KV sharing, per-layer embeddings, the 26B MoE) are still missing. The loader prints a warning. See docs/INFERENCE_NOTES.md for the architecture notes, QAT specifics, and per-family recommended sampling settings (e.g. Gemma: --temp 1.0 --top-p 0.95 --top-k 64).

Projector files such as mmproj-* are detected and excluded from text-model selection.

Development

Project layout
Area Files
GGUF parsing + file mapping gguf.go, mmap.go (Unix) / mmap_windows.go (Win32 file mapping)
Model loading + forward pass model.go, forward_batch.go (batched prefill)
Compute kernels + worker pool simd.go; assembly in *_amd64.s / *_arm64.s behind dot_f32_*.go, vector_ops_*.go, quant_*.go, q4k_q8_*.go dispatch shims
Tokenizers tokenizer.go (SentencePiece + GPT-2/Tekken BPE)
Sampling sampling.go
Generation orchestration + chat templates runtime.go
Tool calling / reasoning / skills tools.go, extract.go, agent.go, skills.go
Model discovery + selection catalog.go
HTTP server server.go, web_ui/
CLI cmd/gopherllm/main.go, lib.go (package doc + version), kernel_bench.go

A full architecture walkthrough — load path, inference data flow, kernel dispatch tiers, and how to add a quant kernel / architecture / endpoint — is in docs/ARCHITECTURE.md.

The same map, with more detail, is in the package comment in doc.go. Every SIMD kernel has a portable Go scalar reference implementation, and differential tests assert they agree — when touching a kernel, run the Q4K/ Q6K/DotF32/VectorOps test groups first. Model-behavior research notes (Gemma 4 / QAT specifics, per-family sampling recommendations) live in docs/INFERENCE_NOTES.md.

Run the full local check:

make check

Check test coverage:

make coverage      # per-function summary in the terminal
make coverage-html  # same, plus an interactive HTML report

Run a focused benchmark:

go test -run '^$' -bench=BenchmarkMatvecQ4K -benchmem .

Profile a real-model benchmark:

bin/gopherllm /path/to/model.gguf --prompt "test" --max-tokens 128 \
  --temp 0 --bench --bench-json --bench-runs 1 \
  --cpuprofile /tmp/gopherllm.prof

Local build artifacts are kept in bin/ and .cache/, both ignored by git.

GitHub Actions runs go test, go vet, and go build on Linux, macOS, and Windows, plus the make cross-build release matrix on Linux.

Documentation

Overview

Package gopherllm is a pure-Go GGUF inference runtime: it memory-maps a quantized model file, runs the transformer forward pass on CPU (with optional ARM64 NEON / x86-64 AVX2 assembly kernels), and exposes generation, chat, streaming, embeddings, tokenization, and GGUF analysis directly to Go programs — no external process, no HTTP round-trips. 

import gopherllm "github.com/SimonWaldherr/GopherLLM"

model, err := gopherllm.Open(ctx, "model.gguf")
if err != nil { ... }
defer model.Close()

res, err := model.Generate(ctx, "Explain GGUF in one sentence.",
    gopherllm.WithMaxTokens(128), gopherllm.WithTemperature(0.7))
fmt.Println(res.Text)

API layers

  • Model (api.go): the primary embedding API — context-first methods with functional options (Open, Generate, Chat, Stream, Embed, Tokenize, Detokenize). Start here.
  • NewHandler (server.go): the OpenAI-/Ollama-compatible HTTP API as a mountable http.Handler for applications that expose the model over HTTP themselves.
  • Runner (runtime.go): the lower-level engine underneath both, exposed for advanced uses (the agentic skill loop via RunAgenticChat, kernel benchmarking, custom loops).
  • AnalyzeGGUF / SearchTokens (analyze.go): header-only model structure reports and vocabulary inspection without loading weights.

The library never writes to stdout/stderr on its own; pass WithLogWriter (or HandlerOptions.LogWriter) to opt into diagnostics.

A rough map of the internals:

  • gguf.go GGUF container parsing (header, metadata, tensor table)
  • mmap*.go memory-mapped file access (per-OS)
  • model.go model config, weight loading, transformer forward pass
  • forward_batch.go batched prefill (prompt tokens processed per chunk)
  • simd.go, quant_extra.go matvec/dot kernels + dequantization + pool
  • *_amd64.s / *_arm64.s hand-written SIMD kernels behind runtime dispatch
  • tokenizer.go SentencePiece and GPT-2/Tekken BPE tokenizers
  • sampling.go temperature/top-k/top-p/min-p sampling
  • runtime.go Runner: generation loop, chat templates per model family
  • tools.go, extract.go, agent.go, skills.go tool calling, reasoning extraction, the server-side skill loop
  • catalog.go model discovery/selection in a models directory
  • cmd/gopherllm CLI built on all of the above
Example

Example demonstrates the simplest possible use: load a model, generate a completion, print it. 

package main

import (
	"context"
	"fmt"
	"log"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	ctx := context.Background()
	model, err := gopherllm.Open(ctx, "model.gguf")
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	res, err := model.Generate(ctx, "Explain GGUF in one sentence.",
		gopherllm.WithMaxTokens(128),
		gopherllm.WithTemperature(0.7),
	)
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(res.Text)
}

Index

Examples

Constants

View Source 
const LoadSkillToolName = "load_skill"

LoadSkillToolName is the name of the tool RunAgenticChat resolves internally (see agent.go) rather than returning to the caller.

View Source 
const Version = "0.3.0-go"

Version is reported by the CLI's --version and usage header. (The package documentation lives in doc.go.)

Variables

View Source 
var ErrGenerationCanceled = errors.New("generation canceled")

Functions

func ArchitectureSupported 

func ArchitectureSupported(arch string) bool

ArchitectureSupported reports whether the loader accepts this general.architecture value. Notes on specific families:

  • qwen3 (incl. the DeepSeek-R1-0528 Qwen3 distills): the qwen2 graph plus per-head QK-norm, which loads via the optional attn_q_norm/attn_k_norm tensors and applies exactly as for Gemma 3/4.
  • deepseek2 (MLA attention) is NOT supported; DeepSeek-R1 distills ship as qwen2/qwen3/llama and work through those graphs.
  • Devstral and Mistral-Small GGUFs usually declare llama or mistral3; their [INST]/Tekken behavior is picked up from tokenizer metadata, not the arch string.

func ArgmaxOutputToken added in v0.2.0 

func ArgmaxOutputToken(config Config, weights ModelWeights, buf *DecodeBuffer) (uint32, bool)

func AxpyF32 

func AxpyF32(out []float32, alpha float32, x []float32)

func CosineSimilarity 

func CosineSimilarity(a, b []float32) (float32, error)

func DefaultModelDir 

func DefaultModelDir() string

DefaultModelDir returns the directory scanned when --model-dir is not given: $GOPHERLLM_MODEL_DIR if set, else the deprecated $RUSTY_LLM_MODEL_DIR (the project's pre-rename spelling, kept so existing environments keep working), else the LM Studio community models directory under $HOME. (MODEL_DIR is a Makefile variable, not read here.)

func DequantRowMXFP4 

func DequantRowMXFP4(row []byte, cols int) []float32

func DequantRowQ2K 

func DequantRowQ2K(row []byte, cols int) []float32

func DequantRowQ2KInto 

func DequantRowQ2KInto(row []byte, cols int, out []float32)

func DequantRowQ3K 

func DequantRowQ3K(row []byte, cols int) []float32

func DequantRowQ3KInto 

func DequantRowQ3KInto(row []byte, cols int, out []float32)

func DequantRowQ4K 

func DequantRowQ4K(row []byte, cols int) []float32

func DequantRowQ4KInto 

func DequantRowQ4KInto(row []byte, cols int, out []float32)

func DequantRowQ4_0 

func DequantRowQ4_0(row []byte, cols int) []float32

func DequantRowQ4_1 

func DequantRowQ4_1(row []byte, cols int) []float32

func DequantRowQ4_1Into 

func DequantRowQ4_1Into(row []byte, cols int, out []float32)

func DequantRowQ5K 

func DequantRowQ5K(row []byte, cols int) []float32

func DequantRowQ5_0 

func DequantRowQ5_0(row []byte, cols int) []float32

func DequantRowQ5_0Into 

func DequantRowQ5_0Into(row []byte, cols int, out []float32)

func DequantRowQ5_1 

func DequantRowQ5_1(row []byte, cols int) []float32

func DequantRowQ5_1Into 

func DequantRowQ5_1Into(row []byte, cols int, out []float32)

func DequantRowQ6K 

func DequantRowQ6K(row []byte, cols int) []float32

func DequantRowQ6KInto 

func DequantRowQ6KInto(row []byte, cols int, out []float32)

func DequantRowQ8_0 

func DequantRowQ8_0(row []byte, cols int) []float32

func DequantRowQ8_1 

func DequantRowQ8_1(row []byte, cols int) []float32

func DequantRowQ8_1Into 

func DequantRowQ8_1Into(row []byte, cols int, out []float32)

func DotF32 

func DotF32(a, b []float32) float32

func DotMXFP4F32 

func DotMXFP4F32(row []byte, x []float32, cols int) float32

func DotQ2KF32 

func DotQ2KF32(row []byte, x []float32, cols int) float32

func DotQ3KF32 

func DotQ3KF32(row []byte, x []float32, cols int) float32

func DotQ4KF32 

func DotQ4KF32(row []byte, x []float32, cols int) float32

func DotQ4_0F32 

func DotQ4_0F32(row []byte, x []float32, cols int) float32

func DotQ4_1F32 

func DotQ4_1F32(row []byte, x []float32, cols int) float32

func DotQ5KF32 

func DotQ5KF32(row []byte, x []float32, cols int) float32

func DotQ5_0F32 

func DotQ5_0F32(row []byte, x []float32, cols int) float32

func DotQ5_1F32 

func DotQ5_1F32(row []byte, x []float32, cols int) float32

func DotQ6KF32 

func DotQ6KF32(row []byte, x []float32, cols int) float32

func DotQ8_0F32 

func DotQ8_0F32(row []byte, x []float32, cols int) float32

func DotQ8_1F32 

func DotQ8_1F32(row []byte, x []float32, cols int) float32

func F16ToF32 

func F16ToF32(h uint16) float32

func F32ToF16 added in v0.3.0 

func F32ToF16(f float32) uint16

F32ToF16 converts a float32 to IEEE 754 half-precision bits with round-to-nearest-even, the scalar counterpart of F16ToF32.

func Forward 

func Forward(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int) []float32

Forward runs one token through the transformer and returns its next-token logits; ForwardInto is the allocation-free form. Both append the token's K/V to the cache at position pos as a side effect.

func ForwardBatchInto 

func ForwardBatchInto(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, tokens []uint32, startPos int, computeLast bool, logits *[]float32)

ForwardBatchInto processes a chunk of prompt tokens (positions startPos..startPos+len(tokens)-1) through the standard transformer, populating the KV cache. The matvecs are batched so each weight is streamed once for the whole chunk. When computeLast is set, the final token's logits are written to logits. Only the non-fused standard path is supported (callers must check).

func ForwardBodyInto 

func ForwardBodyInto(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int)

ForwardBodyInto is the transformer body shared by logits, prefill, and embedding paths: embed the token, then per layer run pre-norm attention (RoPE'd Q/K, K/V appended to the cache, online-softmax attention over all cached positions, output projection, residual add) followed by a pre-norm SwiGLU FFN, and finally apply the output norm, leaving the normed hidden state in buf.XN for the caller to project (or pool, for embeddings). Attention heads are spread across the worker pool once the attended span is long enough to amortize dispatch (see the comment at the call site); the Q/K/V and gate/up matvecs go through the fused multi-matrix kernels when the quant types allow (tryMatvec3Into/tryMatvec2Into).

func ForwardGemma4Into 

func ForwardGemma4Into(config Config, weights Gemma4Weights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int, logits *[]float32)

func ForwardGptOssInto 

func ForwardGptOssInto(config Config, weights GptOssWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int, logits *[]float32)

func ForwardHidden 

func ForwardHidden(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int) []float32

func ForwardHiddenGemma4 

func ForwardHiddenGemma4(config Config, weights Gemma4Weights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int) []float32

func ForwardHiddenGptOss 

func ForwardHiddenGptOss(config Config, weights GptOssWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int) []float32

func ForwardInto 

func ForwardInto(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int, logits *[]float32)

func ForwardNemotronHBodyInto added in v0.3.0 

func ForwardNemotronHBodyInto(cfg Config, weights NemotronHWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int)

func ForwardNemotronHInto added in v0.3.0 

func ForwardNemotronHInto(cfg Config, weights NemotronHWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int, logits *[]float32)

func ForwardPrefill 

func ForwardPrefill(config Config, weights ModelWeights, cache *KVCache, buf *DecodeBuffer, token uint32, pos int)

func LoadGemma4Model 

func LoadGemma4Model(data []byte, gguf *GGUFFile, borrowQuantized, prepareQuantized, useMetal bool, logw io.Writer) (Config, Gemma4Weights, error)

func LoadGptOssModel 

func LoadGptOssModel(data []byte, gguf *GGUFFile, borrowQuantized, prepareQuantized, useMetal bool, logw io.Writer) (Config, GptOssWeights, error)

func LoadModel 

func LoadModel(data []byte, gguf *GGUFFile, borrowQuantized, prepareQuantized, useMetal bool, logw io.Writer) (Config, ModelWeights, error)

LoadModel loads the standard llama-style weight set from a parsed GGUF. With borrowQuantized set (the mmap path), quantized tensors are zero-copy sub-slices of data — the caller must keep data alive for the model's lifetime; without it they are copied into owned memory (the in-memory test path). Models without a separate output.weight tie the output projection to the token embeddings.

func LoadNemotronHModel added in v0.3.0 

func LoadNemotronHModel(data []byte, gguf *GGUFFile, borrow, prepareQuantized, useMetal bool, logw io.Writer) (Config, NemotronHWeights, error)

func MatvecF32 

func MatvecF32(data, x []float32, rows, cols int) []float32

func MatvecF32Into 

func MatvecF32Into(data, x []float32, rows, cols int, out *[]float32)

func MatvecMXFP4Into 

func MatvecMXFP4Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecPreparedQ4K2IntoWithXSums 

func MatvecPreparedQ4K2IntoWithXSums(aData []byte, aPrep *PreparedQuantizedWeight, aRows, aCols int, bData []byte, bPrep *PreparedQuantizedWeight, bRows, bCols int, x []float32, xSums *[]float32, aOut, bOut *[]float32) bool

func MatvecPreparedQ4K3IntoWithXSums 

func MatvecPreparedQ4K3IntoWithXSums(aData []byte, aPrep *PreparedQuantizedWeight, aRows, aCols int, bData []byte, bPrep *PreparedQuantizedWeight, bRows, bCols int, cData []byte, cPrep *PreparedQuantizedWeight, cRows, cCols int, x []float32, xSums *[]float32, aOut, bOut, cOut *[]float32) bool

func MatvecPreparedQ4KInto 

func MatvecPreparedQ4KInto(data []byte, p *PreparedQuantizedWeight, x []float32, rows, cols int, out *[]float32) bool

func MatvecPreparedQ6KInto 

func MatvecPreparedQ6KInto(data []byte, p *PreparedQuantizedWeight, x []float32, rows, cols int, out *[]float32) bool

func MatvecQ2KInto 

func MatvecQ2KInto(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ3KInto 

func MatvecQ3KInto(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ4K2Into 

func MatvecQ4K2Into(aData []byte, aRows, aCols int, bData []byte, bRows, bCols int, x []float32, aOut, bOut *[]float32) bool

func MatvecQ4K2IntoWithXSums 

func MatvecQ4K2IntoWithXSums(aData []byte, aRows, aCols int, bData []byte, bRows, bCols int, x []float32, xSums *[]float32, aOut, bOut *[]float32) bool

func MatvecQ4K2Q6KIntoWithXSums added in v0.2.0 

func MatvecQ4K2Q6KIntoWithXSums(aData []byte, aRows, aCols int, bData []byte, bRows, bCols int, cData []byte, cRows, cCols int, x []float32, q4Sums *[]float32, aOut, bOut, cOut *[]float32) bool

func MatvecQ4KInto 

func MatvecQ4KInto(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ4_0 

func MatvecQ4_0(data []byte, x []float32, rows, cols int) []float32

func MatvecQ4_0Into 

func MatvecQ4_0Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ4_1Into 

func MatvecQ4_1Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ5KInto 

func MatvecQ5KInto(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ5_0Into 

func MatvecQ5_0Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ5_1Into 

func MatvecQ5_1Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ6K2Into 

func MatvecQ6K2Into(aData []byte, aRows, aCols int, bData []byte, bRows, bCols int, x []float32, aOut, bOut *[]float32) bool

func MatvecQ6K3Into 

func MatvecQ6K3Into(aData []byte, aRows, aCols int, bData []byte, bRows, bCols int, cData []byte, cRows, cCols int, x []float32, aOut, bOut, cOut *[]float32) bool

func MatvecQ6KInto 

func MatvecQ6KInto(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ8_0 

func MatvecQ8_0(data []byte, x []float32, rows, cols int) []float32

func MatvecQ8_0Into 

func MatvecQ8_0Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MatvecQ8_1Into 

func MatvecQ8_1Into(data []byte, x []float32, rows, cols int, out *[]float32)

func MetalAvailable 

func MetalAvailable() bool

func MetalError 

func MetalError() string

func NewHandler 

func NewHandler(initialRunner *Runner, opts HandlerOptions) http.Handler

NewHandler returns the complete GopherLLM HTTP API (OpenAI-compatible, Ollama-compatible, and native endpoints — see the README's endpoint table) as a mountable http.Handler. It owns no listener and writes nothing except to opts.LogWriter, so it composes with any router, middleware stack, or server the host application already has.

Example

ExampleNewHandler mounts the OpenAI/Ollama-compatible HTTP API inside an existing application server, under a path prefix, with the host's own middleware and lifecycle — no bundled server. 

package main

import (
	"context"
	"log"
	"net/http"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	model, err := gopherllm.Open(context.Background(), "model.gguf")
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	mux := http.NewServeMux()
	mux.Handle("/llm/", http.StripPrefix("/llm", model.HTTPHandler(gopherllm.HandlerOptions{
		Defaults: gopherllm.DefaultGenerationOptions(),
	})))
	// mux also serves the application's own routes...
	log.Fatal(http.ListenAndServe("127.0.0.1:8080", mux))
}

func PrintModelList 

func PrintModelList(entries []ModelEntry)

func ProjectLogitsInto added in v0.2.0 

func ProjectLogitsInto(config Config, weights ModelWeights, buf *DecodeBuffer, logits *[]float32)

func Q4KMatvec3Into 

func Q4KMatvec3Into(wq, wk, wv Q4KMatrix, x []float32, q, k, v *[]float32) bool

Q4KMatvec3Into computes three Q4_K matvecs (attention Q/K/V projections) against the same activation vector in one fused pass: the per-32-element activation sums are computed once and the combined row range is spread over the worker pool as a single parallel dispatch instead of three. Returns false (nothing written) if shapes are incompatible, in which case the caller falls back to three separate matvecs.

func Q4KMatvec3IntoWithXSums 

func Q4KMatvec3IntoWithXSums(wq, wk, wv Q4KMatrix, x []float32, xSums *[]float32, q, k, v *[]float32) bool

func ResolveModelPath 

func ResolveModelPath(selection *string, modelDir string) (string, error)

ResolveModelPath turns the CLI's model selector into a concrete .gguf path. A selector that is an existing file wins outright; an existing directory (or a nil selector) opens the interactive picker over that directory; anything else is matched against the discovered models by SelectModel.

func RunKernelBench 

func RunKernelBench(r *Runner, modelPath string, runs, requestedLayer int, jsonOut bool) error

RunKernelBench times each weight matvec of one transformer layer (plus the embedding row lookup and the output projection) in isolation, printing a table or, with jsonOut, the machine-readable llm-kernel-bench.v1 document that `make kernel-bench` emits for cross-runtime comparisons.

func RunnerFromPath 

func RunnerFromPath(path string) (*Runner, LoadInfo, error)

RunnerFromPath memory-maps a GGUF file and loads it with zero-copy borrowed quantized weights. Silent; prefer Open, which adds context support, configurable logging, and a higher-level Model wrapper.

func RunnerFromPathWithOptions 

func RunnerFromPathWithOptions(path string, options LoadOptions) (*Runner, LoadInfo, error)

func Sample 

func Sample(logits []float32, config SamplerConfig, rng *Rng, recent []uint32) uint32

Sample picks the next token id from logits. recent is the trailing token window the repeat penalty applies to. NOTE: logits is mutated in place (penalties and softmax are applied destructively) — callers must not reuse the slice's prior contents afterwards.

func SampleWithScratch 

func SampleWithScratch(logits []float32, config SamplerConfig, rng *Rng, recent []uint32, candidates *[]TokenProb) uint32

SampleWithScratch is Sample with a caller-owned candidate scratch buffer so the per-token decode loop allocates nothing. Same logits-mutation caveat.

func ScaleAddF32 

func ScaleAddF32(out []float32, alpha float32, x []float32)

func ScaleF32 

func ScaleF32(out []float32, alpha float32)

func Serve 

func Serve(initialRunner *Runner, opts ServeOptions) error

Serve builds the API handler and runs a blocking http.Server on opts.Addr. Library consumers who want to control the server lifecycle, add middleware, TLS, or mount the API under a path prefix should use NewHandler instead: 

handler := gopherllm.NewHandler(model.Runner(), gopherllm.HandlerOptions{...})
mux.Handle("/llm/", http.StripPrefix("/llm", handler))

func SetNumThreads 

func SetNumThreads(n int)

SetNumThreads overrides the worker count used by the parallel matvec dispatch (default GOMAXPROCS). The CLI's --threads flag calls this and sets GOMAXPROCS to the same value.

Types

type APIMessage 

type APIMessage struct {
	Role       string     `json:"role"`
	Content    any        `json:"content"`
	ToolCalls  []ToolCall `json:"tool_calls,omitempty"`
	ToolCallID string     `json:"tool_call_id,omitempty"`
	Name       string     `json:"name,omitempty"`
}

type Analysis 

type Analysis struct {
	Architecture string
	Name         string
	Version      uint32
	Supported    bool

	Params        int64 // total weight count across all tensors
	FileBytes     int64 // sum of tensor bytes (excludes the small header)
	BitsPerWeight float64

	Layers        int
	Dim           int
	HiddenDim     int
	Heads         int
	KVHeads       int
	HeadDim       int
	VocabSize     int
	ContextLength int
	SlidingWindow int
	SWALayers     int // layers using the sliding window (0 = none, -1 = all)

	RopeTheta       float32
	RopeScalingType string

	TokenizerModel string
	TokenizerPre   string
	ChatTemplate   bool
	TemplateKind   string
	BOSID, EOSID   uint32

	DTypes         []DTypeStat  // sorted by byte share, descending
	LargestTensors []TensorStat // top 5 by bytes

	// KVCacheBytesAtFullContext / At4K estimate the f32 KV cache footprint.
	KVCacheBytesAtFullContext int64
	KVCacheBytesAt4K          int64
}

Analysis is a structural report over a GGUF header: identity, geometry, quantization mix, tokenizer properties, and derived estimates. Everything here comes from the header and metadata — no tensor data is read — so analyzing a multi-gigabyte file is instant.

func AnalyzeGGUF 

func AnalyzeGGUF(g *GGUFFile, tok *Tokenizer) *Analysis

AnalyzeGGUF builds an Analysis from a parsed GGUF. tok may be nil (tokenizer metadata is then reported from raw metadata only, without template detection).

Example

ExampleAnalyzeGGUF inspects a model file's structure without loading any weights. 

package main

import (
	"log"
	"os"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	mmap, err := gopherllm.OpenMmap("model.gguf")
	if err != nil {
		log.Fatal(err)
	}
	defer mmap.Close()
	gguf, err := gopherllm.ParseGGUF(mmap.Bytes())
	if err != nil {
		log.Fatal(err)
	}
	tok, _ := gopherllm.TokenizerFromMetadata(gguf.Metadata)
	gopherllm.AnalyzeGGUF(gguf, tok).WriteText(os.Stdout)
}

func (*Analysis) WriteText 

func (a *Analysis) WriteText(w io.Writer)

WriteText renders the analysis as a human-readable report.

type ChatMessage 

type ChatMessage struct {
	Role    ChatRole
	Content string
	// ToolCalls is set on an assistant message that is replaying a prior turn
	// in which the model requested one or more tool calls.
	ToolCalls []ToolCall
	// ToolCallID and Name identify which prior tool call a ChatRoleTool
	// message is answering.
	ToolCallID string
	Name       string
}

func AssistantMessage 

func AssistantMessage(content string) ChatMessage

func ToolResultMessage 

func ToolResultMessage(callID, name, content string) ChatMessage

ToolResultMessage renders the output of tool call callID (named name) back into the conversation for the model to see on its next turn.

func UserMessage 

func UserMessage(content string) ChatMessage

type ChatRole 

type ChatRole int
const (
	ChatRoleSystem ChatRole = iota
	ChatRoleUser
	ChatRoleAssistant
	// ChatRoleTool carries the result of a previously requested tool call back
	// to the model. ToolCallID must match the id the assistant's ToolCalls
	// entry used.
	ChatRoleTool
)

type Config 

type Config struct {
	Arch                      string
	Dim                       int
	HiddenDim                 int
	NLayers                   int
	NHeads                    int
	NKVHeads                  int
	VocabSize                 int
	MaxSeqLen                 int
	RopeTheta                 float32
	RMSNormEps                float32
	AttentionScale            float32
	EmbeddingScale            float32
	ResidualScale             float32
	LogitScale                float32
	HeadDim                   int
	KVDim                     int
	KVMul                     int
	ValueDim                  int
	SlidingWindow             int
	ExpertCount               int
	ExpertUsedCount           int
	RopeDimensionCount        int
	RopeScalingFactor         float32
	RopeAttentionFactor       float32
	RopeOriginalContextLength int
	RopeScalingType           string
	RopeYarnBetaFast          float32
	RopeYarnBetaSlow          float32
	RopeYarnLogMultiplier     float32
	RopeFactorsLong           []float32
	RopeFactorsShort          []float32
	// Gemma-family mechanics (all inert at their zero values; see
	// docs/INFERENCE_NOTES.md for the researched semantics):
	// UseGELU switches the FFN activation from SiLU to tanh-approximated GELU.
	// AttnLogitSoftcap/FinalLogitSoftcap apply cap*tanh(v/cap) to attention
	// scores / final logits (Gemma 2: 50.0 / 30.0). SWAPattern, when non-nil,
	// restricts the sliding window to layers whose entry is true (Gemma 4
	// ships it as bool-array metadata; Gemma 2's alternating pattern is
	// synthesized); nil means SlidingWindow applies to every layer.
	UseGELU           bool
	AttnLogitSoftcap  float32
	FinalLogitSoftcap float32
	SWAPattern        []bool
	// Nemotron-H / Soofi S hybrid Mamba-2 configuration. These fields are
	// unused by the standard transformer path. A zero-valued per-layer entry
	// means that the layer does not expose that component.
	LayerHeads            []int
	LayerKVHeads          []int
	LayerFFNDim           []int
	SSMConv               int
	SSMInner              int
	SSMState              int
	SSMHeads              int
	SSMGroups             int
	ExpertWeightsNorm     bool
	ExpertWeightsScale    float32
	ExpertWeightsNormClip float32
}

Config is the model's hyperparameter set, read from GGUF metadata by ConfigFromGGUF and then refined against actual tensor shapes by inferAttentionShape (GGUF metadata is frequently missing or wrong about head dims, so the tensor shapes are authoritative).

Attention shape vocabulary used throughout the forward pass: HeadDim is the per-head Q/K width, ValueDim the per-head V width (usually equal), NKVHeads the number of K/V heads (< NHeads under grouped-query attention), KVMul = NHeads/NKVHeads the number of query heads sharing each KV head, and KVDim = NKVHeads*ValueDim the per-position V cache width. The scale factors (Embedding/Residual/Logit/Attention) default to 1 (or 0 meaning "use 1/sqrt(HeadDim)" for AttentionScale) and are only non-trivial for architectures whose GGUFs carry them.

func ConfigFromGGUF 

func ConfigFromGGUF(gguf *GGUFFile) Config

type DTypeStat 

type DTypeStat struct {
	Type    GGMLType
	Tensors int
	Bytes   int64
	Params  int64
}

DTypeStat summarizes one tensor dtype's share of a model.

type DecodeBuffer 

type DecodeBuffer struct {
	X                       []float32
	XN                      []float32
	XN2                     []float32
	Q                       []float32
	K                       []float32
	V                       []float32
	QKV                     []float32
	AttnOut                 []float32
	Proj                    []float32
	Gate                    []float32
	Up                      []float32
	GateUp                  []float32
	Hidden                  []float32
	MOE                     []float32
	RouterLogits            []float32
	TopExperts              []ExpertScore
	ExpertProbs             []float32
	SamplerCandidates       []TokenProb
	Q4KXSums                []float32
	RopeInvFreq             []float32
	RopeSin                 []float32
	RopeCos                 []float32
	RopeMscale              float32
	RopeGptOssInvFreq       []float32
	RopeGptOssConcentration float32
	MambaIn                 []float32
	MambaConv               []float32
	MambaZ                  []float32
	MambaX                  []float32
	MambaB                  []float32
	MambaC                  []float32
	MambaDT                 []float32
	MambaY                  []float32
	MambaKernel             []float32
	ExpertRow               []float32
	ExpertHidden            []float32
	// contains filtered or unexported fields
}

DecodeBuffer is reusable request scratch for single-token decode and batched prefill: activation vectors (X residual stream, XN/XN2 normed views, Q/K/V/AttnOut/Proj attention buffers, Gate/Up/Hidden FFN buffers), the sampler's candidate scratch, and precomputed RoPE tables (per-pair inverse frequencies plus per-position sin/cos filled in prepareRopeScratch). One DecodeBuffer serves successive requests, so decode allocates nothing per token and prefill reuses its activation slabs. Not safe for concurrent use; Runner.genLock serializes requests.

func NewDecodeBuffer 

func NewDecodeBuffer(config Config, maxHeadDim, maxNKVHeads, maxValueDim int) *DecodeBuffer

type EmbeddingResult 

type EmbeddingResult struct {
	Embedding  []float32
	TokenCount int
}

type EmbeddingsRequest 

type EmbeddingsRequest struct {
	Model string `json:"model"`
	Input any    `json:"input"`
}

func (EmbeddingsRequest) Inputs 

func (e EmbeddingsRequest) Inputs() []string

type ExpertScore 

type ExpertScore struct {
	Index int
	Score float32
}

type ExpertWeight added in v0.3.0 

type ExpertWeight struct {
	Weight  Weight
	Input   int
	Output  int
	Experts int
}

ExpertWeight retains the third GGUF tensor dimension. Weight itself stores two-dimensional matrices, while the expert tensors are laid out as [input, output, expert].

type GGMLType 

type GGMLType uint32

GGMLType identifies a tensor's on-disk element encoding, matching the ggml type ids used by llama.cpp. F32/F16 are plain element arrays; the quantized types pack fixed-size blocks of elements together with their scale factors — see BlockBytes/DataSize for the exact per-block sizes the matvec kernels in simd.go rely on. 

const (
	GGMLTypeF32     GGMLType = 0
	GGMLTypeF16     GGMLType = 1
	GGMLTypeQ4_0    GGMLType = 2
	GGMLTypeQ4_1    GGMLType = 3
	GGMLTypeQ5_0    GGMLType = 6
	GGMLTypeQ5_1    GGMLType = 7
	GGMLTypeQ8_0    GGMLType = 8
	GGMLTypeQ8_1    GGMLType = 9
	GGMLTypeQ2_K    GGMLType = 10
	GGMLTypeQ3_K    GGMLType = 11
	GGMLTypeQ4_K    GGMLType = 12
	GGMLTypeQ5_K    GGMLType = 13
	GGMLTypeQ6_K    GGMLType = 14
	GGMLTypeQ8_K    GGMLType = 15
	GGMLTypeBF16    GGMLType = 30
	GGMLTypeMXFP4   GGMLType = 39
	GGMLTypeUnknown GGMLType = 255
)

func (GGMLType) BlockBytes 

func (t GGMLType) BlockBytes() (int, bool)

BlockBytes returns the byte size of one block for the simple (32-element) quant formats, e.g. Q8_0 = 2-byte f16 scale + 32 int8 quants = 34 bytes, Q4_0 = 2-byte f16 scale + 16 packed-nibble bytes = 18 bytes. K-quants and MXFP4 are not representable here (ok=false); use DataSize instead.

func (GGMLType) BlockSize 

func (t GGMLType) BlockSize() int

BlockSize returns how many elements one quantization block encodes (1 for the plain float types). Note the K-quants (Q4_K/Q5_K/Q6_K) actually use 256-element superblocks; this legacy accessor is only used with the 32-wide simple quants, and DataSize handles the K-quant sizes explicitly.

func (GGMLType) DataSize 

func (t GGMLType) DataSize(n int) (int, bool)

DataSize returns the exact byte size of n elements of this type. The per-block layouts, which the dot kernels in simd.go index by hand: 

Q4_0:  18 B / 32 elems  = f16 scale + 16 nibble-packed bytes
Q4_1:  20 B / 32 elems  = f16 scale + f16 min + 16 nibble bytes
Q5_0:  22 B / 32 elems  = f16 scale + 4 B 5th-bit plane + 16 nibbles
Q5_1:  24 B / 32 elems  = f16 scale + f16 min + 4 B 5th bits + 16 nibbles
Q8_0:  34 B / 32 elems  = f16 scale + 32 int8
Q8_1:  36 B / 32 elems  = f16 scale + f16 quant-sum + 32 int8
Q2_K:  84 B / 256 elems = 16 B packed 4-bit scale/min pairs +
       64 B 2-bit quants + f16 d + f16 dmin
Q3_K: 110 B / 256 elems = 32 B high-bit plane + 64 B 2-bit lows +
       12 B packed 6-bit scales + f16 d
Q4_K: 144 B / 256 elems = f16 d + f16 dmin + 12 B packed 6-bit
       scales/mins (8 sub-blocks) + 128 nibble-packed bytes
Q5_K: 176 B / 256 elems = Q4_K layout + 32 B of 5th-bit planes
Q6_K: 210 B / 256 elems = 128 B low nibbles + 64 B 2-bit highs +
       16 int8 sub-block scales + f16 d
MXFP4: 17 B / 32 elems  = 16 nibble-packed FP4 values + 1 shared
       power-of-two exponent byte

ok is false for types this runtime cannot size (callers then fall back to offset-difference inference; see inferTensorSizes).

func (GGMLType) String 

func (t GGMLType) String() string

type GGUFFile 

type GGUFFile struct {
	Metadata   map[string]MetaValue
	Tensors    []TensorInfo
	DataOffset int
	Version    uint32
}

GGUFFile is a parsed GGUF header: all metadata, all tensor descriptors, and the alignment-adjusted offset where tensor data begins within the original byte slice. It does not own or copy any tensor data.

func ParseGGUF 

func ParseGGUF(data []byte) (*GGUFFile, error)

ParseGGUF parses a GGUF header, logging a one-line summary to stderr. ParseGGUFQuiet is the same without the log line (used by model discovery, which parses every file in a directory).

func ParseGGUFQuiet 

func ParseGGUFQuiet(data []byte) (*GGUFFile, error)

func (*GGUFFile) GetF32 

func (g *GGUFFile) GetF32(key string, def float32) float32

func (*GGUFFile) GetString 

func (g *GGUFFile) GetString(key string) (string, bool)

func (*GGUFFile) GetU32 

func (g *GGUFFile) GetU32(key string, def uint32) uint32

GetU32/GetF32/GetString are convenience metadata lookups; the numeric variants return def when the key is absent or has an incompatible kind.

func (*GGUFFile) GetU32Array added in v0.3.0 

func (g *GGUFFile) GetU32Array(key string) ([]uint32, bool)

GetU32Array returns an integer metadata array, if present.

type Gemma4LayerWeights 

type Gemma4LayerWeights struct {
	AttnNorm   []float32
	AttnQ      Weight
	AttnK      Weight
	AttnV      Weight
	AttnOutput Weight
	FFNNorm    []float32
	FFNDown    Weight
	FFNUp      Weight
	FFNGate    Weight
	HeadDim    int
	NKVHeads   int
	ValueDim   int
	HasAttnV   bool
}

type Gemma4Weights 

type Gemma4Weights struct {
	TokenEmbd  Weight
	OutputNorm []float32
	Output     Weight
	Layers     []Gemma4LayerWeights
	Standard   ModelWeights
}

type GenOption 

type GenOption func(*GenerationOptions)

GenOption configures a single generation request on top of DefaultGenerationOptions.

func WithGenerationOptions 

func WithGenerationOptions(base GenerationOptions) GenOption

WithGenerationOptions replaces the entire options struct (escape hatch for callers that already hold a GenerationOptions); later GenOptions still apply on top.

func WithMaxTokens 

func WithMaxTokens(n int) GenOption

WithMaxTokens caps the number of generated tokens (default 256).

func WithMinP 

func WithMinP(p float32) GenOption

WithMinP drops candidates below fraction p of the best token's probability.

func WithRepeatPenalty 

func WithRepeatPenalty(p float32) GenOption

WithRepeatPenalty penalizes recently generated tokens; 1 disables it.

func WithSeed 

func WithSeed(seed uint64) GenOption

WithSeed makes sampling deterministic for a fixed seed (0 = time-based).

func WithStop 

func WithStop(sequences ...string) GenOption

WithStop adds stop sequences that end generation when they appear.

func WithSystemPrompt 

func WithSystemPrompt(s string) GenOption

WithSystemPrompt replaces the default system prompt; pass "" for none.

func WithTemperature 

func WithTemperature(t float32) GenOption

WithTemperature sets the sampling temperature; 0 selects greedy decoding.

func WithToolChoice 

func WithToolChoice(choice string) GenOption

WithToolChoice sets the tool_choice behavior ("none" suppresses tools).

func WithTools 

func WithTools(tools ...ToolDefinition) GenOption

WithTools offers OpenAI-shaped tool definitions to the model; calls the model makes come back in Result.ToolCalls with FinishReason "tool_calls".

func WithTopK 

func WithTopK(k int) GenOption

WithTopK restricts sampling to the k most likely tokens; 0 disables it.

func WithTopP 

func WithTopP(p float32) GenOption

WithTopP sets nucleus sampling mass in (0, 1]; 1 disables it.

type GenerateRequest 

type GenerateRequest struct {
	Prompt        string           `json:"prompt"`
	Messages      []APIMessage     `json:"messages"`
	MaxTokens     *int             `json:"max_tokens"`
	Temp          *float32         `json:"temp"`
	Temperature   *float32         `json:"temperature"`
	TopP          *float32         `json:"top_p"`
	TopK          *int             `json:"top_k"`
	MinP          *float32         `json:"min_p"`
	RepeatPenalty *float32         `json:"repeat_penalty"`
	Seed          *uint64          `json:"seed"`
	SystemPrompt  *string          `json:"system_prompt"`
	Stop          any              `json:"stop"`
	Tools         []ToolDefinition `json:"tools"`
	ToolChoice    any              `json:"tool_choice"`
}

func (GenerateRequest) ToMessagesAndOptions 

func (g GenerateRequest) ToMessagesAndOptions(def GenerationOptions) ([]ChatMessage, GenerationOptions)

type GenerationOptions 

type GenerationOptions struct {
	MaxTokens     int
	Sampler       SamplerConfig
	Seed          uint64
	SystemPrompt  string
	StopSequences []string
	// Tools lists the functions the model may call. When non-empty, it is
	// rendered into the prompt using the active chat template's tool-calling
	// convention (native for Mistral, a generic <tool_call> JSON convention
	// otherwise).
	Tools []ToolDefinition
	// ToolChoice controls which of Tools are offered. "none" suppresses tool
	// rendering entirely; a value of the form "function:<name>" (as produced
	// by an OpenAI-style tool_choice object naming one function) narrows
	// offering to just that tool; any other value (including the default
	// "auto") offers all of Tools.
	ToolChoice string
	// contains filtered or unexported fields
}

func DefaultGenerationOptions 

func DefaultGenerationOptions() GenerationOptions

func (GenerationOptions) Validate 

func (o GenerationOptions) Validate() error

type GenerationResult 

type GenerationResult struct {
	Text string
	// ReasoningText holds any chain-of-thought the model emitted separately
	// from its answer (e.g. DeepSeek-R1/QwQ <think> blocks, or gpt-oss's
	// analysis channel), stripped out of Text.
	ReasoningText string
	// ToolCalls holds structured function calls extracted from the model's
	// raw output, stripped out of Text. Empty unless GenerationOptions.Tools
	// was non-empty for this request.
	ToolCalls []ToolCall
	// FinishReason is "stop" (natural end or stop-sequence match), "length"
	// (max_tokens or context exhausted), or "tool_calls" (ToolCalls is
	// non-empty).
	FinishReason string
	Stats        GenerationStats
}

func RunAgenticChat 

func RunAgenticChat(r *Runner, messages []ChatMessage, options GenerationOptions, skills []Skill, onToken func(string) bool) (GenerationResult, error)

RunAgenticChat runs a chat generation, automatically resolving any load_skill call the model makes: it looks up the named skill's full body and feeds it back as a tool result, then lets the model continue, before ever returning to the caller. A tool call for anything else — i.e. every tool the CALLER supplied, as opposed to the server's own load_skill — is left untouched in the result for the caller to execute and continue via a follow-up request with a ToolResultMessage, exactly like ordinary (non-agentic) tool use. A turn that mixes a skill call with a caller tool call is treated as needing the caller (not resolved internally), so the caller never has calls silently dropped out from under it.

A turn's raw output isn't known to be "the final answer" versus a tool call until generation for that turn completes, so whenever ANY tool activity is possible this turn (skills configured, or the caller supplied tools), onToken only fires once, with the complete, already-classified content of the winning turn — never with raw, mid-formation tool-call syntax. This also holds for plain (non-skill) tool use, since a client streaming "[TOOL_CALLS]get_weather[ARGS]..." as if it were visible answer text would be exactly the kind of leak this is meant to prevent. When there is no tool activity at all for this request, this is a zero-overhead passthrough to GenerateChatStreamUntil with full incremental streaming — the common case is unaffected.

type GenerationStats 

type GenerationStats struct {
	PromptTokens    int
	GeneratedTokens int
	TTFT            time.Duration
	PrefillTime     time.Duration
	DecodeTime      time.Duration
	TotalTime       time.Duration
}

type GptOssWeights 

type GptOssWeights struct {
	Standard ModelWeights
}

type HandlerOptions 

type HandlerOptions struct {
	// Defaults are the generation settings requests inherit unless they
	// override individual fields.
	Defaults GenerationOptions
	// MaxConcurrentRequests bounds in-flight generation requests (default 8).
	// Requests beyond the bound queue rather than failing.
	MaxConcurrentRequests int
	// ChatUI serves the embedded browser chat at /chat (plus its assets).
	ChatUI bool
	// ModelDir enables GET /models discovery and POST /models/load hot-swap
	// within that directory.
	ModelDir string
	// ModelPath is the initially loaded model's path (reported by /models).
	ModelPath string
	// SkillsDir, if set, is scanned once at handler construction for SKILL.md
	// files (see skills.go). Every chat/generate endpoint offers a load_skill
	// tool and resolves it server-side via RunAgenticChat.
	SkillsDir string
	// LogWriter receives handler diagnostics (skill load notes). Defaults to
	// io.Discard.
	LogWriter io.Writer
}

HandlerOptions configures the mountable HTTP API handler.

type KVCache 

type KVCache struct {
	K          [][]float32
	V          [][]float32
	PerPosKDim int
	PerPosVDim int
	MaxLen     int
	// F16 selects half-precision row storage: K16/V16 replace K/V entirely
	// and attention converts rows in-register (see kv_f16.go). Halves the
	// cache's memory footprint and the bytes attention streams per token.
	F16 bool
	// Nemotron is present only for hybrid Mamba-2 architectures. K/V remains
	// in this cache; the companion stores the recurrent convolution/SSM state.
	Nemotron *NemotronHCache
	K16      [][]uint16
	V16      [][]uint16
}

KVCache stores the attention keys and values of every processed position, one flat slice per layer laid out position-major: position p's keys occupy K[layer][p*PerPosKDim : (p+1)*PerPosKDim] (all KV heads concatenated), and likewise for V. Sized to prompt+max_tokens (capped at the model context length) and reused by Runner when capacity permits; there is no ring/eviction and generation stops at its request-local cache length.

func NewKVCache 

func NewKVCache(layers, kDim, vDim, maxLen int) *KVCache

NewKVCache allocates an f32 cache for `layers` layers of maxLen positions with the given per-position K and V widths (see KVCache).

func NewKVCacheF16 added in v0.3.0 

func NewKVCacheF16(layers, kDim, vDim, maxLen int) *KVCache

NewKVCacheF16 is NewKVCache with half-precision row storage.

type KernelBenchRow 

type KernelBenchRow struct {
	Name    string  `json:"name"`
	DType   string  `json:"dtype"`
	Rows    int     `json:"rows"`
	Cols    int     `json:"cols"`
	Runs    int     `json:"runs"`
	AvgMS   float64 `json:"avg_ms"`
	TotalMS float64 `json:"total_ms"`
}

KernelBenchRow is one kernel's timing in the --kernel-bench report: a named matvec (attn_q, ffn_down, output, ...) from a real layer of the loaded model, so the numbers reflect the model's true shapes and quant types rather than synthetic sizes.

type LayerWeights 

type LayerWeights struct {
	AttnNorm  []float32
	WQ        Weight
	BQ        []float32
	WK        Weight
	BK        []float32
	WV        Weight
	BV        []float32
	WQKV      Weight
	HasQKV    bool
	WO        Weight
	FFNNorm   []float32
	W1        Weight
	W2        Weight
	W3        Weight
	WGateUp   Weight
	HasGateUp bool
	// Optional Gemma-family norms, nil when the tensors are absent:
	// AttnQNorm/AttnKNorm are per-head RMS norms of length HeadDim applied
	// after the Q/K projections (before RoPE); PostAttnNorm/PostFFNNorm are
	// full-width RMS norms applied to the attention/FFN outputs before their
	// residual adds.
	AttnQNorm    []float32
	AttnKNorm    []float32
	PostAttnNorm []float32
	PostFFNNorm  []float32
}

LayerWeights holds one transformer block. Attention is either split (WQ/WK/WV, with optional biases BQ/BK/BV) or fused into a single WQKV (HasQKV); the SwiGLU FFN is likewise either split (W1 = gate, W3 = up, W2 = down — llama.cpp naming) or fused gate+up in WGateUp (HasGateUp).

type LoadInfo 

type LoadInfo struct {
	FileSizeBytes int
	LoadTime      time.Duration
}

type LoadOptions 

type LoadOptions struct {
	PrepareQuantized bool
	UseMetal         bool
	LogWriter        io.Writer
}

type MetaValue 

type MetaValue struct {
	Kind  string
	Value any
}

MetaValue is one decoded GGUF metadata value. Kind is the GGUF wire-type name ("u32", "str", "array", ...) and Value holds the corresponding Go value (arrays are []MetaValue). The As* accessors below do tolerant conversions — GGUF producers are inconsistent about integer widths, so e.g. AsU32 accepts any integer kind rather than only u32.

func (MetaValue) AsBool 

func (v MetaValue) AsBool() (bool, bool)

func (MetaValue) AsBoolArray 

func (v MetaValue) AsBoolArray() ([]bool, bool)

AsBoolArray decodes an array of bools (e.g. Gemma 4's per-layer attention.sliding_window_pattern).

func (MetaValue) AsF32 

func (v MetaValue) AsF32() (float32, bool)

func (MetaValue) AsF32Array 

func (v MetaValue) AsF32Array() ([]float32, bool)

func (MetaValue) AsString 

func (v MetaValue) AsString() (string, bool)

func (MetaValue) AsStringArray 

func (v MetaValue) AsStringArray() ([]string, bool)

func (MetaValue) AsU32 

func (v MetaValue) AsU32() (uint32, bool)

func (MetaValue) AsU32Array added in v0.3.0 

func (v MetaValue) AsU32Array() ([]uint32, bool)

AsU32Array decodes an integer metadata array. GGUF writers use different integer widths for per-layer architecture fields, so accept every value that AsU32 accepts.

type MetalWeight 

type MetalWeight struct{}

type MmapFile 

type MmapFile struct {
	// contains filtered or unexported fields
}

MmapFile exposes a model file as one immutable byte slice, memory-mapped where the platform allows so multi-gigabyte weights are paged in on demand rather than copied. Quantized Weights borrow sub-slices of it directly (loadWeight's borrow mode), so it must stay open for the Runner's lifetime; Runner.Close unmaps it.

func OpenMmap 

func OpenMmap(path string) (*MmapFile, error)

OpenMmap uses the platform mmap syscall without CGO. If mmap is unavailable for a specific file, it falls back to os.ReadFile while preserving the same immutable byte-slice API.

func (*MmapFile) Bytes 

func (m *MmapFile) Bytes() []byte

func (*MmapFile) Close 

func (m *MmapFile) Close() error

func (*MmapFile) Len 

func (m *MmapFile) Len() int

type Model 

type Model struct {
	// contains filtered or unexported fields
}

Model is a loaded LLM ready for generation, chat, embeddings, and tokenization. It wraps a Runner; the same concurrency contract applies (safe to share across goroutines, requests execute one at a time).

func Open 

func Open(ctx context.Context, path string, opts ...Option) (*Model, error)

Open memory-maps and loads a GGUF model file. The returned Model borrows quantized weights from the mapping zero-copy; Close releases it. ctx only gates the start of the load (weight loading itself is not interruptible).

func OpenBytes 

func OpenBytes(ctx context.Context, data []byte, opts ...Option) (*Model, error)

OpenBytes loads a model from an in-memory GGUF image, copying quantized tensors into owned memory (data may be released afterwards).

func (*Model) Chat 

func (m *Model) Chat(ctx context.Context, messages []ChatMessage, opts ...GenOption) (GenerationResult, error)

Chat runs a multi-turn conversation through the model's chat template.

Example

ExampleModel_Chat demonstrates multi-turn chat with tool calling: the model requests a tool, the application executes it and replays the result. 

package main

import (
	"context"
	"fmt"
	"log"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	ctx := context.Background()
	model, err := gopherllm.Open(ctx, "model.gguf")
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	weatherTool := gopherllm.ToolDefinition{
		Type: "function",
		Function: gopherllm.ToolFunctionDef{
			Name:        "get_weather",
			Description: "Get the current weather for a city",
			Parameters:  []byte(`{"type":"object","properties":{"city":{"type":"string"}},"required":["city"]}`),
		},
	}
	messages := []gopherllm.ChatMessage{gopherllm.UserMessage("What's the weather in Berlin?")}
	res, err := model.Chat(ctx, messages, gopherllm.WithTools(weatherTool))
	if err != nil {
		log.Fatal(err)
	}
	if res.FinishReason == "tool_calls" {
		call := res.ToolCalls[0]
		// ... execute the tool, then continue the conversation:
		messages = append(messages,
			gopherllm.ChatMessage{Role: gopherllm.ChatRoleAssistant, ToolCalls: res.ToolCalls},
			gopherllm.ToolResultMessage(call.ID, call.Function.Name, `{"temperature_c": 18}`),
		)
		res, err = model.Chat(ctx, messages, gopherllm.WithTools(weatherTool))
		if err != nil {
			log.Fatal(err)
		}
	}
	fmt.Println(res.Text)
}

func (*Model) Close 

func (m *Model) Close() error

Close releases the model's memory-mapped weight file. No Model method may be called afterwards.

func (*Model) Config 

func (m *Model) Config() Config

func (*Model) Detokenize 

func (m *Model) Detokenize(ids []uint32) string

Detokenize decodes token ids back to text.

func (*Model) Embed 

func (m *Model) Embed(ctx context.Context, text string) (EmbeddingResult, error)

Embed returns the mean-pooled, L2-normalized final-hidden-state embedding of text. Cancellation granularity is the whole call (embedding runs a prompt-length forward pass).

Example

ExampleModel_Embed computes a text embedding for semantic search. 

package main

import (
	"context"
	"fmt"
	"log"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	ctx := context.Background()
	model, err := gopherllm.Open(ctx, "model.gguf")
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	emb, err := model.Embed(ctx, "semantic search query")
	if err != nil {
		log.Fatal(err)
	}
	fmt.Println(len(emb.Embedding), "dimensions from", emb.TokenCount, "tokens")
}

func (*Model) GGUF 

func (m *Model) GGUF() *GGUFFile

func (*Model) Generate 

func (m *Model) Generate(ctx context.Context, prompt string, opts ...GenOption) (GenerationResult, error)

Generate runs a single-prompt completion. Cancellation via ctx takes effect between prefill chunks and between decoded tokens; the returned error is then the context's error.

func (*Model) HTTPHandler 

func (m *Model) HTTPHandler(opts HandlerOptions) http.Handler

HTTPHandler returns the model's OpenAI/Ollama-compatible HTTP API as a mountable handler (see NewHandler). The handler shares this Model's underlying Runner; generation requests serialize with direct Model calls.

func (*Model) Info 

func (m *Model) Info() LoadInfo

Info returns file size and load timing. Config, Tokenizer, GGUF, and Name expose the model's static properties.

func (*Model) Name 

func (m *Model) Name() string

Name returns the model's self-declared name (general.name), or the empty string.

func (*Model) NearestTokens 

func (m *Model) NearestTokens(token string, k int) ([]TokenMatch, error)

NearestTokens on the Model API. token may be a numeric id or a literal token text (resolved via exact vocabulary search).

func (*Model) Runner 

func (m *Model) Runner() *Runner

Runner exposes the underlying low-level Runner for callers that need capabilities the Model API doesn't surface (custom forward passes, the agentic skill loop, the HTTP handler).

func (*Model) Stream 

func (m *Model) Stream(ctx context.Context, messages []ChatMessage, onDelta func(delta string) error, opts ...GenOption) (GenerationResult, error)

Stream is Chat with incremental delivery: onDelta receives each new chunk of valid-UTF-8 output text as it is generated. Returning a non-nil error from onDelta stops generation; that error is returned (wrapped) alongside the partial result. When tools are offered, output is delivered in a single final call instead of incrementally so raw tool-call syntax never leaks (see RunAgenticChat for why).

Example

ExampleModel_Stream shows incremental token delivery with cancellation: the context deadline stops generation cleanly mid-stream. 

package main

import (
	"context"
	"fmt"
	"log"
	"os"

	gopherllm "github.com/SimonWaldherr/GopherLLM"
)

func main() {
	ctx := context.Background()
	model, err := gopherllm.Open(ctx, "model.gguf", gopherllm.WithLogWriter(os.Stderr))
	if err != nil {
		log.Fatal(err)
	}
	defer model.Close()

	messages := []gopherllm.ChatMessage{
		gopherllm.UserMessage("Write a haiku about Go."),
	}
	_, err = model.Stream(ctx, messages, func(delta string) error {
		fmt.Print(delta)
		return nil // return an error to stop generation
	}, gopherllm.WithMaxTokens(64), gopherllm.WithSeed(42))
	if err != nil {
		log.Fatal(err)
	}
}

func (*Model) Tokenize 

func (m *Model) Tokenize(text string) []uint32

Tokenize encodes text with the model's tokenizer (including BOS when the model declares it). Detokenize is its inverse over generated ids.

func (*Model) Tokenizer 

func (m *Model) Tokenizer() *Tokenizer

type ModelEntry 

type ModelEntry struct {
	ID           string
	Repository   string
	FileName     string
	Path         string
	SizeBytes    int64
	Architecture string
	ModelName    string
	IsProjector  bool
	IsSupported  bool
}

ModelEntry describes one GGUF file found under the model directory. ID is the root-relative path without the .gguf suffix (unique within a scan and what --list-models prints); IsProjector marks multimodal companion files (mmproj-*/clip) that must not be offered for text generation.

func DiscoverModels 

func DiscoverModels(root string, logw io.Writer) ([]ModelEntry, error)

DiscoverModels recursively finds every .gguf under root and parses each file's header (mmap'd, weights untouched) to fill in architecture, name, and support status. Unparseable files are skipped with a stderr note rather than failing the whole scan. Results are sorted by ID.

func PromptModelSelection 

func PromptModelSelection(dir string, entries []ModelEntry, in io.Reader, out io.Writer) (ModelEntry, error)

PromptModelSelection runs the interactive terminal picker: a numbered menu that accepts an index, a filter substring (re-listing the matches), or q/quit/exit to abort.

func SelectModel 

func SelectModel(entries []ModelEntry, selector string) (ModelEntry, error)

SelectModel matches selector against the usable (supported, non-projector) entries: exact matches on id/repo/filename/name/path beat substring matches, a unique match wins, and ambiguity or matching only an unsupported/projector file produces a descriptive error listing the choices.

func (ModelEntry) Status 

func (m ModelEntry) Status() string

type ModelWeights 

type ModelWeights struct {
	TokenEmbd  Weight
	OutputNorm []float32
	Output     Weight
	Layers     []LayerWeights
}

type NemotronAttentionWeights added in v0.3.0 

type NemotronAttentionWeights struct {
	Q Weight
	K Weight
	V Weight
	O Weight
}

type NemotronHCache added in v0.3.0 

type NemotronHCache struct {
	Conv      []float32
	State     []float32
	Layers    int
	Channels  int
	ConvLen   int
	Heads     int
	HeadDim   int
	StateSize int
}

NemotronHCache stores the recurrent Mamba-2 state. Attention K/V continues to use KVCache so the existing f16 cache and online-attention kernels apply unchanged. Conv is [layer, channel, d_conv-1]; State is [layer, head, head_dim, d_state].

type NemotronHLayerWeights added in v0.3.0 

type NemotronHLayerWeights struct {
	Norm      []float32
	Kind      nemotronLayerKind
	Attention NemotronAttentionWeights
	Mamba     NemotronMambaWeights
	MoE       NemotronMoEWeights
}

type NemotronHWeights added in v0.3.0 

type NemotronHWeights struct {
	TokenEmbd  Weight
	OutputNorm []float32
	Output     Weight
	Layers     []NemotronHLayerWeights
}

type NemotronMambaWeights added in v0.3.0 

type NemotronMambaWeights struct {
	In       Weight
	Conv     Weight
	ConvBias []float32
	DtBias   []float32
	A        []float32
	D        []float32
	Norm     []float32
	Out      Weight
}

type NemotronMoEWeights added in v0.3.0 

type NemotronMoEWeights struct {
	Router     Weight
	RouterBias []float32
	Up         ExpertWeight
	Down       ExpertWeight
	LatentIn   *Weight
	LatentOut  *Weight
	SharedUp   *Weight
	SharedDown *Weight
}

type OllamaChatRequest 

type OllamaChatRequest struct {
	Model    string           `json:"model"`
	Messages []OllamaMessage  `json:"messages"`
	Stream   *bool            `json:"stream"`
	Options  OllamaOptions    `json:"options"`
	Tools    []ToolDefinition `json:"tools"`
}

func (OllamaChatRequest) ChatMessages 

func (o OllamaChatRequest) ChatMessages() []ChatMessage

func (OllamaChatRequest) GenerationOptions 

func (o OllamaChatRequest) GenerationOptions(def GenerationOptions) GenerationOptions

type OllamaEmbedRequest added in v0.3.0 

type OllamaEmbedRequest struct {
	Model string `json:"model"`
	Input any    `json:"input"`
}

OllamaEmbedRequest is the request body for /api/embed, the batched successor to the deprecated single-prompt /api/embeddings.

func (OllamaEmbedRequest) Inputs added in v0.3.0 

func (o OllamaEmbedRequest) Inputs() []string

type OllamaEmbeddingRequest 

type OllamaEmbeddingRequest struct {
	Model  string `json:"model"`
	Prompt string `json:"prompt"`
	Input  any    `json:"input"`
}

func (OllamaEmbeddingRequest) Inputs 

func (o OllamaEmbeddingRequest) Inputs() []string

type OllamaGenerateRequest 

type OllamaGenerateRequest struct {
	Model   string        `json:"model"`
	Prompt  string        `json:"prompt"`
	System  string        `json:"system"`
	Stream  *bool         `json:"stream"`
	Options OllamaOptions `json:"options"`
	Stop    any           `json:"stop"`
}

func (OllamaGenerateRequest) GenerationOptions 

func (o OllamaGenerateRequest) GenerationOptions(def GenerationOptions) GenerationOptions

type OllamaMessage 

type OllamaMessage struct {
	Role    string `json:"role"`
	Content string `json:"content"`
}

type OllamaOptions 

type OllamaOptions struct {
	// NumCtx is accepted for wire compatibility (real Ollama clients set it
	// routinely) but not actionable here: a Runner's KV cache is sized once
	// from the loaded GGUF's context_length at model-load time, and this
	// server has no per-request context-window resize.
	NumCtx        *int     `json:"num_ctx"`
	NumPredict    *int     `json:"num_predict"`
	Temperature   *float32 `json:"temperature"`
	TopP          *float32 `json:"top_p"`
	TopK          *int     `json:"top_k"`
	MinP          *float32 `json:"min_p"`
	RepeatPenalty *float32 `json:"repeat_penalty"`
	Seed          *uint64  `json:"seed"`
	Stop          any      `json:"stop"`
}

type OpenAIChatRequest 

type OpenAIChatRequest struct {
	Model               string            `json:"model"`
	Messages            []APIMessage      `json:"messages"`
	Stream              bool              `json:"stream"`
	StreamOptions       *OpenAIStreamOpts `json:"stream_options"`
	MaxTokens           *int              `json:"max_tokens"`
	MaxCompletionTokens *int              `json:"max_completion_tokens"`
	Temperature         *float32          `json:"temperature"`
	TopP                *float32          `json:"top_p"`
	TopK                *int              `json:"top_k"`
	MinP                *float32          `json:"min_p"`
	RepeatPenalty       *float32          `json:"repeat_penalty"`
	Seed                *uint64           `json:"seed"`
	SystemPrompt        *string           `json:"system_prompt"`
	Stop                any               `json:"stop"`
	Tools               []ToolDefinition  `json:"tools"`
	ToolChoice          any               `json:"tool_choice"`
}

func (OpenAIChatRequest) ChatMessages 

func (o OpenAIChatRequest) ChatMessages() []ChatMessage

func (OpenAIChatRequest) Options 

func (o OpenAIChatRequest) Options(def GenerationOptions) GenerationOptions

type OpenAICompletionRequest 

type OpenAICompletionRequest struct {
	Model               string   `json:"model"`
	Prompt              any      `json:"prompt"`
	MaxTokens           *int     `json:"max_tokens"`
	MaxCompletionTokens *int     `json:"max_completion_tokens"`
	Temperature         *float32 `json:"temperature"`
	TopP                *float32 `json:"top_p"`
	TopK                *int     `json:"top_k"`
	MinP                *float32 `json:"min_p"`
	RepeatPenalty       *float32 `json:"repeat_penalty"`
	Seed                *uint64  `json:"seed"`
	SystemPrompt        *string  `json:"system_prompt"`
	Stop                any      `json:"stop"`
}

func (OpenAICompletionRequest) Options 

func (o OpenAICompletionRequest) Options(def GenerationOptions) GenerationOptions

func (OpenAICompletionRequest) PromptString 

func (o OpenAICompletionRequest) PromptString() string

type OpenAIStreamOpts added in v0.3.0 

type OpenAIStreamOpts struct {
	IncludeUsage bool `json:"include_usage"`
}

OpenAIStreamOpts is the OpenAI "stream_options" object; IncludeUsage gates whether the final SSE chunk carries a "usage" field (off by default, per spec — unlike a non-streaming response, which always includes usage).

type Option 

type Option func(*loadSettings)

Option configures model loading.

func WithLogWriter 

func WithLogWriter(w io.Writer) Option

WithLogWriter directs load-progress and warning diagnostics (GGUF summary, per-layer load progress, experimental-architecture warnings) to w. The default is io.Discard: as a library, GopherLLM produces no output unless asked to.

func WithMetal 

func WithMetal(enabled bool) Option

WithMetal enables experimental Metal-backed kernels when the current build supports them.

func WithPrepareQuantized 

func WithPrepareQuantized(enabled bool) Option

WithPrepareQuantized precomputes supported quantized scale data at load time for the prepared quantized kernels.

func WithThreads 

func WithThreads(n int) Option

WithThreads sets the compute worker count. NOTE: the worker pool is shared process-wide (all Models in the process use one pool), so this is a global knob despite being a load option; the last value set wins.

type Pair 

type Pair struct {
	Left  string
	Right string
}

type PreparedQuantizedWeight 

type PreparedQuantizedWeight struct {
	Type   GGMLType
	Rows   int
	Cols   int
	Blocks int
	Scales []float32
	Mins   []float32
}

func PrepareQuantizedWeight 

func PrepareQuantizedWeight(data []byte, typ GGMLType, rows, cols int) *PreparedQuantizedWeight

type Q4KMatrix 

type Q4KMatrix struct {
	Data []byte
	Rows int
	Cols int
}

Q4KMatrix is a borrowed view of a Q4_K weight tensor: Rows x Cols elements packed as (Cols/256) 144-byte superblocks per row (see GGMLType.DataSize).

type Rng 

type Rng struct {
	// contains filtered or unexported fields
}

Rng is a xorshift64 generator. Generation is fully deterministic for a fixed seed (the CLI's --seed): the same seed replays the same tokens.

func NewRng 

func NewRng(seed uint64) *Rng

NewRng seeds an Rng; a zero seed is remapped to a fixed non-zero constant because xorshift has an all-zero fixed point.

func (*Rng) NextF32 

func (r *Rng) NextF32() float32

NextF32 returns a uniform float in [0, 1) with 24 bits of precision.

type Runner 

type Runner struct {
	// contains filtered or unexported fields
}

Runner is a fully loaded model ready to generate: parsed GGUF header, tokenizer, config, and weights (one of the three kind-specific sets). Generations and embeddings are serialized by genLock — a Runner is safe to share across goroutines (the HTTP server does), but runs one request at a time. Close releases the memory-mapped weight file; quantized weights borrow from it, so no method may be called after Close.

func RunnerFromGGUFBytes 

func RunnerFromGGUFBytes(data []byte) (*Runner, error)

RunnerFromGGUFBytes loads a model from an in-memory GGUF, copying quantized tensors into owned memory. It is silent; use Open with WithLogWriter for load-progress diagnostics.

func RunnerFromGGUFBytesWithOptions 

func RunnerFromGGUFBytesWithOptions(data []byte, options LoadOptions) (*Runner, error)

func (*Runner) Architecture 

func (r *Runner) Architecture() string

func (*Runner) Close 

func (r *Runner) Close() error

func (*Runner) Config 

func (r *Runner) Config() Config

func (*Runner) Embed 

func (r *Runner) Embed(text string) (EmbeddingResult, error)

Embed produces a text embedding by mean-pooling the final-layer hidden states over all input tokens and L2-normalizing the result (so dot product equals cosine similarity). Dimension is the model's hidden size — note this uses the generation model's hidden states, not a dedicated embedding head.

func (*Runner) GGUF 

func (r *Runner) GGUF() *GGUFFile

func (*Runner) Generate 

func (r *Runner) Generate(prompt string, options GenerationOptions) (GenerationResult, error)

func (*Runner) GenerateChat 

func (r *Runner) GenerateChat(messages []ChatMessage, options GenerationOptions) (GenerationResult, error)

func (*Runner) GenerateChatStream 

func (r *Runner) GenerateChatStream(messages []ChatMessage, options GenerationOptions, onToken func(string)) (GenerationResult, error)

func (*Runner) GenerateChatStreamUntil 

func (r *Runner) GenerateChatStreamUntil(messages []ChatMessage, options GenerationOptions, onToken func(string) bool) (GenerationResult, error)

GenerateChatStreamUntil is the generation entry point everything else wraps (Generate, GenerateChat, GenerateStream, ... are thin adapters over it): render messages through the model's chat template, prefill the prompt (batched when the architecture allows), then decode token by token until EOS, a stop sequence, max_tokens, or the context limit. onToken receives valid-UTF-8 text increments (bytes are buffered across token boundaries until they complete a rune, and the tail is held back while it could still be a stop-sequence prefix); returning false cancels generation, yielding the partial result with ErrGenerationCanceled. The final result carries content with reasoning and tool calls already extracted (classifyOutput) and a FinishReason of "stop", "length", or "tool_calls".

func (*Runner) GenerateStream 

func (r *Runner) GenerateStream(prompt string, options GenerationOptions, onToken func(string)) (GenerationResult, error)

func (*Runner) ModelName 

func (r *Runner) ModelName() (string, bool)

func (*Runner) NearestTokens 

func (r *Runner) NearestTokens(id uint32, k int) ([]TokenMatch, error)

NearestTokens returns the k vocabulary tokens whose embedding vectors are most cosine-similar to token id's embedding (excluding id itself) — the "which tokens does the model treat as related?" view. It dequantizes and scans the full embedding table (parallelized), so expect O(vocab*dim) work: fractions of a second for 32K vocabularies, a few seconds at 262K.

func (*Runner) Tokenizer 

func (r *Runner) Tokenizer() *Tokenizer

type SamplerConfig 

type SamplerConfig struct {
	Temperature   float32
	TopP          float32
	TopK          int
	MinP          float32
	RepeatPenalty float32
}

SamplerConfig holds the user-tunable sampling knobs. Disabled values: TopK <= 0 (consider all tokens), TopP >= 1, MinP <= 0, RepeatPenalty == 1. Temperature 0 means greedy decoding. See the file comment above for how the knobs compose, and GenerationOptions.Validate for the accepted ranges.

func DefaultSamplerConfig 

func DefaultSamplerConfig() SamplerConfig

DefaultSamplerConfig is a generic chat baseline. Model vendors publish family-specific recommendations that override these (e.g. Gemma wants temp 1.0 / top-p 0.95 / top-k 64 — see docs/INFERENCE_NOTES.md).

type ServeOptions 

type ServeOptions struct {
	Addr                     string
	Defaults                 GenerationOptions
	MaxConcurrentConnections int
	ChatUI                   bool
	ChatHistoryPath          string
	ChatHistoryLock          *sync.Mutex
	ModelDir                 string
	ModelPath                string
	SkillsDir                string
	// LogWriter receives startup and handler diagnostics; Serve defaults it
	// to os.Stderr (CLI behavior), unlike NewHandler's io.Discard.
	LogWriter io.Writer
}

ServeOptions is HandlerOptions plus the listen address, for the Serve convenience wrapper (used by the CLI). ChatHistoryPath/ChatHistoryLock are retained for compatibility but unused.

type Skill 

type Skill struct {
	Name        string
	Description string
	Body        string
	Path        string
}

Skills give a model on-demand access to written-out instructions it doesn't need in context by default: a name and one-line description are always visible (via the load_skill tool's description), and the full body is only loaded when the model actually asks for it — the same progressive-disclosure design as Claude's Agent Skills. Layout on disk mirrors that convention too: 

skills/
  pdf-fill/SKILL.md
  git-review/SKILL.md

each SKILL.md is a Markdown file with a YAML-ish frontmatter header: 

---
name: pdf-fill
description: Fill out a PDF form given field values.
---
Full instructions the model receives once it loads this skill...

A flat "skills/*.md" layout (no per-skill subdirectory) is also accepted, using the filename (minus extension) as the fallback name.

func LoadSkills 

func LoadSkills(dir string) ([]Skill, error)

LoadSkills discovers every skill under dir. An empty dir returns (nil, nil) so the feature is a no-op unless explicitly configured.

type TensorInfo 

type TensorInfo struct {
	Name   string
	Dims   []uint64
	DType  GGMLType
	Offset uint64
}

TensorInfo describes one tensor from the GGUF header. Dims follow GGUF's convention of fastest-varying dimension first, so for a 2-D weight Dims[0] is the input (column) count and Dims[1] the output (row) count. Offset is relative to GGUFFile.DataOffset.

func (TensorInfo) Numel 

func (t TensorInfo) Numel() int

Numel returns the total element count (product of Dims).

type TensorStat 

type TensorStat struct {
	Name   string
	Type   GGMLType
	Params int64
	Bytes  int64
}

TensorStat identifies one tensor and its size for the largest-tensors list.

type TokenMatch 

type TokenMatch struct {
	ID    uint32
	Text  string
	Score float32
}

TokenMatch is one vocabulary entry, with Score meaning depending on the producing call: SearchTokens leaves it 0; NearestTokens sets cosine similarity.

func SearchTokens 

func SearchTokens(tok *Tokenizer, query string, limit int) []TokenMatch

SearchTokens finds vocabulary entries whose decoded text contains query (case-insensitive), exact matches first, capped at limit (<=0 means 100). Both the raw vocab form (with byte/▁ markers) and the human-decoded form are searched, so "hello", " hello", and "▁hello"-style pieces all match.

type TokenProb 

type TokenProb struct {
	Token int
	Prob  float32
}

TokenProb pairs a token id with its (possibly unnormalized) weight while a candidate set is being filtered and sampled.

type Tokenizer 

type Tokenizer struct {
	Vocab       []string
	Scores      []float32
	TokenToID   map[string]uint32
	MergeRanks  map[Pair]int
	ByteEncoder map[byte]rune
	ByteDecoder map[rune]byte
	Mode        TokenizerMode
	Pre         string
	AddBOS      bool
	BOSID       uint32
	EOSID       uint32
}

Tokenizer implements the two GGUF tokenizer families:

  • TokenizerSentencePiece (tokenizer.ggml.model = "llama"): text is mapped to "▁"-prefixed pieces and greedily merged by vocabulary Scores, with <0xNN> byte tokens as the fallback for uncovered bytes.
  • TokenizerGPT2BPE ("gpt2", incl. Mistral's Tekken and Qwen): text is pre-tokenized by a regex-equivalent splitter (Pre selects the Tekken variant), bytes are mapped through the GPT-2 printable-byte alphabet (ByteEncoder/ByteDecoder), and pieces are merged by MergeRanks.

AddBOS mirrors tokenizer.ggml.add_bos_token: Encode prepends BOSID when set, EncodeWithoutBOS never does (chat renderers place BOS themselves).

func TokenizerFromMetadata 

func TokenizerFromMetadata(metadata map[string]MetaValue) (*Tokenizer, error)

TokenizerFromMetadata builds a Tokenizer from a GGUF's tokenizer.* keys: vocabulary, merge ranks/scores, BOS/EOS ids, and the model/pre strings that select the encoding mode.

func (*Tokenizer) DecodeToken 

func (t *Tokenizer) DecodeToken(id uint32) string

DecodeToken renders one token id back to text: GPT-2 tokens go through the byte alphabet, SentencePiece <0xNN> byte tokens decode to their raw byte, and "▁" markers become spaces. Byte tokens may produce partial UTF-8; the generation loop buffers output until it is valid (validUTF8PrefixLen).

func (*Tokenizer) Encode 

func (t *Tokenizer) Encode(text string) []uint32

func (*Tokenizer) EncodeWithoutBOS 

func (t *Tokenizer) EncodeWithoutBOS(text string) []uint32

func (*Tokenizer) SpecialID 

func (t *Tokenizer) SpecialID(token string) (uint32, bool)

func (*Tokenizer) VocabSize 

func (t *Tokenizer) VocabSize() int

type TokenizerMode 

type TokenizerMode int
const (
	TokenizerSentencePiece TokenizerMode = iota
	TokenizerGPT2BPE
)

type ToolCall 

type ToolCall struct {
	ID       string           `json:"id"`
	Type     string           `json:"type"` // always "function"
	Function ToolCallFunction `json:"function"`
}

ToolCall is one function call requested by the assistant, OpenAI-compatible.

type ToolCallFunction 

type ToolCallFunction struct {
	Name      string `json:"name"`
	Arguments string `json:"arguments"`
}

ToolCallFunction is the OpenAI-compatible function payload of a tool call. Arguments is a JSON-encoded object (a string, matching the OpenAI wire format), not a nested object, so it round-trips through JSON unchanged regardless of what the caller's argument schema looks like.

type ToolDefinition 

type ToolDefinition struct {
	Type     string          `json:"type"` // always "function"
	Function ToolFunctionDef `json:"function"`
}

ToolDefinition is one entry of an OpenAI-compatible "tools" array.

type ToolFunctionDef 

type ToolFunctionDef struct {
	Name        string          `json:"name"`
	Description string          `json:"description,omitempty"`
	Parameters  json.RawMessage `json:"parameters,omitempty"`
}

ToolFunctionDef describes a callable function, OpenAI-compatible.

type Weight 

type Weight struct {
	F32      []float32
	Raw      []byte
	Type     GGMLType
	Rows     int
	Cols     int
	Prepared *PreparedQuantizedWeight
	Metal    *MetalWeight
}

Weight is one loaded tensor in exactly one of two states: F32 non-nil (plain floats, converted at load time from F32/F16 storage) or Raw non-nil (still-quantized bytes, usually borrowed zero-copy from the mmap'd file, dequantized on the fly inside the matvec kernels). Rows/Cols only apply to the quantized form; the F32 form infers rows from len(F32)/cols at the call site.

func (Weight) ArgmaxMatvec added in v0.2.0 

func (w Weight) ArgmaxMatvec(x []float32) (uint32, bool)

ArgmaxMatvec returns argmax(W*x) without materializing the full logits vector. Observed bottleneck: Ministral-3 3B Q4_K_M spends most decode time in the 131k-row output projection. For deterministic decoding, the sampler only needs the winning token, so this saves the logits writeback and second full vocab scan. Risk is limited by using it only for exact greedy-compatible sampler settings; rollback is to disable the runtime fast-path.

func (Weight) Matvec 

func (w Weight) Matvec(x []float32) []float32

Matvec computes out = W·x, allocating the result. MatvecInto is the allocation-free form used on the decode hot path; it dispatches to the quant-type-specific parallel kernel in simd.go.

func (Weight) MatvecInto 

func (w Weight) MatvecInto(x []float32, out *[]float32)

func (Weight) Row 

func (w Weight) Row(row, cols int) []float32

Row dequantizes a single weight row (used for token-embedding lookups). RowInto is the allocation-free form.

func (Weight) RowF32 

func (w Weight) RowF32(row, cols int) []float32

func (Weight) RowInto 

func (w Weight) RowInto(row, cols int, out *[]float32)

Source Files

View all Source files

Directories

Path Synopsis
cmd
gopherllm command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.