README
¶
goinfer
An entire LLM in one file — instant boot (~0.5s), <100 MB heap, runs offline. No cgo, no Python, no model download.
Run open-weight LLMs in pure Go. No cgo, no Python, no llama.cpp — one cross-compiled static binary, with HuggingFace logit parity.
goinfer is a pure-Go, no-cgo decoder-only LLM runtime. It loads open-weight models — Gemma 3/4, Qwen 2.5/3, Llama 2/3, Mistral, Mixtral, GPT-2, Mellum — directly from safetensors (single or sharded), GGUF, GPTQ, or AWQ checkpoints and runs them in-process: f32/bf16/f16 plus int8 and int4 quantization, KV-cache, all standard samplers, and constrained/structured decoding (a model that cannot emit malformed JSON). Forward-pass numerics are parity-gated against HuggingFace; matmul is SIMD-accelerated (NEON on arm64, AVX2/FMA on amd64). Because it's pure Go with no cgo, it cross-compiles to a single static binary — no Python, no native runtime, no provider API.
Not to be confused with provider-orchestration libraries (e.g. teilomillet/gollm) that call remote LLM APIs. goinfer runs the weights itself, locally, in-process.
Built on aikit's embedding and tensor
primitives.
Try it: an LLM in one file
demo/chat is a local coding assistant that's a single static
binary — the runtime and the model in one file. Download it, run it, chat
offline: no install, no Python, no cgo, no model download.
Grab a binary from the latest release (macOS / Linux / Windows, Intel + ARM), or run from source against your own GGUF:
go run ./demo/chat --model ~/models/qwen2.5-coder-0.5b-instruct-q4_k_m.gguf
Run Gemma 4 (pure Go, no cgo)
goinfer runs Google's Gemma 4 end to end in pure Go — including the E-models (E2B/E4B) with their Per-Layer-Embedding stack, and the 12B dense. Grab the QAT GGUF and point the demo at it (the Gemma 4 chat template is applied automatically):
# E2B (~3 GB Q4_0) — the small one; E4B and the 12B dense work the same way
go run ./demo/chat --model ~/models/gemma-4-E2B_q4_0-it.gguf
you> What is the capital of France?
The capital of France is Paris.
Every Gemma 4 forward is parity-gated against the HuggingFace bf16 reference (argmax-exact + logit cosine). Text models only — the 26B-A4B MoE and 31B multimodal vision towers are out of scope.
See demo/chat/README.md for commands, canned demos, and
how the single-file binary is built.
Status
Pre-1.0; the forward-pass / quantization contract is parity-gated and stable, the
loader and architecture-descriptor surface is still moving as new model families
land. See CHANGELOG.md.
Install
go get github.com/townsendmerino/goinfer
Packages
| Package | Purpose | Deps beyond stdlib |
|---|---|---|
decoder |
generic decoder-only forward pass; f32/bf16/f16 + int8/int4; safetensors/GGUF/GPTQ/AWQ; KV-cache; samplers | aikit/embed, aikit/linalg, goinfer/tokenizer |
tokenizer |
BPE tokenizers the decoder LLMs ship — byte-level + SentencePiece byte-fallback, from tokenizer.json or a bare .gguf; HF-exact id parity |
aikit/embed, golang.org/x/text |
constrain |
constrained / structured decoding — a logit mask that forces output to satisfy a grammar; ships a streaming JSON grammar | — |
gpu (opt-in, -tags gpu) |
WebGPU compute backend for matmul (Metal / Vulkan / DX12) | cogentcore/webgpu (cgo), aikit/encoder, goinfer/decoder |
The cgo WebGPU dependency is confined to the gpu submodule; the default build is
pure Go, no cgo.
Quick start
See demo/gemma for a working CLI: load a tokenizer (GGUF or HF), load a decoder,
stream tokens with optional sampling and JSON-constrained output.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
prequant
command
Command prequant builds a goinfer prequant bundle (.giw) from a GGUF model.
|
Command prequant builds a goinfer prequant bundle (.giw) from a GGUF model. |
|
Package constrain implements constrained / structured decoding: a logit mask that forces a language model's output to satisfy a grammar (e.g.
|
Package constrain implements constrained / structured decoding: a logit mask that forces a language model's output to satisfy a grammar (e.g. |
|
cuda
module
|
|
|
Package decoder runs autoregressive, decoder-only transformer language models as a pure-Go forward pass.
|
Package decoder runs autoregressive, decoder-only transformer language models as a pure-Go forward pass. |
|
demo
|
|
|
chat
command
Command chat is goinfer's interactive demo: a terminal REPL around a local decoder-only LLM running on the pure-Go decoder — no cgo, no Python, no model download step (in the embed build the model ships *inside* the binary).
|
Command chat is goinfer's interactive demo: a terminal REPL around a local decoder-only LLM running on the pure-Go decoder — no cgo, no Python, no model download step (in the embed build the model ships *inside* the binary). |
|
gemma
command
Command gemma is a demo CLI that runs a local decoder-only LLM through aikit's pure-Go decoder and streams the completion to stdout.
|
Command gemma is a demo CLI that runs a local decoder-only LLM through aikit's pure-Go decoder and streams the completion to stdout. |
|
gemma-web
command
Command gemma-web serves a local, single-page web chat GUI for a decoder LLM checkpoint running on aikit's pure-Go decoder.
|
Command gemma-web serves a local, single-page web chat GUI for a decoder LLM checkpoint running on aikit's pure-Go decoder. |
|
agent
module
|
|
|
gpu
module
|
|
|
internal
|
|
|
giw
Package giw frames a prequant "goinfer weights" bundle: the serialized, already-quantized decoder weights (decoder.SerializeWeights) plus a tiny metadata-only GGUF carrying the tokenizer (the source GGUF truncated at the tensor-data boundary — ~MBs, no weights).
|
Package giw frames a prequant "goinfer weights" bundle: the serialized, already-quantized decoder weights (decoder.SerializeWeights) plus a tiny metadata-only GGUF carrying the tokenizer (the source GGUF truncated at the tensor-data boundary — ~MBs, no weights). |
|
metal
module
|
|
|
Package tokenizer implements the BPE tokenizers the decoder LLMs ship, loaded from the HF tokenizer.json.
|
Package tokenizer implements the BPE tokenizers the decoder LLMs ship, loaded from the HF tokenizer.json. |