jitllm

module
v0.2.2-draft.2 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 11, 2026 License: Apache-2.0

README

jitllm

jitllm

An operating system for LLM inference: generate the compute for your hardware, page models larger than memory, and move execution without losing the conversation.

jitllm.org · Documentation · Latest release

  • Every kernel JIT-generated at load time for your CPU (AVX2, SSE, NEON) or GPU (CUDA, Vulkan, Metal), specialised to the model's shapes and weight formats. No precompiled kernels or interpreted compute fallback.
  • Disk, RAM and VRAM form the memory system. Weights, routed MoE experts and KV state page as needed. Blocks move between CPU and GPUs at run time, carrying their attention and recurrent state with them.
  • Runs everywhere, depends on nothing. One binary per platform (Linux, macOS, Windows; x86 and Arm) that finds CUDA, Vulkan or Metal at run time and otherwise runs on the CPU.

Serve it as an OpenAI- and Anthropic-compatible API with streaming and tool calling from one standalone binary, jitllmd, or embed it as a Go library. There are desktop and terminal apps too.

Install

macOS: download jitllm_<version>_darwin_arm64.dmg (Apple silicon) or _amd64.dmg (Intel) from the latest release, open it and drag jitllm to Applications (or take the signed app alone, jitllm_<version>_darwin_<arch>.app.zip, and unzip it there). The app carries jitllm and jitllmd; on first launch it offers to put them on your PATH, and its Settings can start the server at login. With Homebrew (available once the tap is published):

brew install --cask jitllm/tap/jitllm   # the app, with jitllm and jitllmd on PATH
brew install jitllm/tap/jitllm          # jitllm and jitllmd only

Windows: run jitllm-setup_<version>_windows_amd64.exe (or _arm64.exe) from the latest release. It installs for your account only, with no administrator prompt: the CLI, the server and the desktop app into %LOCALAPPDATA%\Programs\jitllm, on your PATH, with Start menu shortcuts, an optional "start the server at login", and an uninstaller under Settings > Apps. With winget (available once published): winget install jitllm.jitllm. To run the desktop app without installing anything, download the portable jitllm-desktop_<version>_windows_amd64.exe (or _arm64.exe) and run it from anywhere; it shares its settings and chats with an installed copy.

Until releases are signed, macOS and Windows warn the first time: packaging/README.md says what each warning means and how to open the app.

Any platform, from a script. Linux and macOS:

curl -fsSL https://jitllm.org/install.sh | sh

Windows (PowerShell):

irm https://jitllm.org/install.ps1 | iex

The script installs the CLI, jitllm, and the server, jitllmd, from the latest release, after checking each archive against the release's checksums: into ~/.local/bin (/usr/local/bin as root), or on Windows into %LOCALAPPDATA%\Programs\jitllm, which it adds to your PATH. Add the apps with | sh -s -- all (or desktop, tui), or $env:JITLLM_PROGRAMS = "all" before the Windows line. Each program is one self-contained binary and its own archive on the Releases page, for Linux, macOS and Windows on x86-64 and arm64. GPUs need only their driver.

AMD GPUs: they run through Vulkan with only the driver. With ROCm installed, jitllm also uses them through a native backend (hip:0, ...), which has not yet been tested on AMD hardware (#34). To stay on the tested path, pick the Vulkan device explicitly (-devices vulkan:0).

To build from source instead, with Go 1.26 or newer and no C toolchain:

git clone https://github.com/jitllm/jitllm.git && cd jitllm
go build ./cmd/jitllm
(cd server && go build -o ../jitllmd ./cmd/jitllmd)
(cd ui && CGO_ENABLED=0 go build -o ../jitllm-desktop .)
(cd tui && go build -o ../jitllm-tui .)

Quick start

# Fetch Qwen3-30B-A3B from the catalog and convert it (18.6 GB): a mixture of
# experts with 30B weights and 3B used per token. With less memory than it
# needs, it pages from disk.
mkdir -p models
jitllm convert -o models qwen3-30b-a3b models/qwen3.jlm

# Ask it something from the command line. It thinks first, then answers.
jitllm run -chat models/qwen3.jlm "Why is the sky blue?"

# Or serve it.
jitllmd serve -addr 127.0.0.1:8080 -models ./models \
  -load qwen3.jlm -id qwen3 -devices auto

From another terminal:

curl http://127.0.0.1:8080/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3","messages":[{"role":"user","content":"Why is the sky blue?"}],"stream":true}'

Flags go before the model path. jitllm <command> with no arguments prints the usage.

Models run from .jlm containers, converted once from GGUF or Hugging Face weights. See the CLI and conversion guide for other inputs and flags, and the server reference for endpoints and daemon commands.

Docker
docker volume create jitllm-models
docker run --rm -v jitllm-models:/models --entrypoint jitllm ghcr.io/jitllm/jitllm convert qwen3-30b-a3b
docker run -d -p 8080:8080 -v jitllm-models:/models ghcr.io/jitllm/jitllm -load Qwen3-30B-A3B-Q4_K_M.jlm -id qwen3

Add --gpus all (with the NVIDIA Container Toolkit) to run on an NVIDIA GPU. The image is jitllmd with the jitllm CLI beside it, for linux/amd64 and linux/arm64; docker build -t jitllm . builds it from a checkout. See docs/docker.md.

Desktop app

Model downloads, conversion, text and image chat, and a live view of where each block of the model sits, in one window. Install it with | sh -s -- desktop (or take its archive from the Releases page) and run jitllm-desktop.

Terminal app

The same screens in a terminal, over SSH too, sharing the desktop app's settings and chats. The arrow keys move the model's blocks between the CPU and the GPU, and the conversation keeps its history across the move. Install it with | sh -s -- tui and run jitllm-tui models/qwen3.jlm.

Supported models

convert reads 65 GGUF architecture names (57 graphs), 30 Hugging Face safetensors classes and 17 vision projectors; anything else is refused at conversion, by name.

  • Text: Llama 2/3, Mistral, Mixtral, SmolLM2/3, Qwen2 to Qwen3.6 (dense, MoE, Next and VL text), Gemma 1 to 4 and 3n, Phi-2/3/4 and Phi-3.5-MoE, DeepSeek V2/V3/R1/V3.2/V4, Kimi-K2, Kimi Linear, Kimi-K3, GLM-4/4.5/4.6/4.7-Flash, gpt-oss, Llama 4, Granite, OLMo 2/3, OLMoE, ERNIE 4.5, Hunyuan, MiniMax-M2/M3, Ling 2.0, dots.llm1, Apertus, EXAONE 4, Seed-OSS, Command-R/A, DBRX, Falcon, StarCoder 1/2, StableLM, Nemotron
  • State-space hybrids: Mamba, Mamba-2, Jamba, Falcon-H1, Granite 4 hybrid, Nemotron-H, LFM2
  • Vision: SmolVLM, LLaVA, Qwen2/2.5/3-VL, Qwen3.5, GLM-4.xV, Kimi-VL, HunyuanOCR, Gemma 3/3n/4, InternVL, MiniCPM-V, Janus-Pro, Pixtral/Mistral 3, Phi-4 vision, Llama 4
  • Embeddings: BERT family, nomic-embed-text, Qwen3-Embedding, EmbeddingGemma

Weights: F32, F16, BF16, Q4_0, Q5_0, Q5_1, Q8_0, Q3_K-Q6_K and MXFP4. The full list, generated from the code, is docs/models.md; the vision guide covers pictures.

Performance

Results vary by model, hardware and workload. The scoreboard compares engines measured in the same pass on the same machine, with hardware, backends, prompt lengths and generation lengths alongside the results. The performance overview preserves the comparison tables and their limitations; scripts/vs-llamacpp.sh measures your machine.

Documentation

The five principles govern every execution path. See how it works for the runtime and source layout, or memory and placement for budgets and relocation.

Embed it in Go

The CLI, server and desktop app use the same engine. This example applies the model's chat template and generates a greedy response:

package main

import (
    "fmt"
    "log"

    "github.com/jitllm/jitllm/engine/model"
)

func main() {
    m, err := model.Open("models/qwen3.jlm")
    if err != nil {
        log.Fatal(err)
    }
    defer m.Close()

    ids, err := m.ChatIDs([]model.ChatMessage{
        {Role: "user", Content: "Why is the sky blue?"},
    }, true)
    if err != nil {
        log.Fatal(err)
    }
    state := m.NewState(8192)
    defer state.Close()

    logits, err := state.Prefill(ids)
    for err == nil && state.Pos() < 8192 {
        id := model.Greedy(logits)
        if m.Vocab.IsEOG(id) {
            break
        }
        fmt.Print(m.Vocab.Decode([]int32{id}))
        logits, err = state.Forward(id)
    }
    if err != nil {
        log.Fatal(err)
    }
    fmt.Println()
}

See the Go library guide for budgets, devices, KV caching, embeddings and batching.

Directories

Path Synopsis
cmd
jitllm command
Command jitllm is the jitllm CLI.
Command jitllm is the jitllm CLI.
Package convert turns a source model into a jlm container.
Package convert turns a source model into a jlm container.
gguf
Package gguf parses GGUF model files into a meta.File.
Package gguf parses GGUF model files into a meta.File.
hf
Package hf acquires GGUF weights from HuggingFace.
Package hf acquires GGUF weights from HuggingFace.
library
Package library is the list of models jitllm can fetch by name: for each, a Hugging Face repository, the file in it, and the vision tower beside it when there is one.
Package library is the list of models jitllm can fetch by name: for each, a Hugging Face repository, the file in it, and the vision tower beside it when there is one.
safetensors
Package safetensors is a parser for the safetensors container, and nothing else.
Package safetensors is a parser for the safetensors container, and nothing else.
dev
bench
Package bench is the measurement harness.
Package bench is the measurement harness.
httpbench command
httpbench is the one HTTP load harness every engine is measured with: a closed-loop concurrency sweep of streamed OpenAI-compatible requests, two or more endpoints run as interleaved arms.
httpbench is the one HTTP load harness every engine is measured with: a closed-loop concurrency sweep of streamed OpenAI-compatible requests, two or more endpoints run as interleaved arms.
httpbench/hb
Package hb is the HTTP load harness behind dev/httpbench: one closed-loop concurrency sweep, measured the same way against every OpenAI-compatible engine, with two or more engines run as interleaved arms.
Package hb is the HTTP load harness behind dev/httpbench: one closed-loop concurrency sweep, measured the same way against every OpenAI-compatible engine, with two or more engines run as interleaved arms.
tools/fmtbench command
Command fmtbench prices each quant format's CPU decode matvec against a read wall measured in the same process, every shape sized DRAM-resident.
Command fmtbench prices each quant format's CPU decode matvec against a read wall measured in the same process, every shape sized DRAM-resident.
tools/mvbench command
Command mvbench measures the GPU decode matvec against a device read-bandwidth ceiling measured in the same process.
Command mvbench measures the GPU decode matvec against a device read-bandwidth ceiling measured in the same process.
tools/ptxdump command
Command ptxdump lowers named kernels to PTX so ptxas and cuobjdump can be pointed at them: the SASS is the only thing that says what a kernel actually issues (the MMA matvec turned out to be mostly address arithmetic).
Command ptxdump lowers named kernels to PTX so ptxas and cuobjdump can be pointed at them: the SASS is the only thing that says what a kernel actually issues (the MMA matvec turned out to be mostly address arithmetic).
tools/readbench command
readbench prices the jlm reader: parse, then page-in by four strategies.
readbench prices the jlm reader: parse, then page-in by four strategies.
tools/stridebench command
stridebench prices the access pattern the packed kernel actually makes.
stridebench prices the access pattern the packed kernel actually makes.
engine
grammar
Package grammar constrains generation to a context-free grammar: GBNF text (llama.cpp's grammar format) or a JSON Schema compiled to it, matched one code point at a time by a pushdown automaton and turned, at each step, into the set of vocabulary tokens that keep the output inside the grammar.
Package grammar constrains generation to a context-free grammar: GBNF text (llama.cpp's grammar format) or a JSON Schema compiled to it, matched one code point at a time by a pushdown automaton and turned, at each step, into the set of vocabulary tokens that keep the output inside the grammar.
model
Package model loads a jlm container and runs its transformer graph on generated kernels, on the host and on whichever devices take its blocks.
Package model loads a jlm container and runs its transformer graph on generated kernels, on the host and on whichever devices take its blocks.
nn
sched
Package sched runs work across the machine's performance cores.
Package sched runs work across the machine's performance cores.
format
jlm
Package jlm is jitllm's own model container: weights already in the device layout, one page per block, the tokenizer travelling with them.
Package jlm is jitllm's own model container: weights already in the device layout, one page per block, the tokenizer travelling with them.
quant
Package quant is the quantization formats themselves: the type enum, each format's block geometry, and the reference decoders.
Package quant is the quantization formats themselves: the type enum, each format's block geometry, and the reference decoders.
internal
cmd/goheap
Package goheap is the process-wide collector setup both binaries share: cmd/jitllm and server/cmd/jitllmd.
Package goheap is the process-wide collector setup both binaries share: cmd/jitllm and server/cmd/jitllmd.
modelsdoc
Package modelsdoc renders docs/models.md, the supported-models list, from the converter's own tables: convert.Architectures (the GGUF names), convert.HFClasses (the Hugging Face classes), convert.Projectors (the vision projectors), quant's type lists and the download catalogue.
Package modelsdoc renders docs/models.md, the supported-models list, from the converter's own tables: convert.Architectures (the GGUF names), convert.HFClasses (the Hugging Face classes), convert.Projectors (the vision projectors), quant's type lists and the download catalogue.
oracle
Package oracle is the engine's arithmetic written in plain Go: the reference every generated kernel is gated against, and nothing else.
Package oracle is the engine's arithmetic written in plain Go: the reference every generated kernel is gated against, and nothing else.
schemacheck
Package schemacheck is the structured-output gates' JSON Schema validator, for the subset engine/grammar compiles: what an output is held to, written independently of the grammar that produced it.
Package schemacheck is the structured-output gates' JSON Schema validator, for the subset engine/grammar compiles: what an output is held to, written independently of the grammar that produced it.
testlock
Package testlock serialises GPU tests across processes.
Package testlock serialises GPU tests across processes.
testmodels
Package testmodels is where a test finds a model file.
Package testmodels is where a test finds a model file.
jit
cpu
Package cpu emits x86-64 and arm64 machine code at load time.
Package cpu emits x86-64 and arm64 machine code at load time.
gpu/amdgpu
Package amdgpu lowers jit/gpu/ir kernels to LLVM IR text for AMD GPUs.
Package amdgpu lowers jit/gpu/ir kernels to LLVM IR text for AMD GPUs.
gpu/backend
Package backend puts one interface over the three GPU runtimes so a benchmark or a tuner can ask the same question of each.
Package backend puts one interface over the three GPU runtimes so a benchmark or a tuner can ask the same question of each.
gpu/ffi
Package ffi binds C functions without cgo, on top of goffi.
Package ffi binds C functions without cgo, on top of goffi.
gpu/hip
Package hip binds AMD's ROCm runtime (libamdhip64 on Linux, amdhip64_N.dll from the HIP SDK on Windows) and code object manager (libamd_comgr) through jit/gpu/ffi, with no cgo.
Package hip binds AMD's ROCm runtime (libamdhip64 on Linux, amdhip64_N.dll from the HIP SDK on Windows) and code object manager (libamd_comgr) through jit/gpu/ffi, with no cgo.
gpu/ir
Package ir is a very small typed kernel IR for NN compute.
Package ir is a very small typed kernel IR for NN compute.
gpu/kernels
Package kernels holds the NN kernels written against the IR.
Package kernels holds the NN kernels written against the IR.
gpu/metal
Package metal is Apple-only.
Package metal is Apple-only.
gpu/msl
Package msl lowers the IR to Metal Shading Language.
Package msl lowers the IR to Metal Shading Language.
gpu/ptx
Package ptx is an assembler for PTX text: registers, labels and a buffer.
Package ptx is an assembler for PTX text: registers, labels and a buffer.
gpu/spirv
Package spirv emits SPIR-V compute modules from the IR.
Package spirv emits SPIR-V compute modules from the IR.
gpu/tier
Package tier is jitllm's GPU decode tier: an nn.Device backed by the generated device kernels (jit/gpu).
Package tier is jitllm's GPU decode tier: an nn.Device backed by the generated device kernels (jit/gpu).
gpu/vulkan
Package vulkan runs SPIR-V compute kernels through the Vulkan loader.
Package vulkan runs SPIR-V compute kernels through the Vulkan loader.
scripts
genmodels command
Command genmodels writes docs/models.md, the supported-models list, from the converter's own tables (internal/modelsdoc):
Command genmodels writes docs/models.md, the supported-models list, from the converter's own tables (internal/modelsdoc):
gentok command
Command gentok generates tok/pretok/table.go: for every tokenizer.ggml.pre name a GGUF can carry, the ordered pipeline of typed stages that name denotes.
Command gentok generates tok/pretok/table.go: for every tokenizer.ggml.pre name a GGUF can carry, the ordered pipeline of typed stages that name denotes.
genunicode command
Command genunicode generates tok/unicodedata.go from llama.cpp's checked-in Unicode tables.
Command genunicode generates tok/unicodedata.go from llama.cpp's checked-in Unicode tables.
modelfree command
Command modelfree runs every test of the four modules on whatever host CI gives it, with no model download and no GPU, and writes what was SKIPPED into the job summary.
Command modelfree runs every test of the four modules on whatever host CI gives it, with no model download and no GPU, and writes what was SKIPPED into the job summary.
server module
tok
Package tok implements the SentencePiece tokenizer that llama-architecture GGUF models carry (tokenizer.ggml.model == "llama").
Package tok implements the SentencePiece tokenizer that llama-architecture GGUF models carry (tokenizer.ggml.model == "llama").
jinja
Package jinja implements a Jinja2-compatible template engine.
Package jinja implements a Jinja2-compatible template engine.
pretok
Package pretok is the pre-tokenizer model: the stages a tokenizer runs before merging, the parser that reads them out of a tokenizer.json, and the table generated from the artefacts themselves.
Package pretok is the pre-tokenizer model: the stages a tokenizer runs before merging, the parser that reads them out of a tokenizer.json, and the table generated from the artefacts themselves.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL