jitllm

module
v0.2.6 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 15, 2026 License: Apache-2.0

README

jitllm

Your models. Your hardware. Full speed.

An operating system for LLM inference.
JIT kernels that match your hardware · paging so big models still run · auto-tuning & layer orchestration

Latest release Apache-2.0 license

CUDA supported Vulkan supported Metal on Apple Silicon ROCm untested CPU x86 and Arm
Linux x86-64 and arm64 macOS arm64 and x86-64 Windows x86-64 and arm64

jitllm.org · Get started · Benchmarks · Docs · Download

Server · CLI · Desktop · TUI · Embed

Why jitllm

Inference engines are bulky and rigid. There's a lot to set up, and even after you get a model running, getting the most out of your hardware is another job.

I wanted to build something easier to live with: one binary you can drop onto a machine and start running models. Something that does more of the work of figuring out the hardware itself.

The inspiration came from operating systems and JIT compilers, two things I've worked on a lot. An OS detects hardware, uses the right drivers, schedules processes, and manages memory. A JIT compiler generates code for the machine it's running on. jitllm brings those ideas together to make inference faster, easier to run, and more flexible.

That's why I call it an operating system for LLM inference. It detects your CPUs and GPUs, generates optimized kernels, and orchestrates model placement at runtime. Models larger than memory page from disk. Layers can move between devices while keeping their conversation state, so changing placement doesn't mean starting over.

The engine

  • Kernels written for the machine in front of it. Every kernel is emitted at load for the model's shapes and weight formats: x86 (SSE, AVX2, AVX-512, VNNI) and Arm (NEON, dot-product) machine code on the CPU, PTX for CUDA (tensor cores from Volta on), SPIR-V for Vulkan, Metal shaders, and AMDGPU for ROCm (not yet tested on AMD hardware). No vendor math library, no interpreted fallback.
  • Layers move mid-conversation. Blocks relocate between the CPU and any GPU, or from one GPU to another, carrying their KV pages and recurrent state, and the answer stays the same.
  • Models bigger than memory. Disk, RAM and VRAM are one paged hierarchy: weights page in on demand (read with O_DIRECT, never mmap), every expert of a mixture is its own page so a token reads only the experts it routes to, and KV history pages and streams through the card.
  • Broad coverage, held to the reference. 57 graphs (65 GGUF architecture names, 30 Hugging Face classes) and 17 vision projectors: dense, mixture-of-experts, MLA, gated-delta and state-space hybrids, vision-language, embeddings and decision models, each gated against llama.cpp or the model's own transformers class.
  • No garbage on the hot path. A warm decode token allocates nothing on the Go heap, on the host and every GPU backend; weights and page frames live off the heap.
  • Faster tokens, same distribution. Speculative decoding with a model's own prediction head or prompt lookup, an int8 KV cache, prefix caching and shared prefixes across sessions.

The server

One binary, jitllmd, many models and many sessions at once:

  • OpenAI, Anthropic and Connect APIs: chat, completions, messages, embeddings, Batches and Files, with streaming.
  • Tool calling in each model's own format (24 families, from Hermes and Llama to gpt-oss Harmony, DeepSeek, Kimi and GLM), with tool_choice enforced by constrained decoding, and structured output from a JSON schema.
  • Pictures through image_url and image blocks on vision models, log-probabilities, n choices from one prefill, and decision answers through /v1/systemone.
  • Continuous batching on the CPU and the GPU, prompts fed in chunks beside running decodes, KV-aware admission, and a fairness level (-fairness 0-100) that time-slices sessions through the card by parking them to the host and back, with per-request priority.
  • A control plane: load and unload models, create, park and resume sessions, move blocks between devices (jitllmd models, sessions, place, stats), and /metrics for Prometheus.

Desktop app

Model downloads, conversion, text and image chat, and a live view of where each block of the model sits, in one window. Get jitllm.app (.dmg) on macOS or the setup or portable .exe on Windows from the latest release, or install it from a terminal and run jitllm-desktop:

curl -fsSL https://jitllm.org/install.sh | sh -s -- desktop
$env:JITLLM_PROGRAMS = "desktop"; irm https://jitllm.org/install.ps1 | iex

Terminal app

The same screens in a terminal, over SSH too, sharing the desktop app's settings and chats. The arrow keys move the model's blocks between the CPU and the GPU, and the conversation keeps its history across the move. Install it and run jitllm-tui models/qwen3.jlm:

curl -fsSL https://jitllm.org/install.sh | sh -s -- tui
$env:JITLLM_PROGRAMS = "tui"; irm https://jitllm.org/install.ps1 | iex

Install

macOS: download jitllm_<version>_darwin_arm64.dmg (Apple silicon) or _amd64.dmg (Intel) from the latest release, open it and drag jitllm to Applications (or take the signed app alone, jitllm_<version>_darwin_<arch>.app.zip, and unzip it there). The app carries jitllm and jitllmd; on first launch it offers to put them on your PATH, and its Settings can start the server at login. With Homebrew:

brew tap jitllm/jitllm
brew trust jitllm/jitllm
brew install --cask jitllm   # the app, with jitllm and jitllmd on PATH
brew install jitllm          # jitllm and jitllmd only

Windows: run jitllm-setup_<version>_windows_amd64.exe (or _arm64.exe) from the latest release. It installs for your account only, with no administrator prompt: the CLI, the server and the desktop app into %LOCALAPPDATA%\Programs\jitllm, on your PATH, with Start menu shortcuts, an optional "start the server at login", and an uninstaller under Settings > Apps. With winget (available once published): winget install jitllm.jitllm. To run the desktop app without installing anything, download the portable jitllm-desktop_<version>_windows_amd64.exe (or _arm64.exe) and run it from anywhere; it shares its settings and chats with an installed copy.

Until releases are signed, macOS and Windows warn the first time: packaging/README.md says what each warning means and how to open the app.

Any platform, from a script. Linux and macOS:

curl -fsSL https://jitllm.org/install.sh | sh

Windows (PowerShell):

irm https://jitllm.org/install.ps1 | iex

The script installs the CLI, jitllm, and the server, jitllmd, from the latest release, after checking each archive against the release's checksums: into ~/.local/bin (/usr/local/bin as root), or on Windows into %LOCALAPPDATA%\Programs\jitllm, which it adds to your PATH. Add the apps with | sh -s -- all (or desktop, tui), or $env:JITLLM_PROGRAMS = "all" before the Windows line. Each program is one self-contained binary and its own archive on the Releases page, for Linux, macOS and Windows on x86-64 and arm64. GPUs need only their driver.

AMD GPUs: they run through Vulkan with only the driver. With ROCm installed, jitllm also uses them through a native backend (hip:0, ...), which has not yet been tested on AMD hardware (#34). To stay on the tested path, pick the Vulkan device explicitly (-devices vulkan:0).

Quick start

# Fetch Qwen3-30B-A3B from the catalog and convert it (18.6 GB) into the
# models folder: a mixture of experts with 30B weights and 3B used per token.
# With less memory than it needs, it pages from disk.
jitllm convert qwen3-30b-a3b

# Chat with it. The installed service loads it on first use; with no service
# running, jitllm runs it in its own process instead and says so.
jitllm run qwen3-30b-a3b

# Or ask once.
jitllm run -chat qwen3-30b-a3b "Why is the sky blue?"

On Linux the install script starts jitllmd as a systemd user service on localhost:4242; the macOS app and the Windows installer offer the same at login, and jitllmd service install does it on any of them. Every model in the models folder is available to it by name, loaded on demand:

curl http://localhost:4242/v1/chat/completions \
  -H 'Content-Type: application/json' \
  -d '{"model":"qwen3-30b-a3b","messages":[{"role":"user","content":"Why is the sky blue?"}],"stream":true}'

jitllmd service status shows the service, jitllmd models -alias FILE=NAME gives a model another name, and jitllm run path/to/model.jlm runs a file directly. Flags go before the model. jitllm <command> with no arguments prints the usage.

Bring GGUF or Hugging Face (safetensors) weights, from the catalog or your own files. See the CLI and conversion guide for inputs and flags, and the server reference for endpoints and daemon commands.

Docker

docker volume create jitllm-models
docker run --rm -v jitllm-models:/models --entrypoint jitllm ghcr.io/jitllm/jitllm convert qwen3-30b-a3b
docker run -d -p 4242:4242 -v jitllm-models:/models ghcr.io/jitllm/jitllm -load Qwen3-30B-A3B-Q4_K_M.jlm -id qwen3

Add --gpus all (with the NVIDIA Container Toolkit) to run on an NVIDIA GPU. The image is jitllmd with the jitllm CLI beside it, for linux/amd64 and linux/arm64; docker build -t jitllm . builds it from a checkout. See docs/docker.md.

Supported models

convert reads 65 GGUF architecture names (57 graphs), 30 Hugging Face safetensors classes and 17 vision projectors; anything else is refused at conversion, by name.

Kind Families
Text and code Llama 2/3, Mistral, Mixtral, SmolLM2/3, Qwen2 to Qwen3.6 (dense, MoE, Next and VL text), Gemma 1 to 4 and 3n, Phi-2/3/4 and Phi-3.5-MoE, DeepSeek V2/V3/R1/V3.2/V4, Kimi-K2, Kimi Linear, Kimi-K3, GLM-4/4.5/4.6/4.7-Flash, gpt-oss, Llama 4, Granite, OLMo 2/3, OLMoE, ERNIE 4.5, Hunyuan, MiniMax-M2/M3, Ling 2.0, dots.llm1, Apertus, EXAONE 4, Seed-OSS, Command-R/A, DBRX, Falcon, StarCoder 1/2, StableLM, Nemotron
State-space and hybrid Mamba, Mamba-2, Jamba, Falcon-H1, Granite 4 hybrid, Nemotron-H, LFM2
Vision SmolVLM, LLaVA, Qwen2/2.5/3-VL, Qwen3.5, GLM-4.xV, Kimi-VL, HunyuanOCR, Gemma 3/3n/4, InternVL, MiniCPM-V, Janus-Pro, Pixtral/Mistral 3, Phi-4 vision, Llama 4
Embeddings BERT family, nomic-embed-text, Qwen3-Embedding, EmbeddingGemma
Decision models Laya (ModernBERT), d1-3B, Lev 4B, answering one question against fixed labels through /v1/systemone

Weights: F32, F16, BF16, Q4_0, Q5_0, Q5_1, Q8_0, Q3_K-Q6_K and MXFP4. The full list, generated from the code, is docs/models.md; the vision guide covers pictures.

Performance

Every ratio is jitllm's rate over the other engine's, measured in the same pass on the same machine:

  • Up to 4.62x llama.cpp's prefill (Qwen3-Next-80B on four V100s), ahead on 34 of 35 models and faster decode on all 35, from 0.4B to 120B.
  • Up to 5.8x sooner to the first token than vLLM 0.18.1 and 1.26x its decode (Llama-3.1-8B, one V100), and up to 5.53x its batched decode.
  • Up to 3.72x llama.cpp's batched decode at 16 to 128 sequences (V100).
  • Up to 1.53x llama.cpp's decode on an Apple M4 (Metal), up to 1.67x its prefill on the M4's CPU.
  • Up to 1.67x llama.cpp's prefill on a 12-core Xeon, and up to 6x mistral.rs's decode.

The performance overview has every row with its hardware, backends and lengths, and where jitllm is still behind; the scoreboard has the commands and controls behind each.

Why Go?

Most inference engines are written in C, C++ or Rust, and some of the best work is. The choice here is practical:

  • It is the language the author knows best for writing a JIT: emitting machine code, mapping it executable and calling into it.
  • It is portable. One toolchain cross-compiles every target, and with no cgo there is no C toolchain or vendor SDK per target.
  • It compiles fast, so experiments are fast. Most of this engine was found by trying an idea and measuring it.
  • It is built for servers. Go's high-performance networking and synchronization let jitllmd serve many sessions at once, batch them in one step loop and time-slice them fairly, in one process with no framework. The server reaches the first token up to 5.8x sooner than vLLM on the same card, and a warm decode token allocates nothing, so the collector never touches the hot path.

The hot paths are generated machine code either way; Go carries the parts around them (paging, placement, scheduling).

Build

All you need is Go: Go 1.26 or newer, no C toolchain, no GPU SDK, no Python.

git clone https://github.com/jitllm/jitllm.git && cd jitllm
go build ./cmd/jitllm
(cd server && go build -o ../jitllmd ./cmd/jitllmd)
(cd ui && CGO_ENABLED=0 go build -o ../jitllm-desktop .)
(cd tui && go build -o ../jitllm-tui .)

Each is one static binary that finds CUDA, Vulkan or Metal at run time.

Built for agents

  • For an agent using jitllm: the OpenAI- and Anthropic-compatible APIs work with the SDKs and agent frameworks as they are; tool calls are read in each model's own format and tool_choice is enforced by constrained decoding, so a forced call is a call. jitllm capabilities and GET /v1/capabilities say what this build runs (architectures, quant types, backends, API fields and their refusals), generated from the code; jitllm doctor -json reports the host, devices and a model's compatibility, each problem with a next action; tested recipes go from install to a running server, a connected client, tools, conversion, placement and diagnosis; and llms.txt indexes the documentation.
  • For an agent changing jitllm: AGENTS.md is the one file that holds the project's binding rules, with a table that routes each subsystem to its evidence; the model list is generated from the converter and CI fails when it drifts; go run ./scripts/checks maps changed paths to the gates that cover them; and the model-free suite (go run ./scripts/modelfree) runs every check that needs no model or GPU, reporting a missing model or device as a skip, never as a pass.

Directories

Path Synopsis
cmd
jitllm command
Command jitllm is the jitllm CLI.
Command jitllm is the jitllm CLI.
Package convert turns a source model into a jlm container.
Package convert turns a source model into a jlm container.
gguf
Package gguf parses GGUF model files into a meta.File.
Package gguf parses GGUF model files into a meta.File.
hf
Package hf acquires GGUF weights from HuggingFace.
Package hf acquires GGUF weights from HuggingFace.
library
Package library is the list of models jitllm can fetch by name: for each, a Hugging Face repository, the file in it, and the vision tower beside it when there is one.
Package library is the list of models jitllm can fetch by name: for each, a Hugging Face repository, the file in it, and the vision tower beside it when there is one.
safetensors
Package safetensors is a parser for the safetensors container, and nothing else.
Package safetensors is a parser for the safetensors container, and nothing else.
dev
agenteval command
Command agenteval measures whether an agent can use and change jitllm: it sets up each task in a fresh workspace, hands the task's prompt to the agent command it is given, then checks the result by machine and writes a JSON report.
Command agenteval measures whether an agent can use and change jitllm: it sets up each task in a fresh workspace, hands the task's prompt to the agent command it is given, then checks the result by machine and writes a JSON report.
bench
Package bench is the measurement harness.
Package bench is the measurement harness.
decisionbench command
Command decisionbench reproduces the user's decision-model table against any server speaking TypeSafe's POST /v1/systemone -- jitllm's, llama-server's, `lev serve` -- so every engine in the table is measured by one harness (docs/design/decision-models.md, section 5).
Command decisionbench reproduces the user's decision-model table against any server speaking TypeSafe's POST /v1/systemone -- jitllm's, llama-server's, `lev serve` -- so every engine in the table is measured by one harness (docs/design/decision-models.md, section 5).
httpbench command
httpbench is the one HTTP load harness every engine is measured with: a closed-loop concurrency sweep of streamed OpenAI-compatible requests, two or more endpoints run as interleaved arms.
httpbench is the one HTTP load harness every engine is measured with: a closed-loop concurrency sweep of streamed OpenAI-compatible requests, two or more endpoints run as interleaved arms.
httpbench/hb
Package hb is the HTTP load harness behind dev/httpbench: one closed-loop concurrency sweep, measured the same way against every OpenAI-compatible engine, with two or more engines run as interleaved arms.
Package hb is the HTTP load harness behind dev/httpbench: one closed-loop concurrency sweep, measured the same way against every OpenAI-compatible engine, with two or more engines run as interleaved arms.
tools/fmtbench command
Command fmtbench prices each quant format's CPU decode matvec against a read wall measured in the same process, every shape sized DRAM-resident.
Command fmtbench prices each quant format's CPU decode matvec against a read wall measured in the same process, every shape sized DRAM-resident.
tools/mvbench command
Command mvbench measures the GPU decode matvec against a device read-bandwidth ceiling measured in the same process.
Command mvbench measures the GPU decode matvec against a device read-bandwidth ceiling measured in the same process.
tools/ptxdump command
Command ptxdump lowers named kernels to PTX so ptxas and cuobjdump can be pointed at them: the SASS is the only thing that says what a kernel actually issues (the MMA matvec turned out to be mostly address arithmetic).
Command ptxdump lowers named kernels to PTX so ptxas and cuobjdump can be pointed at them: the SASS is the only thing that says what a kernel actually issues (the MMA matvec turned out to be mostly address arithmetic).
tools/readbench command
readbench prices the jlm reader: parse, then page-in by four strategies.
readbench prices the jlm reader: parse, then page-in by four strategies.
tools/stridebench command
stridebench prices the access pattern the packed kernel actually makes.
stridebench prices the access pattern the packed kernel actually makes.
engine
grammar
Package grammar constrains generation to a context-free grammar: GBNF text (llama.cpp's grammar format) or a JSON Schema compiled to it, matched one code point at a time by a pushdown automaton and turned, at each step, into the set of vocabulary tokens that keep the output inside the grammar.
Package grammar constrains generation to a context-free grammar: GBNF text (llama.cpp's grammar format) or a JSON Schema compiled to it, matched one code point at a time by a pushdown automaton and turned, at each step, into the set of vocabulary tokens that keep the output inside the grammar.
model
Package model loads a jlm container and runs its transformer graph on generated kernels, on the host and on whichever devices take its blocks.
Package model loads a jlm container and runs its transformer graph on generated kernels, on the host and on whichever devices take its blocks.
nn
sched
Package sched runs work across the machine's performance cores.
Package sched runs work across the machine's performance cores.
format
jlm
Package jlm is jitllm's own model container: weights already in the device layout, one page per block, the tokenizer travelling with them.
Package jlm is jitllm's own model container: weights already in the device layout, one page per block, the tokenizer travelling with them.
quant
Package quant is the quantization formats themselves: the type enum, each format's block geometry, and the reference decoders.
Package quant is the quantization formats themselves: the type enum, each format's block geometry, and the reference decoders.
internal
capabilities
Package capabilities is the machine-readable answer to "does this build support my workload": the architectures, Hugging Face classes and projectors the converter takes, the input formats, the quantization types, the backends (implemented, and verified on which hardware, with the gate that is the evidence), the HTTP API's endpoints, request fields, limits and refused combinations, and the build's version and revision.
Package capabilities is the machine-readable answer to "does this build support my workload": the architectures, Hugging Face classes and projectors the converter takes, the input formats, the quantization types, the backends (implemented, and verified on which hardware, with the gate that is the evidence), the HTTP API's endpoints, request fields, limits and refused combinations, and the build's version and revision.
cmd/goheap
Package goheap is the process-wide collector setup both binaries share: cmd/jitllm and server/cmd/jitllmd.
Package goheap is the process-wide collector setup both binaries share: cmd/jitllm and server/cmd/jitllmd.
localserver
Package localserver is the one place every front end (the CLI, the window, the terminal app and jitllmd itself) learns where the local service is and where its models live: the default address, the per-system models folder, and the check "does a jitllmd answer here, and which version".
Package localserver is the one place every front end (the CLI, the window, the terminal app and jitllmd itself) learns where the local service is and where its models live: the default address, the per-system models folder, and the check "does a jitllmd answer here, and which version".
localserver/service
Package service installs jitllmd as the person's own background service, the way each system starts a per-user program at login: a systemd user unit on Linux, a launchd user agent on macOS, and on Windows the login script and Run key the installer (packaging/windows) writes.
Package service installs jitllmd as the person's own background service, the way each system starts a per-user program at login: a systemd user unit on Linux, a launchd user agent on macOS, and on Windows the login script and Run key the installer (packaging/windows) writes.
modelsdoc
Package modelsdoc renders docs/models.md, the supported-models list, from the converter's own tables: convert.Architectures (the GGUF names), convert.HFClasses (the Hugging Face classes), convert.Projectors (the vision projectors), quant's type lists and the download catalogue.
Package modelsdoc renders docs/models.md, the supported-models list, from the converter's own tables: convert.Architectures (the GGUF names), convert.HFClasses (the Hugging Face classes), convert.Projectors (the vision projectors), quant's type lists and the download catalogue.
oracle
Package oracle is the engine's arithmetic written in plain Go: the reference every generated kernel is gated against, and nothing else.
Package oracle is the engine's arithmetic written in plain Go: the reference every generated kernel is gated against, and nothing else.
schemacheck
Package schemacheck is the structured-output gates' JSON Schema validator, for the subset engine/grammar compiles: what an output is held to, written independently of the grammar that produced it.
Package schemacheck is the structured-output gates' JSON Schema validator, for the subset engine/grammar compiles: what an output is held to, written independently of the grammar that produced it.
testlock
Package testlock serialises GPU tests across processes.
Package testlock serialises GPU tests across processes.
testmodels
Package testmodels is where a test finds a model file.
Package testmodels is where a test finds a model file.
testmodels/fresh
Package fresh makes a test's container one this build reads.
Package fresh makes a test's container one this build reads.
jit
cpu
Package cpu emits x86-64 and arm64 machine code at load time.
Package cpu emits x86-64 and arm64 machine code at load time.
gpu/amdgpu
Package amdgpu lowers jit/gpu/ir kernels to LLVM IR text for AMD GPUs.
Package amdgpu lowers jit/gpu/ir kernels to LLVM IR text for AMD GPUs.
gpu/backend
Package backend puts one interface over the three GPU runtimes so a benchmark or a tuner can ask the same question of each.
Package backend puts one interface over the three GPU runtimes so a benchmark or a tuner can ask the same question of each.
gpu/ffi
Package ffi binds C functions without cgo, on top of goffi.
Package ffi binds C functions without cgo, on top of goffi.
gpu/hip
Package hip binds AMD's ROCm runtime (libamdhip64 on Linux, amdhip64_N.dll from the HIP SDK on Windows) and code object manager (libamd_comgr) through jit/gpu/ffi, with no cgo.
Package hip binds AMD's ROCm runtime (libamdhip64 on Linux, amdhip64_N.dll from the HIP SDK on Windows) and code object manager (libamd_comgr) through jit/gpu/ffi, with no cgo.
gpu/ir
Package ir is a very small typed kernel IR for NN compute.
Package ir is a very small typed kernel IR for NN compute.
gpu/kernels
Package kernels holds the NN kernels written against the IR.
Package kernels holds the NN kernels written against the IR.
gpu/metal
Package metal is Apple-only.
Package metal is Apple-only.
gpu/msl
Package msl lowers the IR to Metal Shading Language.
Package msl lowers the IR to Metal Shading Language.
gpu/ptx
Package ptx is an assembler for PTX text: registers, labels and a buffer.
Package ptx is an assembler for PTX text: registers, labels and a buffer.
gpu/spirv
Package spirv emits SPIR-V compute modules from the IR.
Package spirv emits SPIR-V compute modules from the IR.
gpu/tier
Package tier is jitllm's GPU decode tier: an nn.Device backed by the generated device kernels (jit/gpu).
Package tier is jitllm's GPU decode tier: an nn.Device backed by the generated device kernels (jit/gpu).
gpu/vulkan
Package vulkan runs SPIR-V compute kernels through the Vulkan loader.
Package vulkan runs SPIR-V compute kernels through the Vulkan loader.
scripts
checks command
Command checks says which checks a change needs, and runs them.
Command checks says which checks a change needs, and runs them.
genmodels command
Command genmodels writes docs/models.md, the supported-models list, from the converter's own tables (internal/modelsdoc):
Command genmodels writes docs/models.md, the supported-models list, from the converter's own tables (internal/modelsdoc):
gentok command
Command gentok generates tok/pretok/table.go: for every tokenizer.ggml.pre name a GGUF can carry, the ordered pipeline of typed stages that name denotes.
Command gentok generates tok/pretok/table.go: for every tokenizer.ggml.pre name a GGUF can carry, the ordered pipeline of typed stages that name denotes.
genunicode command
Command genunicode generates tok/unicodedata.go from llama.cpp's checked-in Unicode tables.
Command genunicode generates tok/unicodedata.go from llama.cpp's checked-in Unicode tables.
modelfree command
Command modelfree runs every test of the four modules on whatever host CI gives it, with no model download and no GPU, and writes what was SKIPPED into the job summary.
Command modelfree runs every test of the four modules on whatever host CI gives it, with no model download and no GPU, and writes what was SKIPPED into the job summary.
server module
tok
Package tok implements the SentencePiece tokenizer that llama-architecture GGUF models carry (tokenizer.ggml.model == "llama").
Package tok implements the SentencePiece tokenizer that llama-architecture GGUF models carry (tokenizer.ggml.model == "llama").
jinja
Package jinja implements a Jinja2-compatible template engine.
Package jinja implements a Jinja2-compatible template engine.
pretok
Package pretok is the pre-tokenizer model: the stages a tokenizer runs before merging, the parser that reads them out of a tokenizer.json, and the table generated from the artefacts themselves.
Package pretok is the pre-tokenizer model: the stages a tokenizer runs before merging, the parser that reads them out of a tokenizer.json, and the table generated from the artefacts themselves.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL