blisvllm

package module

v0.32.0 Latest Latest Go to latest Published: Oct 28, 2026 License: Apache-2.0 Imports: 2 Imported by: 0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/inference-sim/blis-vllm

Links

Open Source Insights

README ¶

blis-vllm

A Go model of the vLLM inference engine's scheduling, KV-cache, connector and metrics behaviour, for a discrete-event simulator. It implements the engine-neutral contract in blis-schemas (engine.Engine), so a simulator such as inference-sim builds it once and then holds only that interface. The design is inference-sim#2021.

One release per build. This build models vLLM v0.32.0 (5df63a7203cdb9328d824a27b682e77c7b9928be); see provenance.go. A blis-vllm release is tagged to the vLLM release it models, such as v0.32.0-blis.1.
Checked against vLLM itself. CI feeds the same inputs to vLLM's own classes (Python, CPU only, the GPU mocked) and to this module, and compares every step. See docs/testing.md and the feature matrix.
Time is a value. Nothing here reads a clock or starts a goroutine. The simulator passes the simulated instant into every call.
Rules packs. The per-release validation rules (rules/, its own Go module so the latency kernel can import it without importing the engine) moved here from blis-schemas.

Layout

Path	What
`provenance.go`	the module's identity: the vLLM release it models and its own revision
`rules/`	rules packs per vLLM release (separate module)
`core/`	the vLLM v0.32.0 engine, diff-tested against vLLM's own classes (`core/vllm`: behind blis-schemas `engine.Engine`, built by the `vllm` Factory)
`compat/`	the compatibility engine: inference-sim's engine as it ran before the extraction (`compat/engine`, `compat/kv`, `compat/lora`, `compat/kernelmodel`, moved verbatim, each file stating its source commit), which keeps the existing 0.29.0/0.31.0 scenarios byte-identical until they are re-baselined to 0.32.0 (inference-sim#2027), plus the leaf packages it uses and `compat/build`, the engine-side configuration the simulator used to decode itself (the scenario's KV offload block and the `--kv-offload-config` file; inference-sim#2021 E3)
`testdata/`	inputs of `compat/kernelmodel`'s tests: copies of inference-sim's vendored catalog, registry and scenarios (`testdata/README.md`)
`docs/feature-matrix.md`	every feature our sources use, its v0.32.0 citation, owner, oracle tier and status
`docs/testing.md`	the oracle tiers, the shared GPU mock, provenance, golden-trace policy

Licence

Apache-2.0. Files translated from vLLM keep vLLM's licence and name the upstream file and commit they were derived from; see NOTICE.

Documentation ¶

Overview ¶

Package blisvllm is a Go model of the vLLM inference engine's scheduling, KV-cache, connector and metrics behaviour, for a discrete-event simulator to call through the engine-neutral contract in github.com/inference-sim/blis-schemas/engine.

One build of this module models one vLLM release, Upstream. Its behaviour is checked against that release's own code in CI (see docs/testing.md): the same inputs, fed to the real vLLM classes in Python and to this module in Go, must make the same decisions step by step.

The simulator owns time. Nothing in this module reads a clock or starts a goroutine: every call carries the simulated instant as a value, and where a vLLM rule makes work wait until a later instant, the engine reports that instant through NextWake.

Index ¶

Constants
Variables
type Provenance
- func ProvenanceFor(behaviour string) Provenance
type UpstreamRelease

Constants ¶

View Source

const Kind = "vllm"

Kind is the engine kind this module registers with blis-schemas' engine factory.

View Source

const ModulePath = "github.com/inference-sim/blis-vllm"

ModulePath is this module's import path.

Variables ¶

View Source

var Upstream = UpstreamRelease{
	Version: "0.32.0",
	Tag:     "v0.32.0",
	Commit:  "5df63a7203cdb9328d824a27b682e77c7b9928be",
}

Upstream identifies the vLLM release this build models.

Functions ¶

This section is empty.

Types ¶

type Provenance ¶

type Provenance struct {
	EngineKind      string `json:"engine_kind" yaml:"engine_kind"`
	UpstreamVersion string `json:"upstream_version" yaml:"upstream_version"`
	UpstreamCommit  string `json:"upstream_commit" yaml:"upstream_commit"`
	// Module is this module's version as the build recorded it: a release tag, or a
	// pseudo-version naming the commit; "(devel)" in a build of this repository itself.
	Module string `json:"module" yaml:"module"`
	// Behaviour names the behaviour the engine executes: "vllm-0.32.0", or a compatibility
	// behaviour a scenario selected (see Compat).
	Behaviour string `json:"behaviour" yaml:"behaviour"`
}

Provenance is what a run must record to be reproduced: the engine kind, the upstream release it models and this module's own revision. A simulator writes it into its results beside the kernel's provenance.

func ProvenanceFor ¶

func ProvenanceFor(behaviour string) Provenance

ProvenanceFor returns the provenance of an engine executing behaviour.

type UpstreamRelease ¶

type UpstreamRelease struct {
	Version string
	Tag     string
	Commit  string
}

UpstreamRelease is one vLLM release.

Source Files ¶

View all Source files

provenance.go

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
metricsmanifest command Command metricsmanifest lists every vllm:* metric family a vLLM release defines, from the release's own source, and writes it as docs/metrics-manifest.json with each family's class: exact (blis-vllm computes it from the decisions it makes), derived (it needs inputs the simulator supplies, such as time or transfer outcomes, and is reported given them) or unavailable (it needs real hardware, a process or a feature outside the simulation).	Command metricsmanifest lists every vllm:* metric family a vLLM release defines, from the release's own source, and writes it as docs/metrics-manifest.json with each family's class: exact (blis-vllm computes it from the decisions it makes), derived (it needs inputs the simulator supplies, such as time or transfer outcomes, and is reported given them) or unavailable (it needs real hardware, a process or a feature outside the simulation).
provenance command Command provenance prints, as JSON, the Go side of a CI run's provenance: blis-vllm's identity as the build records it (blisvllm.ProvenanceFor), the commit the run checks, the Go toolchain and the platform.	Command provenance prints, as JSON, the Go side of a CI run's provenance: blis-vllm's identity as the build records it (blisvllm.ProvenanceFor), the commit the run checks, the Go toolchain and the platform.
compat
build Package build is the compatibility engine's configuration: it decodes and resolves the engine-side blocks of a scenario (its KV offload block, its engine and pools) and the command-line restatements of them, so the simulator hands them over unread (#2021 B2, B6).	Package build is the compatibility engine's configuration: it decodes and resolves the engine-side blocks of a scenario (its KV offload block, its engine and pools) and the command-line restatements of them, so the simulator hands them over unread (#2021 B2, B6).
contract Package contract puts the compatibility engine (compat/engine) behind blis-schemas' engine.Engine, the contract blis-vllm's v0.32.0 engine (core/vllm) implements, so a simulator drives every release through one driver (inference-sim GOAL.md phase 2c).	Package contract puts the compatibility engine (compat/engine) behind blis-schemas' engine.Engine, the contract blis-vllm's v0.32.0 engine (core/vllm) implements, so a simulator drives every release through one driver (inference-sim GOAL.md phase 2c).
dplb Package dplb is the compatibility engine's model of vLLM's internal data-parallel load balancer, the front-end rule that sends a request reaching a data-parallel replica's endpoint to one of its engine ranks.	Package dplb is the compatibility engine's model of vLLM's internal data-parallel load balancer, the front-end rule that sends a request reaching a data-parallel replica's endpoint to one of its engine ranks.
engine A data-parallel wave's continuation rule: whether a lockstep group runs another step.	A data-parallel wave's continuation rule: whether a lockstep group runs another step.
internal/util Package util provides generic utility functions shared across sim/ sub-packages.	Package util provides generic utility functions shared across sim/ sub-packages.
kernelmodel capacity.go derives the KV-block budget from the kernel's own memory methods.	capacity.go derives the KV-block budget from the kernel's own memory methods.
kv Package kv implements block-based KV cache management for the BLIS simulator.	Package kv implements block-based KV cache management for the BLIS simulator.
kvkey Package kvkey is the single gated home for KV cache block keying: deriving hierarchical content keys at a chosen stride, and interning those keys to compact integer handles.	Package kvkey is the single gated home for KV cache block keying: deriving hierarchical content keys at a chosen stride, and interning those keys to compact integer handles.
kvtransfer Package kvtransfer models vLLM's KV-offload I/O thread pool as a bounded priority queueing station with N servers per storage tier.	Package kvtransfer models vLLM's KV-offload I/O thread pool as a bounded priority queueing station with N servers per storage tier.
lora Package lora implements the LoRA (Low-Rank Adaptation) control-plane subsystem for BLIS: adapter identity, a pre-declared adapter registry (id → rank), a per-instance resident-adapter set with LRU eviction, and the three Digital-Twin adapter cost terms (cold-load latency, per-step compute overhead, HBM footprint).	Package lora implements the LoRA (Low-Rank Adaptation) control-plane subsystem for BLIS: adapter identity, a pre-declared adapter registry (id → rank), a per-instance resident-adapter set with LRU eviction, and the three Digital-Twin adapter cost terms (cold-load latency, per-step compute overhead, HBM footprint).
nixl Package nixl is the compatibility engine's model of vLLM's NIXL connector rules that a P/D simulator's cluster applies around a KV handoff: how long a prefill engine holds a finished request's KV for the decode side's pull (#1934, INV-PD-7).	Package nixl is the compatibility engine's model of vLLM's NIXL connector rules that a P/D simulator's cluster applies around a KV handoff: how long a prefill engine holds a finished request's KV for the decode side's pull (#1934, INV-PD-7).
prefix Package hash provides SHA256 hashing utilities for KV cache prefix matching and routing prefix affinity.	Package hash provides SHA256 hashing utilities for KV cache prefix matching and routing prefix affinity.
tokenid Package tokenid defines the simulator's compact token-ID type.	Package tokenid defines the simulator's compact token-ID type.
vllmsettings Package vllmsettings converts a vLLM engine block into the engine settings the kernel prices (blis-latency-kernel/enginesettings).	Package vllmsettings converts a vLLM engine block into the engine settings the kernel prices (blis-latency-kernel/enginesettings).
core
connector/nixl Package nixl is the scheduler side of vLLM's NIXL connector in its pull mode (NixlConnector = NixlPullConnector, vllm/distributed/kv_transfer/kv_connector/factory.py: 177-189@5df63a72): what the engine decides about a P/D request.	Package nixl is the scheduler side of vLLM's NIXL connector in its pull mode (NixlConnector = NixlPullConnector, vllm/distributed/kv_transfer/kv_connector/factory.py: 177-189@5df63a72): what the engine decides about a P/D request.
engine Package engine is vLLM's EngineCore over core/sched: the batch queue that keeps up to max_concurrent_batches batches in flight (step_with_batch_queue), the plain step() for a queue of one, and the data-parallel wave bookkeeping (DPEngineCoreProc's step counter, sync interval and prefill throttling).	Package engine is vLLM's EngineCore over core/sched: the batch queue that keeps up to max_concurrent_batches batches in flight (step_with_batch_queue), the plain step() for a queue of one, and the data-parallel wave bookkeeping (DPEngineCoreProc's step counter, sync interval and prefill throttling).
kv Package kv is vLLM's KV-cache bookkeeping: the block pool, its free queue, prefix-cache block hashing and the per-request block tables, for full-attention models.	Package kv is vLLM's KV-cache bookkeeping: the block pool, its free queue, prefix-cache block hashing and the per-request block tables, for full-attention models.
metrics Package metrics is vLLM's Prometheus stat logger for the vllm:* families whose values the scheduler alone decides (class "exact" in docs/metrics-manifest.json): request counts, KV usage, prefix-cache queries and hits, preemptions, prompt and generation tokens, finished requests by reason, and tokens per step.	Package metrics is vLLM's Prometheus stat logger for the vllm:* families whose values the scheduler alone decides (class "exact" in docs/metrics-manifest.json): request counts, KV usage, prefix-cache queries and hits, preemptions, prompt and generation tokens, finished requests by reason, and tokens per step.
sched Package sched is vLLM's v1 Scheduler: schedule() and update_from_output() for a full-attention model, with chunked prefill, the long-prefill cap, FCFS and priority policies, recompute preemption, prefix caching, speculative decoding (acceptance supplied) and the max_loras batch limit, and the AsyncScheduler's output placeholders and pipeline-parallel decode spacing (Config.Async).	Package sched is vLLM's v1 Scheduler: schedule() and update_from_output() for a full-attention model, with chunked prefill, the long-prefill cap, FCFS and priority policies, recompute preemption, prefix caching, speculative decoding (acceptance supplied) and the max_loras batch limit, and the AsyncScheduler's output placeholders and pipeline-parallel decode spacing (Config.Async).
vllm Package vllm is vLLM v0.32.0's engine behind blis-schemas' engine.Engine: core/sched's Scheduler, core/engine's batch queue and core/metrics' stat logger, driven by a simulator through the engine-neutral contract instead of by vLLM's EngineCore loop.	Package vllm is vLLM v0.32.0's engine behind blis-schemas' engine.Engine: core/sched's Scheduler, core/engine's batch queue and core/metrics' stat logger, driven by a simulator through the engine-neutral contract instead of by vLLM's EngineCore loop.
kerneltests
cmd/bandprobe command Command bandprobe prices one step per line of a CSV and prints both band edges.	Command bandprobe prices one step per line of a CSV and prints both band edges.
cmd/overlap-probe command Command overlap-probe answers one question: does the per-layer overlap term change the SHAPE of the concurrency response, or only its level?	Command overlap-probe answers one question: does the per-layer overlap term change the SHAPE of the concurrency response, or only its level?
cmd/score command Command score reports how well the kernel predicts published measurements.	Command score reports how well the kernel predicts published measurements.
cmd/shape command Command shape scores this kernel's concurrency response against NVIDIA's end-to-end accuracy snapshot, along AISimulate's own published error on the same points.	Command shape scores this kernel's concurrency response against NVIDIA's end-to-end accuracy snapshot, along AISimulate's own published error on the same points.
cmd/worked-table command Command worked-table prints a worked step-time table, computed by the kernel itself from the committed catalog and registry.	Command worked-table prints a worked step-time table, computed by the kernel itself from the committed catalog and registry.
internal/artifacttest Package artifacttest holds the rule every test package uses to decide whether a failure to load a catalog or registry artifact is a skip or a failure.	Package artifacttest holds the rule every test package uses to decide whether a failure to load a catalog or registry artifact is a skip or a failure.
internal/docgen command Command docgen writes the generated parts of the documentation into docs/generated/:	Command docgen writes the generated parts of the documentation into docs/generated/:
internal/harness Package harness holds what this repository's scoring and reporting commands share: opening a kernel from a scenario file, and building the batch shapes a comparison prices.	Package harness holds what this repository's scoring and reporting commands share: opening a kernel from a scenario file, and building the batch shapes a comparison prices.
registry Package registry holds the vLLM entries of the KV-connector registry (blis-schemas spec/connector, inference-sim/inference-sim#2021 B10): which connector names a deployment may use, and which engine model in this module reproduces each.	Package registry holds the vLLM entries of the KV-connector registry (blis-schemas spec/connector, inference-sim/inference-sim#2021 B10): which connector names a deployment may use, and which engine model in this module reproduces each.
rules Package rules holds the vLLM rules packs: one per vLLM release, each the release's accepted values, its defaults and its version-scoped checks, implementing blis-schemas' rules.EngineRules.	Package rules holds the vLLM rules packs: one per vLLM release, each the release's accepted values, its defaults and its version-scoped checks, implementing blis-schemas' rules.EngineRules.
engineconfig Package engineconfig is vLLM's engine configuration, the keys a deployment pool's engine: block states for a vLLM release: the Engine struct, its YAML decoding (Decode, Encode), its defaults and its release-independent field checks (Validate).	Package engineconfig is vLLM's engine configuration, the keys a deployment pool's engine: block states for a vLLM release: the Engine struct, its YAML decoding (Decode, Encode), its defaults and its release-independent field checks (Validate).
pack Package pack is the shape of a vLLM rules pack: one release's accepted values, its defaults and its version-scoped checks, as data (Pack), and the behaviour a consumer resolves against it.	Package pack is the shape of a vLLM rules pack: one release's accepted values, its defaults and its version-scoped checks, as data (Pack), and the behaviour a consumer resolves against it.
v0_29 Package v0_29 is the rules pack for vLLM 0.29.x.	Package v0_29 is the rules pack for vLLM 0.29.x.
v0_31 Package v0_31 is the rules pack for vLLM 0.31.0 (tag v0.31.0, commit db9527a468), the release blis-latency-kernel encodes.	Package v0_31 is the rules pack for vLLM 0.31.0 (tag v0.31.0, commit db9527a468), the release blis-latency-kernel encodes.
v0_32 Package v0_32 is the rules pack for vLLM 0.32.0 (tag v0.32.0, commit 5df63a72), the release blis-vllm executes.	Package v0_32 is the rules pack for vLLM 0.32.0 (tag v0.32.0, commit 5df63a72), the release blis-vllm executes.
vllmengine Package vllmengine is how a simulator gets blis-vllm's vLLM v0.32.0 engine (core/vllm) without naming any of its packages: Register makes Kind "vllm" available in blis-schemas' engine registry, built from nothing but an engine.Spec, and the simulator drives it through engine.Engine.	Package vllmengine is how a simulator gets blis-vllm's vLLM v0.32.0 engine (core/vllm) without naming any of its packages: Register makes Kind "vllm" available in blis-schemas' engine registry, built from nothing but an engine.Spec, and the simulator drives it through engine.Engine.