blisvllm

package module
v0.32.0-blis.3 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: Apache-2.0 Imports: 3 Imported by: 0

README

blis-vllm

A Go model of the vLLM inference engine's scheduling, KV-cache, connector and metrics behaviour, for a discrete-event simulator. It implements the engine-neutral contract in blis-substrate contracts/ (engine.Engine), so a simulator such as inference-sim builds it once and then holds only that interface. The design is inference-sim#2021.

  • One release per build. This build models vLLM v0.32.0 (5df63a7203cdb9328d824a27b682e77c7b9928be); see provenance.go. Its releases are tagged to the vLLM release they model, v0.32.0-blis.N (design §14.1); the inference-sim release that runs each one pins it in its go.mod.
  • One-way dependencies. blis-substrate ← blis-vllm ← inference-sim. blis-vllm imports blis-substrate's contracts/ packages and, until the engine boundary moves (release plan §3.3, M2), its cost model (kernel/); the cost model imports nothing from this module, and inference-sim's simulator imports blis-vllm at one build site, cmd/engine_build.go.
  • Checked against vLLM itself. CI feeds the same inputs to vLLM's own classes (Python, CPU only, the GPU mocked) and to this module, and compares every step. See docs/testing.md and the feature matrix.
  • Time is a value. Nothing here reads a clock or starts a goroutine. The simulator passes the simulated instant into every call.
  • Rules packs. The per-release validation rules (rules/) moved here from blis-schemas. A caller hands them to the cost model through blis-substrate's contracts/rules.EngineRules, so the cost model imports nothing from this module.

Layout

Path What
provenance.go the module's identity: the vLLM release it models and its own revision
rules/ rules packs per vLLM release, and the vLLM engine block's configuration (rules/engineconfig)
core/ the vLLM v0.32.0 engine, diff-tested against vLLM's own classes (core/vllm: behind blis-substrate's engine.Engine, built by the vllm Factory)
vllmengine/ how a simulator gets the v0.32.0 engine: it registers the Factory, prices steps with the latency kernel and sizes the KV cache
compat/ the compatibility engine: inference-sim's engine as it ran before the extraction (compat/engine, compat/kv, compat/lora, compat/kernelmodel, moved verbatim, each file stating its source commit), which keeps the existing 0.29.0/0.31.0 scenarios byte-identical until they are re-baselined to 0.32.0 (inference-sim#2027), plus the leaf packages it uses and compat/build, the engine-side configuration the simulator used to decode itself (the scenario's KV offload block and the --kv-offload-config file; inference-sim#2021 E3)
kerneltests/ the latency kernel's tests that open a kernel from a vLLM engine block, its commands and docs generator, moved here from blis-latency-kernel (kerneltests/_inpackage runs inside blis-substrate's kernel package: kerneltests/scripts/run-inpackage.sh). The catalog and registry they read are blis-substrate's, at the release go.mod pins
testdata/ inputs of compat/kernelmodel's tests: inference-sim's scenarios (testdata/README.md); the catalog and registry are the pinned blis-substrate's, through its loader
docs/feature-matrix.md every feature our sources use, its v0.32.0 citation, owner, oracle tier and status
docs/testing.md the oracle tiers, the shared GPU mock, provenance, golden-trace policy
docs/multiconnector.md the MultiConnector model: the choice rule, what is saved to which child, how it is tested, what it refuses

The field list (fields.json)

fields.json lists every configuration field and environment variable vLLM v0.32.0 reads, each with a status, its evidence and an estimation rule (design VLM-3, §11.2). The type is blis-substrate's contracts/spec/fields; accuracy-dashboard and blis-config-search read the list through inference-sim fields, and hold no list of their own (COV-3, CFG-3).

  • Names come from vLLM's own source at the pinned commit, not by hand: fields/gen_fields.py reads the EngineArgs, AsyncEngineArgs and front-end flag fields (expanding each config object to <field>.<subfield>), every key of vllm/envs.py's environment_variables, and every environment variable vLLM reads by a literal name: os.environ.get, os.getenv, os.environ[...], in os.environ, setdefault, envs.is_set and envs.X under vllm/, getenv in csrc/, and env::var in the Rust front end under rust/, with each read's file and line (--env-reads). It also lists the NCCL, NVSHMEM and UCX variables vLLM sets for those libraries. The generator's docstring states the read rule and the paths it skips, each a separate command or a build step (vllm collect-env, vllm bench, vllm chat, tests, build scripts); a module named for another platform is not skipped. A read whose name is not a literal is listed in status.json's dynamic_reads, with the reason. CI runs python3 -I fields/gen_fields.py vllm-source fields/status.json --check fields.json, which fails while a name vLLM reads has no entry, an entry names a field vLLM does not have, or a dynamic read is unlisted.

  • A configuration field with no command-line flag is not listed. The generator takes a field of EngineArgs or AsyncEngineArgs only if some add_argument call names it. Six such fields have none: model_weights, _api_process_count, _api_process_rank, enable_mm_processor_stats, mamba_config and tokens_only. vLLM's own code sets them, or they are reached through another flag. enable_mm_processor_stats is an ObservabilityConfig flag that its source calls "for internal use only (e.g., benchmarks)" and "not exposed as a CLI argument" (vllm/config/observability.py:94), so a deployment cannot set it. If a later release adds a flag for it, the generator lists it and the completeness check demands a status.

  • The Rust front end is scanned because VLLM_USE_RUST_FRONTEND=1 selects it on any platform, CUDA included (vllm/entrypoints/cli/serve.py, vllm/envs.py), and it runs in the serving process. Eighteen variables are read there; the ones read nowhere else are RUST_LOG, TOKIO_WORKER_THREADS, the VLLM_RS_* thread and tokenizer settings, and the Hugging Face cache, offline and token variables, which also change what a run loads.

  • Status, evidence and estimation are in fields/status.json, edited by hand. A field is modelled only with VLM-3's evidence: an oracle trace that sets it and that the port reproduces, or a test that sets it and checks the result against vLLM's own. A field the port does not read, or reads without such evidence, is not_modelled and cites an open issue (X28). The first version was seeded from accuracy-dashboard's vllm_arg_policy.yaml, as release plan §3.3 says: "Fields that table marked inert are entered as 'inert, unproven'." Estimation rules start from accuracy-dashboard discussion #33. leave_unset means the engine resolves the upstream's default (X19), so a rule is leave_unset only where blis-vllm resolves it for every value: max_num_seqs and max_num_batched_tokens, by ResolveBatch in core/vllm/batchdefaults.go, and gpu_memory_utilization, whose default 0.92 resolveEngine in compat/kernelmodel/engine_settings.go fills in for every release the engine models. gpu_memory_utilization is modelled, and its rule stays leave_unset for a run that did not record it. vLLM takes ceil(total memory x utilization) as the memory it may use (request_memory, vllm/v1/worker/utils.py:716-726@5df63a72), subtracts what the profile run finds is not KV cache (weights, non-torch memory and the peak activation) and the CUDA graph estimate (Worker.determine_available_memory, vllm/v1/worker/gpu_worker.py:733-745; VLLM_MEMORY_PROFILER_ESTIMATE_CUDAGRAPHS defaults on), and cuts the rest into whole pages (get_kv_cache_config_from_groups, vllm/v1/core/kv_cache_utils.py:1588). KVMemory (compat/kernelmodel/capacity.go) takes the same budget and subtracts the kernel's per-rank fixed occupancy (weights, activation peak, CUDA graph, EPLB replicas and communication buffers), so a stated value moves the pool by exactly the memory it adds. The evidence is compat/build/memory_utilization_test.go (0.95 gives more blocks than 0.9, by that rule, and neither is the default's) and compat/kernelmodel/capacity_test.go. What the evidence does not show: the fixed occupancy is the kernel's estimate, not vLLM's profile run, and on the one recorded case (DeepSeek-V4.1-Flash, tp 8, utilization 0.80) the pool is 47.56 GiB against vLLM's logged 61.18 GiB (blis-vllm#73 tracks closing that). The field's value classes are one class, since no comparison of the field with a literal is found in the source (the budget is arithmetic). kv_cache_memory_bytes, which overrides the utilization in vLLM, is refused when stated (blis-vllm#81). block_size is none: the engine resolves an unstated one only for the models, GPUs and KV dtypes in oracle/blocksize/TABLE.tsv (ResolveBlockSize in core/vllm/blocksize.go) and refuses the rest, citing blis-vllm#68, and a field has one estimation rule, which cannot say that. A field's estimation rule applies when the run did not record the value. It never replaces a value the run recorded: a stated value is carried into the scenario as stated, whatever the field's status. A not-modelled field that the engine reads and prices when it is stated is a status to correct (modelled, with its evidence, or refused when stated, VLM-3); the rule does not stand in for that. The v0.32.0 engine refuses a not-modelled key of the engine block that nothing reads when it is stated away from vLLM's default (TestEveryNotModelledFieldIsRefusedOrExplained walks fields.json and gives each refusal a partner that states the default and builds), naming the field and citing its issue. The keys the latency kernel reads and prices at vLLM v0.31.0's policy (blis-vllm#74 to #80, #83, #8) are the walk's allow-list.

    Those fields stay not_modelled in M2, and the list is the M3 scope: the kernel accepts and prices a stated value of each at its vLLM v0.31.0 policy, and updating that policy to v0.32.0's is M3 (blis-vllm#78 holds the date), four fields at a time. cudagraph_mode, cudagraph_capture_sizes and gpu_memory_utilization are the exceptions: they are modelled, with the divergence disclosed in the engine's assumptions (for gpu_memory_utilization, the fixed occupancy above). A point that states one of the others is priced at the kernel's v0.31.0 policy, and its substitution list should say so. The fields, with the kernel site that prices each: all2all_backend (kernel.go:798), disable_custom_all_reduce (layout.go:208), disable_cascade_attn (new.go:2868), mamba_cache_dtype (new.go:270), moe_backend (step_terms.go:393), enable_dbo (new.go:708), enable_eplb and eplb_config.num_redundant_experts (new.go:1353), the three engram_config keys (draft.go:432), dcp_comm_backend (new.go:550), dcp_q_replicate (new.go:533), cp_kv_cache_interleave_size (dcp.go:88) and attention_config.use_trtllm_attention (settings_priced.go:203, which prices true on one lane and refuses false). The sites are in blis-substrate's kernel package (layout.go and dcp.go are in its resolve package); they were read at the pinned layer and are not diffed against v0.32.0.

    Every not_modelled field with the rule leave_unset is named below, with what a deployment that states it gets, so leaving it unset gives vLLM's default (TestReadmeNamesEveryLeaveUnsetField fails when this list and fields.json differ):

    • not a key of the engine block, so a deployment that states it is refused when the block is decoded (an unknown key), and a run that does not state it runs vLLM's default: attention_backend, attention_config.backend (blis-vllm#80), attention_config.indexer_sparse_logits (#84), api_server_count (#105; the engine's data-parallel group is one client, which is vLLM's internal-load-balancing count for every value recorded runs state, so a scorer drops a recorded value and leaves it unset), kv_offloading_backend (#87) and the environment variable VLLM_USE_SIMPLE_KV_OFFLOAD (#87, refused by the block's env);
    • a deployment field, not a key of the engine block, that the engine reads from the deployment: data_parallel_external_lb (#86) and data_parallel_hybrid_lb (#64), both stated as the pool's parallel.data_parallel_load_balancing, and data_parallel_size_local (#55), stated as parallel.dp_local and refused when its packing differs from the engine's;
    • environment variables that change what a run does but not what the cost model prices, where an unset variable gives the same engine path for the values recorded runs state (the harness that writes the stand-in checks that, and names the field and the recorded value on the point): VLLM_FLASHINFER_ALLREDUCE_BACKEND (#80; mnnvl stated across nodes on compute capability 10.0 and up is what vLLM resolves auto to; stated on one node it is not, because auto there allows a trtllm fallback and a stated mnnvl does not, flashinfer_all_reduce.py:143-178@5df63a72, and every tensor-parallel all-reduce is priced from the NCCL sweep), UCX_RCACHE_MAX_UNRELEASED (#62; vLLM sets 1024 itself, nixl_utils.py:20-36@5df63a72) and UCX_TLS (#62; the engine prices a handoff between nodes over the NIC unless UCX_CUDA_IPC_ENABLE_MNNVL=y is stated, which is the path a list of RDMA transports without cuda_ipc takes). UCX_CUDA_IPC_ENABLE_MNNVL is read by the engine (compat/kernelmodel): y prices a handoff between nodes of one NVLink rack over NVLink and names the unchecked VMM registration as an assumption; anything else prices it over the NIC and says so;
    • read by the latency kernel, which prices a stated value at vLLM v0.31.0's policy: engram_config.cpu_offload (#78), where leave_unset fills only a value the run did not record; The two CUDA-graph switches cudagraph_decoder_replay (decoder-replay graphs, blis-vllm#80) and cudagraph_specialize_lora (blis-vllm#78) are not_modelled with none. attention_config.indexer_kv_dtype is modelled: the engine reads it and prices the indexer's key cache. On DeepSeek V4 and V4.1 auto and fp8 give the 132-byte record, as vLLM resolves them, and bf16, nvfp4 and mxfp4 are refused with vLLM's reason or the kernel's (mxfp4's 68-byte record is not what the graph prices); on MiniMax-M3 auto and bf16 size the indexer cache at two bytes a key value and fp8 or fp8_e4m3 at one (vLLM stores the keys as float8_e4m3fn with no scale), the latency kernel pricing the same record, and any other value is refused; a model with no indexer cache refuses every value but auto (blis-vllm#84). The covered class is auto and fp8. speculative_config.synthetic_acceptance_length is modelled (blis-vllm#88): with rejection_sample_method: synthetic it is the run's supplied acceptance length, in [1, num_spec_tokens + 1], taken at the rank of the scenario's pin; synthetic without it is refused. speculative_config.rejection_sample_method and env.VLLM_USE_RUST_FRONTEND are modelled, and env.PYTHONUNBUFFERED is proven_inert, for the DeepSeek-V4.1-Flash H200 rows of InferenceX (blis-vllm#106, #85): what each does in vLLM, what the engine does and what stays an assumption (the Rust front end's admission rate) are in docs/h200-v033-fields.md.
  • Value classes (VLM-3) are in fields/value_classes.json, generated from the same pinned source by fields/gen_classes.py for every modelled and proven_inert field: for a boolean or enumerated field the values (a value some comparison singles out is a class, the rest are one class); for a numeric field the domain (the field's own ge/gt/le/lt) cut at every number the source compares it with. Each class cites the file and line of the comparisons that form it and lists the traces whose scenario states a value in it (covered_by; a key a scenario leaves out runs the runner's default and is not counted). The generator finds a comparison where the field, or a name the same function assigned from it, meets a literal; it does not follow a value passed to another function, which fields/classes_extra.json covers one site at a time with a reason, and the file's limits says so. fields/test_classes.py fails when a modelled field has no classes, a class is backed by no comparison, or a cited line no longer holds it; TestEveryModelledFieldRecordsItsValueClasses fails the same way without the source. fields.json carries the classes themselves (the substrate's loader refuses a modelled or proven-inert field without them): fields/gen_fields.py --value-classes fields/value_classes.json writes them there. fields/value_classes.json stays as the generated basis (the comparisons and traces behind each class), which the tests read, and TestEveryModelledFieldRecordsItsValueClasses fails if the two disagree.

  • Value classes under-approximate. A class edge is a comparison of the field with a literal that the generator finds in the pinned source. It does not find a field read by name through a helper that takes the name as a string (_scheduler_config_int(vllm_config, "max_num_batched_tokens", 0), vllm/models/deepseek_v4/common/ops/cache_utils.py:812-814@5df63a72), a comparison with another field (listed under relations, 25 of them for max_num_batched_tokens), or a threshold made by min, range or arithmetic (max_cudagraph_capture_size also depends on min(max_num_tokens, ...), vllm/config/vllm.py:2446@5df63a72). So a field with one class, max_num_seqs for one, is a field no scanned comparison singles out, not a field the engine treats alike at every value, and a trace that covers a class shows the port matches vLLM at the values that trace states, not at every value of the class. The limits text of fields/value_classes.json says the same.

  • Inertness proofs (design §7.4, tier T1) are oracle/inertness/<field>.json, written by oracle/inertness.py and named by the field's proof. The harness varies the field over one value per value class and runs vLLM's own code: the real Scheduler on oracle scenarios (the scenario's scheduler_config states the probe), EngineArgs.create_engine_config() (every field of the resolved scheduler, cache, parallel and compilation config is compared across the probes except the field itself), and for a front-end field the real OutputProcessor (the requests it hands back to the engine core to abort are compared). A field is inert only if every run's decisions to the engine are identical across the probes and every Python read of the field (found by fields/gen_classes.py --reads, plus reviewed reads the scan cannot follow) has a disposition that holds: a line executed in the named run (coverage is kept per run, and matches a file by its path inside vLLM so it counts the installed wheel CI runs as well as a checkout), or a keyword passed on to a callee that has its own. The other cases §7.4 admits (a surface the engine refuses by name, a platform check false on CUDA) are not accepted by the harness yet; a proof naming one fails as an unknown disposition. A front-end effect is recorded (frontend_effect), not hidden. The read scan is of Python. A proof may list named_in, every file of vllm/ and rust/ that names the field, and the check fails when the tree names it in a file the proof does not list; that is how the Rust front end's reads of stream_interval (rust/src/cmd/src/cli.rs:277, rust/src/llm/src/lib.rs:97-102, rust/src/engine-core-client/src/client/output_channel.rs) are listed and reviewed without being run, and how the proof shows no file of the engine names it. A variable no file of the serving process names has a proof of kind unread. That is a reviewed static fact ("argument": "reviewed_static_fact"), not a run: the scan shows no file names the variable, and what the proof says about why it cannot matter is an argument the harness does not check beyond what its refusal_tests name. The three MC_ variables belong to the Mooncake transfer-engine library, which vLLM loads only through a Mooncake connector; a deployment selects one by connector name or through NIXL's kv_connector_extra_config backends, and the Go tests the proofs name (TestAMooncakeConnectorNameIsRefused, TestTheConnectorBlockCannotStateWhichBackendNixlLoads, TestNoMooncakeConnectorIsServed) pin that the engine refuses both. TestEveryProvenInertFieldHasAPassingProof holds the list to the files, to the value classes (the probes reach every class), to the engine's Go code (no non-test file but the listed ones names the field) and to the refusal tests (each still exists). oracle/test_inertness.py (about 5 minutes with the pinned source) shows the harness failing a field that changes the schedule (enable_chunked_prefill off), a config run and an output-processor run whose engine-facing result changes, a read it was not told about, a stale read, a run that did not execute the line it is credited with (including when an earlier run did), a file that names the field and is not listed, and a variable the source names. stream_interval is proven inert for the engine only. The simulator's delivery stage does not read it yet (inference-sim#2035 is open), so a recorded point that states 20 or 32 is simulated without the grouping its client saw, and the field is not G1-eligible until that stage reads it.

  • Narrower issues. Part of what some not_modelled fields need is now its own issue; the fields keep the citations they had (the fields.json entry and its status are unchanged here): api_server_count (blis-vllm#105), default_chat_template_kwargs and thinking mode (#107), enable_eplb (#108), and decode context parallelism and prefix_match_unit (#109). api_server_count has the estimation rule none and is not delivered: stating it changes nothing the engine simulates only for the single-client data-parallel case, because vLLM scales the in-flight routing score, the optimistic increment and each client's scan start by the client count (vllm/v1/engine/core_client.py:1639-1640,1671,1685@5df63a72); #105 asks for that model.

  • inert_unproven is a transition status, allowed only until M2 (release plan §3.3). The perturbation test over every modelled field, VLM-3's other check, is not here yet.

  • TestTheFieldListLoadsAndNamesThisRelease loads the file with the substrate's strict loader, checks it names this release, is in canonical form, and that every cited trace exists.

Licence

Apache-2.0. Files translated from vLLM keep vLLM's licence and name the upstream file and commit they were derived from; see NOTICE.

Documentation

Overview

Package blisvllm is a Go model of the vLLM inference engine's scheduling, KV-cache, connector and metrics behaviour, for a discrete-event simulator to call through the engine-neutral contract in github.com/inference-sim/blis-substrate/contracts/engine.

One build of this module models one vLLM release, Upstream. Its behaviour is checked against that release's own code in CI (see docs/testing.md): the same inputs, fed to the real vLLM classes in Python and to this module in Go, must make the same decisions step by step.

The simulator owns time. Nothing in this module reads a clock or starts a goroutine: every call carries the simulated instant as a value, and where a vLLM rule makes work wait until a later instant, the engine reports that instant through NextWake.

Index

Constants

View Source
const Kind = "vllm"

Kind is the engine kind this module registers with the contracts' engine factory.

View Source
const ModulePath = "github.com/inference-sim/blis-vllm"

ModulePath is this module's import path.

Variables

View Source
var FieldListJSON []byte

FieldListJSON is fields.json, the field list this build publishes (VLM-3): every configuration field and environment variable the modelled vLLM release reads, with its status, evidence and estimation rule. Its type is blis-substrate's contracts/spec/fields; inference-sim's fields command prints it.

View Source
var Upstream = UpstreamRelease{
	Version: "0.32.0",
	Tag:     "v0.32.0",
	Commit:  "5df63a7203cdb9328d824a27b682e77c7b9928be",
}

Upstream identifies the vLLM release this build models.

Functions

This section is empty.

Types

type Provenance

type Provenance struct {
	EngineKind      string `json:"engine_kind" yaml:"engine_kind"`
	UpstreamVersion string `json:"upstream_version" yaml:"upstream_version"`
	UpstreamCommit  string `json:"upstream_commit" yaml:"upstream_commit"`
	// Module is this module's version as the build recorded it: a release tag, or a
	// pseudo-version naming the commit; "(devel)" in a build of this repository itself.
	Module string `json:"module" yaml:"module"`
	// Behaviour names the behaviour the engine executes: "vllm-0.32.0", or a compatibility
	// behaviour a scenario selected (see Compat).
	Behaviour string `json:"behaviour" yaml:"behaviour"`
}

Provenance is what a run must record to be reproduced: the engine kind, the upstream release it models and this module's own revision. A simulator writes it into its results beside the kernel's provenance.

func ProvenanceFor

func ProvenanceFor(behaviour string) Provenance

ProvenanceFor returns the provenance of an engine executing behaviour.

type UpstreamRelease

type UpstreamRelease struct {
	Version string
	Tag     string
	Commit  string
}

UpstreamRelease is one vLLM release.

Directories

Path Synopsis
cmd
metricsmanifest command
Command metricsmanifest lists every vllm:* metric family a vLLM release defines, from the release's own source, and writes it as docs/metrics-manifest.json with each family's class: exact (blis-vllm computes it from the decisions it makes), derived (it needs inputs the simulator supplies, such as time or transfer outcomes, and is reported given them) or unavailable (it needs real hardware, a process or a feature outside the simulation).
Command metricsmanifest lists every vllm:* metric family a vLLM release defines, from the release's own source, and writes it as docs/metrics-manifest.json with each family's class: exact (blis-vllm computes it from the decisions it makes), derived (it needs inputs the simulator supplies, such as time or transfer outcomes, and is reported given them) or unavailable (it needs real hardware, a process or a feature outside the simulation).
provenance command
Command provenance prints, as JSON, the Go side of a CI run's provenance: blis-vllm's identity as the build records it (blisvllm.ProvenanceFor), the commit the run checks, the Go toolchain and the platform.
Command provenance prints, as JSON, the Go side of a CI run's provenance: blis-vllm's identity as the build records it (blisvllm.ProvenanceFor), the commit the run checks, the Go toolchain and the platform.
compat
build
Package build is the compatibility engine's configuration: it decodes and resolves the engine-side blocks of a scenario (its KV offload block, its engine and pools) and the command-line restatements of them, so the simulator hands them over unread (#2021 B2, B6).
Package build is the compatibility engine's configuration: it decodes and resolves the engine-side blocks of a scenario (its KV offload block, its engine and pools) and the command-line restatements of them, so the simulator hands them over unread (#2021 B2, B6).
contract
Package contract puts the compatibility engine (compat/engine) behind blis-schemas' engine.Engine, the contract blis-vllm's v0.32.0 engine (core/vllm) implements, so a simulator drives every release through one driver (inference-sim GOAL.md phase 2c).
Package contract puts the compatibility engine (compat/engine) behind blis-schemas' engine.Engine, the contract blis-vllm's v0.32.0 engine (core/vllm) implements, so a simulator drives every release through one driver (inference-sim GOAL.md phase 2c).
dplb
Package dplb is the compatibility engine's model of vLLM's internal data-parallel load balancer, the front-end rule that sends a request reaching a data-parallel replica's endpoint to one of its engine ranks.
Package dplb is the compatibility engine's model of vLLM's internal data-parallel load balancer, the front-end rule that sends a request reaching a data-parallel replica's endpoint to one of its engine ranks.
engine
A data-parallel wave's continuation rule: whether a lockstep group runs another step.
A data-parallel wave's continuation rule: whether a lockstep group runs another step.
internal/util
Package util provides generic utility functions shared across sim/ sub-packages.
Package util provides generic utility functions shared across sim/ sub-packages.
kernelmodel
capacity.go derives the KV-block budget from the kernel's own memory methods.
capacity.go derives the KV-block budget from the kernel's own memory methods.
kv
Package kv implements block-based KV cache management for the BLIS simulator.
Package kv implements block-based KV cache management for the BLIS simulator.
kvkey
Package kvkey is the single gated home for KV cache block keying: deriving hierarchical content keys at a chosen stride, and interning those keys to compact integer handles.
Package kvkey is the single gated home for KV cache block keying: deriving hierarchical content keys at a chosen stride, and interning those keys to compact integer handles.
kvtransfer
Package kvtransfer models vLLM's KV-offload I/O thread pool as a bounded priority queueing station with N servers per storage tier.
Package kvtransfer models vLLM's KV-offload I/O thread pool as a bounded priority queueing station with N servers per storage tier.
lora
Package lora implements the LoRA (Low-Rank Adaptation) control-plane subsystem for BLIS: adapter identity, a pre-declared adapter registry (id → rank), a per-instance resident-adapter set with LRU eviction, and the three Digital-Twin adapter cost terms (cold-load latency, per-step compute overhead, HBM footprint).
Package lora implements the LoRA (Low-Rank Adaptation) control-plane subsystem for BLIS: adapter identity, a pre-declared adapter registry (id → rank), a per-instance resident-adapter set with LRU eviction, and the three Digital-Twin adapter cost terms (cold-load latency, per-step compute overhead, HBM footprint).
nixl
Package nixl is the compatibility engine's model of vLLM's NIXL connector rules that a P/D simulator's cluster applies around a KV handoff: how long a prefill engine holds a finished request's KV for the decode side's pull (#1934, INV-PD-7).
Package nixl is the compatibility engine's model of vLLM's NIXL connector rules that a P/D simulator's cluster applies around a KV handoff: how long a prefill engine holds a finished request's KV for the decode side's pull (#1934, INV-PD-7).
prefix
Package hash provides SHA256 hashing utilities for KV cache prefix matching and routing prefix affinity.
Package hash provides SHA256 hashing utilities for KV cache prefix matching and routing prefix affinity.
quantization
Package quantization lowers the `quantization` a deployment states to the weight layout the engine serves, for the catalog checkpoints that state their own quantization recipe.
Package quantization lowers the `quantization` a deployment states to the weight layout the engine serves, for the catalog checkpoints that state their own quantization recipe.
tokenid
Package tokenid defines the simulator's compact token-ID type.
Package tokenid defines the simulator's compact token-ID type.
vllmsettings
Package vllmsettings converts a vLLM engine block into the engine settings the kernel prices (blis-latency-kernel/enginesettings).
Package vllmsettings converts a vLLM engine block into the engine settings the kernel prices (blis-latency-kernel/enginesettings).
core
connector/multi
Package multi is the scheduler side of vLLM's MultiConnector (vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py:135@5df63a72): an ordered list of child connectors behind the one Connector the scheduler calls.
Package multi is the scheduler side of vLLM's MultiConnector (vllm/distributed/kv_transfer/kv_connector/v1/multi_connector.py:135@5df63a72): an ordered list of child connectors behind the one Connector the scheduler calls.
connector/nixl
Package nixl is the scheduler side of vLLM's NIXL connector in its pull mode (NixlConnector = NixlPullConnector, vllm/distributed/kv_transfer/kv_connector/factory.py: 177-189@5df63a72): what the engine decides about a P/D request.
Package nixl is the scheduler side of vLLM's NIXL connector in its pull mode (NixlConnector = NixlPullConnector, vllm/distributed/kv_transfer/kv_connector/factory.py: 177-189@5df63a72): what the engine decides about a P/D request.
connector/offloading
Package offloading is the scheduler side of vLLM's OffloadingConnector (vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py:53@5df63a72) over the CPU tier: OffloadingConnectorScheduler (offloading/scheduler.py:563-2101) with CPUOffloadingManager and its LRU policy (vllm/v1/kv_offload/cpu/manager.py:47, cpu/policies/lru.py:16).
Package offloading is the scheduler side of vLLM's OffloadingConnector (vllm/distributed/kv_transfer/kv_connector/v1/offloading_connector.py:53@5df63a72) over the CPU tier: OffloadingConnectorScheduler (offloading/scheduler.py:563-2101) with CPUOffloadingManager and its LRU policy (vllm/v1/kv_offload/cpu/manager.py:47, cpu/policies/lru.py:16).
engine
Package engine is vLLM's EngineCore over core/sched: the batch queue that keeps up to max_concurrent_batches batches in flight (step_with_batch_queue), the plain step() for a queue of one, and the data-parallel wave bookkeeping (DPEngineCoreProc's step counter, sync interval and prefill throttling).
Package engine is vLLM's EngineCore over core/sched: the batch queue that keeps up to max_concurrent_batches batches in flight (step_with_batch_queue), the plain step() for a queue of one, and the data-parallel wave bookkeeping (DPEngineCoreProc's step counter, sync interval and prefill throttling).
kv
Package kv is vLLM's KV-cache bookkeeping: the block pool, its free queue, prefix-cache block hashing and the per-request block tables, for full-attention models.
Package kv is vLLM's KV-cache bookkeeping: the block pool, its free queue, prefix-cache block hashing and the per-request block tables, for full-attention models.
lower
Package lower is the engine's half of the boundary with the cost model (design section 6, SUB-6): the decisions vLLM makes about a launch, which the cost model must receive and not make.
Package lower is the engine's half of the boundary with the cost model (design section 6, SUB-6): the decisions vLLM makes about a launch, which the cost model must receive and not make.
metrics
Package metrics is vLLM's Prometheus stat logger for the vllm:* families whose values the scheduler alone decides (class "exact" in docs/metrics-manifest.json): request counts, KV usage, prefix-cache queries and hits, preemptions, prompt and generation tokens, finished requests by reason, and tokens per step.
Package metrics is vLLM's Prometheus stat logger for the vllm:* families whose values the scheduler alone decides (class "exact" in docs/metrics-manifest.json): request counts, KV usage, prefix-cache queries and hits, preemptions, prompt and generation tokens, finished requests by reason, and tokens per step.
placement
Package placement is where a pool's engine sits, in the terms the boundary types use: the catalog chip and network, the parallel widths and how the ranks fill nodes, and the node and rack of the engine itself.
Package placement is where a pool's engine sits, in the terms the boundary types use: the catalog chip and network, the parallel widths and how the ranks fill nodes, and the node and rack of the engine itself.
sched
Package sched is vLLM's v1 Scheduler: schedule() and update_from_output() for a full-attention model, with chunked prefill, the long-prefill cap, FCFS and priority policies, recompute preemption, prefix caching, speculative decoding (acceptance supplied) and the max_loras batch limit, and the AsyncScheduler's output placeholders and pipeline-parallel decode spacing (Config.Async).
Package sched is vLLM's v1 Scheduler: schedule() and update_from_output() for a full-attention model, with chunked prefill, the long-prefill cap, FCFS and priority policies, recompute preemption, prefix caching, speculative decoding (acceptance supplied) and the max_loras batch limit, and the AsyncScheduler's output placeholders and pipeline-parallel decode spacing (Config.Async).
vllm
Package vllm is vLLM v0.32.0's engine behind blis-schemas' engine.Engine: core/sched's Scheduler, core/engine's batch queue and core/metrics' stat logger, driven by a simulator through the engine-neutral contract instead of by vLLM's EngineCore loop.
Package vllm is vLLM v0.32.0's engine behind blis-schemas' engine.Engine: core/sched's Scheduler, core/engine's batch queue and core/metrics' stat logger, driven by a simulator through the engine-neutral contract instead of by vLLM's EngineCore loop.
internal
substratedata
Package substratedata locates the catalog/ and registry/ directories of the blis-substrate module go.mod pins, as the go command resolves it (go list -m), in the module cache and never in a sibling checkout.
Package substratedata locates the catalog/ and registry/ directories of the blis-substrate module go.mod pins, as the go command resolves it (go list -m), in the module cache and never in a sibling checkout.
kerneltests
cmd/bandprobe command
Command bandprobe prices one step per line of a CSV and prints both band edges.
Command bandprobe prices one step per line of a CSV and prints both band edges.
cmd/overlap-probe command
Command overlap-probe answers one question: does the per-layer overlap term change the SHAPE of the concurrency response, or only its level?
Command overlap-probe answers one question: does the per-layer overlap term change the SHAPE of the concurrency response, or only its level?
cmd/worked-table command
Command worked-table prints a worked step-time table, computed by the kernel itself from the committed catalog and registry.
Command worked-table prints a worked step-time table, computed by the kernel itself from the committed catalog and registry.
internal/artifacttest
Package artifacttest holds the rule every test package uses to decide whether a failure to load a catalog or registry artifact is a skip or a failure.
Package artifacttest holds the rule every test package uses to decide whether a failure to load a catalog or registry artifact is a skip or a failure.
internal/docgen command
Command docgen writes the generated parts of the documentation into docs/generated/:
Command docgen writes the generated parts of the documentation into docs/generated/:
internal/harness
Package harness holds what this repository's inspection commands share: opening a kernel from a scenario file, and building the batch shapes they price.
Package harness holds what this repository's inspection commands share: opening a kernel from a scenario file, and building the batch shapes they price.
Package registry holds the vLLM entries of the KV-connector registry (blis-schemas spec/connector, inference-sim/inference-sim#2021 B10): which connector names a deployment may use, and which engine model in this module reproduces each.
Package registry holds the vLLM entries of the KV-connector registry (blis-schemas spec/connector, inference-sim/inference-sim#2021 B10): which connector names a deployment may use, and which engine model in this module reproduces each.
Package rules holds the vLLM rules packs: one per vLLM release, each the release's accepted values, its defaults and its version-scoped checks, implementing blis-schemas' rules.EngineRules.
Package rules holds the vLLM rules packs: one per vLLM release, each the release's accepted values, its defaults and its version-scoped checks, implementing blis-schemas' rules.EngineRules.
engineconfig
Package engineconfig is vLLM's engine configuration, the keys a deployment pool's engine: block states for a vLLM release: the Engine struct, its YAML decoding (Decode, Encode), its defaults and its release-independent field checks (Validate).
Package engineconfig is vLLM's engine configuration, the keys a deployment pool's engine: block states for a vLLM release: the Engine struct, its YAML decoding (Decode, Encode), its defaults and its release-independent field checks (Validate).
pack
Package pack is the shape of a vLLM rules pack: one release's accepted values, its defaults and its version-scoped checks, as data (Pack), and the behaviour a consumer resolves against it.
Package pack is the shape of a vLLM rules pack: one release's accepted values, its defaults and its version-scoped checks, as data (Pack), and the behaviour a consumer resolves against it.
v0_29
Package v0_29 is the rules pack for vLLM 0.29.x.
Package v0_29 is the rules pack for vLLM 0.29.x.
v0_31
Package v0_31 is the rules pack for vLLM 0.31.0 (tag v0.31.0, commit db9527a468), the release blis-latency-kernel encodes.
Package v0_31 is the rules pack for vLLM 0.31.0 (tag v0.31.0, commit db9527a468), the release blis-latency-kernel encodes.
v0_32
Package v0_32 is the rules pack for vLLM 0.32.0 (tag v0.32.0, commit 5df63a72), the release blis-vllm executes.
Package v0_32 is the rules pack for vLLM 0.32.0 (tag v0.32.0, commit 5df63a72), the release blis-vllm executes.
Package vllmengine is how a simulator gets blis-vllm's vLLM v0.32.0 engine (core/vllm) without naming any of its packages: Register makes Kind "vllm" available in blis-schemas' engine registry, built from nothing but an engine.Spec, and the simulator drives it through engine.Engine.
Package vllmengine is how a simulator gets blis-vllm's vLLM v0.32.0 engine (core/vllm) without naming any of its packages: Register makes Kind "vllm" available in blis-schemas' engine registry, built from nothing but an engine.Spec, and the simulator drives it through engine.Engine.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL