latencykernel

README

blis-latency-kernel

A Go implementation of the BLIS inference cost model: given a model graph, a chip, a set of calibrated coefficients and a deployment, it prices one forward pass over a batch.

It implements the Kernel interface from blis-schemas and reads its inputs from blis-catalog (declared hardware and model facts) and blis-registry (calibrated coefficients). It holds no data of its own.

The model it realizes is specified in inference-sim/docs/perf-model: thirteen primitives, composed per layer as a max over resources and summed across layers, with the exceptions and their measured evidence recorded there.

What it is for

A discrete-event simulator calls StepTime once per simulated step, so the design splits sharply between what is resolved once and what is computed per call. New does the layout resolution, coefficient scoping, graph planning and all batch-independent arithmetic; a step is then a walk over distinct layer kinds rather than layers, and costs a few hundred nanoseconds. StepTimeInto is the allocation-free variant for callers that supply their own breakdown map.

Nothing on that path is a table lookup. The kernel evaluates closed-form laws, which is what lets it price a part or a shape that was never measured — and also what bounds its accuracy against an interpolating model on shapes that were.

The model it prices arrives as a ModelGraph: a DAG of nine cost primitives, defined in blis-schemas (spec/model) and instantiated in blis-catalog as models/<name>/graph.yaml. This kernel consumes graphs; it neither defines nor stores them, and its library code contains no model names — dispatch is on the primitives, so a model's identity never reaches a branch. See The BLIS Repositories.

Layout

Path	Owns
kernel.go	The interface methods and the step-time composition
new.go	New: resolution, coefficient lifting, precomputation
internal/price	The cost laws as pure functions: efficiency ramp, collective spans, KV bytes
internal/resolve	Scenario to layout and fabric; which conditional collectives survive; how the decode-context knobs resolve
internal/harness	What the scoring commands share
cmd/score	Absolute inter-token latency against published benchmark runs
cmd/shape	Concurrency-response shape against NVIDIA's AISimulate accuracy snapshot
cmd/worked-table	Generates the worked step-time table in the design document

Accuracy

cmd/shape is the standing evaluation: it scores the ratio of step times across concurrency at a fixed deployment against NVIDIA's published snapshot, with both sides normalised to their own lowest concurrency. The current figures, and the conditions under which they hold, are reported by the command itself rather than restated here, because a number in a README goes stale and a number a command prints cannot.

Tests

Behavioural rather than structural: each asserts what a method does for a given input, not that a symbol exists. Several carry a comment naming the defect they were written against, because a test whose purpose is forgotten is the next test to be deleted.

scripts/fetch-testdata.sh
go test ./...

The catalog and registry are fetched at the commits testdata/upstream.lock pins rather than committed, and blis-schemas is an ordinary module dependency. The measurement corpora the scoring commands compare against are third-party data and are not distributed here; cmd/score, cmd/shape and the scoring test suite read them from BLIS_MEASUREMENTS. testdata/README.md has the details. CI (.github/workflows/ci.yml) runs the fetch and the full default suite on every pull request.

Where a law has an exact engine counterpart, the tests check it against that counterpart transcribed from vLLM v0.31.0 -- the decode-context per-rank length, the prefill-context zigzag, the KV page arithmetic, the rank layout every collective group is built from -- over grids or random draws rather than at one point. Where a quantity has no closed form, they assert metamorphic relations: inflate one primitive's coefficients and check which step times move, change one width and check which terms follow.

Sources of truth, pinned

Every claim this code makes about what the engine does is stated against one release, and every pin is a commit, not a moving name:

what	source	pin
engine behaviour	vLLM v0.31.0	db9527a46873454610df6dbedf79a36d6bf1a7f6
interface and document schemas	blis-schemas v0.2.2	a0ba5d42ff753713f4dabc74a4ca38b374a91ef4 (go.mod)
coefficients	blis-registry v0.1.1	f7519b12b3393851a3ad416e26c819dd951d4c0d (testdata/upstream.lock)
catalog	blis-catalog 0.2.1	28e82d4c249893ee25b1412a63d2347c5165b004 (testdata/upstream.lock)

A comment citing vllm/<path>:<lines> means those lines at v0.31.0; read them with git show v0.31.0:<path> in a vLLM checkout. Engine behaviour this repository encodes itself -- how decode-context parallelism combines, which layouts start, how a graph mode is dispatched, what an unstated setting defaults to -- is v0.31.0's. Version-scoped facts the kernel reads from a blis-schemas rules pack instead (which widths the custom all-reduce supports, when sequence-parallel MoE engages, which DCP backend names a release accepts) are that pack's, and the pack is chosen by the scenario's engine_version; only the v0.29 pack is published today.

Where the kernel cannot know what the engine would choose -- a model's own DCP defaults, a backend-preferred block size, a downgraded graph mode -- it does not guess silently. It either refuses the deployment, or prices the engine's stock default and records that default in Provenance() as an entry from set blis-latency-kernel with method assumed, which Evidence() counts. Where v0.31.0's choice follows from the deployment, it is made as the engine makes it and recorded the same way: an unstated block size on a DSA sparse-MLA model, which is 64 rather than the stock 16 (sparse_mla.go). A deployment that states every setting carries no such entry, with four exceptions: a scenario declaring an engine release other than 0.31.0, whose behaviour the kernel still prices at 0.31.0 (the fixtures declare 0.29.0, the one release with a published rules pack); the cudagraph capture ceiling, which blis-schemas has no field to state, so a capturing deployment always runs vLLM's default; a stated DCP backend that could not be checked against the release, because the rules value carries no list of accepted names; and a DSA model on Hopper whose cache selects FlashInfer's SM90 sparse MLA, which needs a FlashInfer release the kernel cannot see installed. An optional coefficient the deployment needs and the scenario's coefficient sets do not carry is recorded the same way, naming what is priced without it.

Where the kernel knowingly prices something differently from v0.31.0 -- because correcting it would move scored results fitted under the current structure, and is therefore a refit rather than a fix -- the code says so at the site, with the engine lines it diverges from. Those notes are headed KNOWN DIVERGENCE, KNOWN over-charge or COVERAGE LIMIT, so grep -rnE "KNOWN DIVERGENCE|KNOWN over-charge|COVERAGE LIMIT" --include='*.go' . lists them.

Using it from a simulator

Everything a simulator needs is on kernel.Kernel:

• Deployment() is the pool as stated: its role and its engine settings, including the admission settings (block_size, max_num_seqs, max_num_batched_tokens) a scheduler sizes itself from.
• Resolved() is what resolution settled: the tensor-, data- and expert-parallel widths (TensorParallel(), DataParallel(), ExpertParallel() apply the floor of one), the all-reduce backend, and every request the layout overrode, with the reason.

The two context-parallel widths are the one exception: kernel.Resolution does not carry them yet, so DecodeContextParallelWidth() and PrefillContextParallelWidth() sit on *Kernel until it does.

Deployment identity -- the model, the chip, the expert geometry -- is not re-exported. A harness that configures another backend for the same deployment takes the documents from OpenInputs and builds the kernel with New(in), so it holds the very chip and graph the kernel priced.

Dependencies run one way

This module depends on blis-schemas and nothing else of substance. It does NOT depend on the simulator: the simulator consumes this kernel, so an import in this direction would close a cycle between the two repositories. The command that compares this kernel against the simulator's earlier roofline and trained-physics models therefore lives in the simulator, as cmd/blisbaseline, where the dependency runs the correct way.

Licence

Apache 2.0, matching the other BLIS repositories.

Documentation

Overview

Package latencykernel implements the blis-schemas Kernel interface: it prices GPU work for one resolved deployment.

Construction resolves everything — catalog facts, registry coefficients, the engine settings a layout overrides, and the model graph flattened to a per-kind plan. After that every method is a pure function of its arguments, so a discrete-event simulator can call StepTime at any simulated instant, from any goroutine, and get the same answer.

Two constructors, for the two shapes a caller comes in. Open takes a scenario filename and the roots its names resolve against, which is what a simulator or a scoring command has. New takes the documents already parsed, which is what a configuration search producing deployment variants that were never written to disk has. Open ends in New, so both validate identically and neither can price a deployment the other would refuse.

Speed is a requirement rather than a nicety: a simulator calls StepTime once per simulated step, millions of times over a run. So the hot path does arithmetic over a short pre-aggregated slice and allocates nothing. The plan holds one entry per distinct layer kind rather than one per layer, which for the deepest model in the catalog is three entries instead of 108.

Index

• Constants
• func LoadBundle(path string) (*scenario.Scenario, *deployment.Deployment, error)
• type Inputs
  • func OpenInputs(scenario string, r Repos, poolIndex int) (Inputs, error)
• type Kernel
  • func New(in Inputs) (*Kernel, error)
  • func Open(scenario string, r Repos) (*Kernel, error)
  • func OpenPool(scenario string, r Repos, poolIndex int) (*Kernel, error)
  • func (k *Kernel) AdmissionOverhead(promptTokens int) time.Duration
  • func (k *Kernel) CompletionOverhead() time.Duration
  • func (k *Kernel) DecodeContextParallelWidth() int
  • func (k *Kernel) Deployment() deployment.Pool
  • func (k *Kernel) Evidence() (measured, total int, assumed []string)
  • func (k *Kernel) FixedBytes() kernel.MemoryBreakdown
  • func (k *Kernel) OutputTokenOverhead() time.Duration
  • func (k *Kernel) PDTransferTime(tokens int, from, to kernel.Placement) time.Duration
  • func (k *Kernel) PrefillContextParallelWidth() int
  • func (k *Kernel) Provenance() []kernel.CoefficientOrigin
  • func (k *Kernel) Resolved() kernel.Resolution
  • func (k *Kernel) SequenceFixedBytes() int64
  • func (k *Kernel) SequenceVariableBytes(tokens int) int64
  • func (k *Kernel) StepTime(b kernel.Batch) kernel.StepEstimate
  • func (k *Kernel) StepTimeInto(b kernel.Batch, per map[kernel.Resource]time.Duration) kernel.StepEstimate
  • func (k *Kernel) TierTime(tier string, dir kernel.Direction, bytes int64, queueDepth int) time.Duration
• type Repos

Constants

const EngineBehaviourVersion = "0.31.0"

EngineBehaviourVersion is the vLLM release whose engine behaviour this kernel encodes.

const KernelAssumptionSet = "blis-latency-kernel"

KernelAssumptionSet is the Set a Provenance entry carries when the kernel supplied the value itself rather than reading it from a registry set or the deployment. No registry set can take this name, since registry sets are named for their coefficient family, so a consumer can tell the two apart without parsing anything else.

Functions

func LoadBundle

func LoadBundle(path string) (*scenario.Scenario, *deployment.Deployment, error)

LoadBundle reads a scenario and the deployment applied to it from one file, as two YAML documents separated by `---`.

blis-schemas keeps the two apart: a Scenario is the immutable problem (model, cluster inventory, coefficient and engine-version references) and a Deployment is the tunable configuration chosen against it (pools, each a parallelism layout with its own engine settings). One `blis run` is one of each.

They share a file here, rather than sitting in a scenario and a sibling deployment file, because a measurement row addresses a deployment by a single filename — `{"scenario": "glm-5-h200-fp8-sglang-tp8.yaml", ...}` — and testdata/measurements holds over three thousand such rows. A sibling-file layout would rewrite every one of them to say nothing it does not already say.

blis-schemas' own LoadScenario and LoadDeployment each open a path and read one document, so neither can reach the second half of a pair; this reproduces their strict decoding over a single stream instead. Strictness is the point: a misspelled key that parsed silently would leave a document that validates while omitting the setting its author intended, which is the failure mode hardest to notice.

Types

type Inputs

type Inputs struct {
	Scenario *scenario.Scenario
	// Deployment is the tunable configuration applied to the scenario: the pools that lay
	// the model out. It is a separate document because a scenario fixes the immutable
	// problem and an optimizer sweeps deployments against it.
	Deployment *deployment.Deployment
	// PoolIndex selects which pool of a disaggregated deployment this kernel prices. Each
	// pool runs its own engine with its own settings, so one kernel per pool.
	PoolIndex    int
	Model        *model.Graph
	Chip         *hardware.Chip
	Fabric       *hardware.Fabric
	Devices      []*hardware.StorageDevice
	Coefficients []*coefficient.Set
	// Rules carry the engine-version behaviour the resolver needs.
	Rules resolve.EngineRules
}

Inputs are the documents a kernel is built from. They arrive already parsed: loading belongs to blis-schemas, and New takes documents rather than paths so a caller holding structs — a config search producing deployment variants never written to disk, a test, a generator — does not have to serialize them first. Open is the path-based half of the pair.

New DOES validate them, at the field layer, which is the half of blis-schemas' two-layer validation that is version-independent and always the author's to fix. See New.

func OpenInputs

func OpenInputs(scenario string, r Repos, poolIndex int) (Inputs, error)

OpenInputs resolves one pool of a scenario file into the documents New prices it from, without building the kernel.

It is OpenPool's loading half, exported for a caller that needs the documents as well as the kernel. A scoring harness that configures a rival backend for the same deployment needs the model's name, the chip's memory and the model's expert geometry -- deployment identity, which the kernel deliberately does not re-export (blis-schemas#36 draws that line: identity for comparison is the harness's concern, not the cost model's). Reading them from these Inputs, and building the kernel with New(in), means the harness holds the SAME chip and graph the kernel priced, not a second load of files that could resolve differently.

What it must not be used for is reaching configuration the kernel resolved: widths and backends are Resolved's to report, and the pool's settings are Deployment's. Reading those from Inputs instead would rebuild the second answer to "which pool is this" that the interface exists to remove.

type Kernel

type Kernel struct {
	// contains filtered or unexported fields
}

Kernel prices one deployment. Obtain one from New; the zero value is not usable.

func New

func New(in Inputs) (*Kernel, error)

New resolves the inputs into a kernel.

Everything that can be decided once is decided here: which coefficients apply, which requested settings the layout overrides, and the model graph flattened to a per-kind plan. The result holds no mutable state, so its methods are pure and safe to call concurrently.

Construction is deliberately strict. A missing coefficient, an unresolvable condition or a non-positive rate is an error rather than a default, because each would otherwise produce a step time that looks plausible and is wrong by whatever the term contributes. validateDocuments extends that strictness to the documents themselves.

func Open

func Open(scenario string, r Repos) (*Kernel, error)

Open resolves one scenario file into a kernel.

The path-based half of the constructor pair. New takes documents a caller already holds; Open takes a filename and the roots its names resolve against, which is what a simulator or a scoring command has. Both end in New, so both inherit its validation and neither can price a deployment the other would reject.

It is exported because the alternative is every consumer reimplementing it. Go forbids importing another module's internal packages, so while this lived in internal/harness, inference-sim carried a copy of the sequence and said so in a comment, and blis-config-search and every future backend repo would have needed their own. The duplication was tolerable only because it was confined to loading — but "which files a scenario implies" is this package's answer to give, not each caller's to guess.

Every artifact is loaded through blis-schemas' own loaders, and the engine rules come from that package's version registry rather than from a caller, so a scenario pinned to an older release cannot silently get current behaviour.

It prices the deployment's FIRST pool, which is what a colocated deployment has. Use OpenPool to price one pool of a disaggregated one.

func OpenPool

func OpenPool(scenario string, r Repos, poolIndex int) (*Kernel, error)

OpenPool resolves ONE pool of a scenario file into a kernel.

A disaggregated deployment states a prefill pool and a decode pool, and the two differ in the quantities that set step time: tensor-parallel width, whether expert parallelism is on, the engine's token budget. One kernel prices one pool — each runs its own engine with its own settings — so a caller serving the roles separately opens one kernel per pool. Pricing both from pool 0 would charge the decode pool the prefill pool's parallelism, and the simulation would still run.

Open is this function at pool 0. Both exist because the common case should not have to name an index, and the disaggregated case cannot be served without one.

func (*Kernel) AdmissionOverhead

func (k *Kernel) AdmissionOverhead(promptTokens int) time.Duration

AdmissionOverhead returns host time before a request can be scheduled.

func (*Kernel) CompletionOverhead

func (k *Kernel) CompletionOverhead() time.Duration

CompletionOverhead returns fixed host time at request completion.

func (*Kernel) DecodeContextParallelWidth

func (k *Kernel) DecodeContextParallelWidth() int

DecodeContextParallelWidth returns how many ranks shard the decode KV cache.

A consumer sizing its own KV budget needs it and cannot derive it from anything else this kernel exposes: it is the only axis that shards a LATENT cache, because KVBytesPerToken floors at one KV head and a latent cache has exactly one, so no tensor-parallel width reduces it. A simulator that read Resolution's TensorParallelWidth alone would size an MLA deployment's cache as if DCP did nothing.

From the resolved layout rather than the deployment document: a Parallelism states a request and resolution settles it.

A method on this type rather than a field of kernel.Resolution only because blis-schemas v0.2.2's Resolution carries the tensor-, data- and expert-parallel widths and not the two context-parallel ones. The argument that put those three in Resolution -- a consumer that can read one width and not the others has to re-derive the rest -- applies to these two as well, so this method and its companion are interim, pending the same fields upstream.

func (*Kernel) Deployment

func (k *Kernel) Deployment() deployment.Pool

Deployment returns the pool this kernel prices, as the document stated it.

The request, not the resolution: a caller wanting a width or a backend reads Resolved, and reads this for the settings resolution does not touch -- the admission settings a scheduler sizes itself from. It is the pool New was given at PoolIndex, so a caller holding several kernels of one disaggregated deployment needs no index of its own to know which pool each prices.

A deep copy, so a caller writing through one of the engine block's pointer fields changes its copy and not the kernel. blis-schemas permits returning a shallow copy and asking callers not to write; this is not a hot path, and a promise the type enforces is worth more than one a reader has to keep.

func (*Kernel) Evidence

func (k *Kernel) Evidence() (measured, total int, assumed []string)

Evidence reports how much of the resolved coefficient set rests on measurement, so a caller can state a prediction's footing.

func (*Kernel) FixedBytes

func (k *Kernel) FixedBytes() kernel.MemoryBreakdown

FixedBytes returns occupancy independent of the request set, per rank.

func (*Kernel) OutputTokenOverhead

func (k *Kernel) OutputTokenOverhead() time.Duration

OutputTokenOverhead returns host time per emitted token.

func (*Kernel) PDTransferTime

func (k *Kernel) PDTransferTime(tokens int, from, to kernel.Placement) time.Duration

PDTransferTime prices moving one request's KV between pools.

Decode-context parallelism deliberately does NOT divide this, and the omission is the reason kvBytesPerToken is left as a per-token figure rather than being sharded once at construction. A prefill pool holds the whole cache for a request -- DCP shards the DECODE cache, and vLLM's own wording is "Number of ranks that shard the decode KV cache" (vllm/config/parallel.py:359-362 at v0.31.0) -- so the bytes that cross between pools are the whole request's, whatever the decode pool's dcp width is. A divisor here would make every PD transfer exactly dcp times too cheap.

func (*Kernel) PrefillContextParallelWidth

func (k *Kernel) PrefillContextParallelWidth() int

PrefillContextParallelWidth returns how many ranks split a prefill sequence.

The companion to DecodeContextParallelWidth, and the two are genuinely independent: PCP splits prefill computation and expands the process world size while leaving the KV cache replicated, where DCP shards the cache without expanding the world size -- it reuses the tensor-parallel ranks when pcp is 1, and spans the PCP axis or the whole TP x PCP block otherwise (vllm/config/parallel.py:359-362 at v0.31.0). A consumer sizing a deployment needs both, and neither can be derived from the other.

Interim for the same reason as DecodeContextParallelWidth.

func (*Kernel) Provenance

func (k *Kernel) Provenance() []kernel.CoefficientOrigin

Provenance reports every coefficient this kernel used and its evidence.

func (*Kernel) Resolved

func (k *Kernel) Resolved() kernel.Resolution

Resolved reports the configuration after resolution, including overridden requests.

func (*Kernel) SequenceFixedBytes

func (k *Kernel) SequenceFixedBytes() int64

SequenceFixedBytes returns per-sequence bytes that do not vary with prompt length: recurrent state, where the model has it and the cache mode keeps it bounded.

func (*Kernel) SequenceVariableBytes

func (k *Kernel) SequenceVariableBytes(tokens int) int64

SequenceVariableBytes returns per-sequence KV bytes for a token count, quantized to the engine's page size.

func (*Kernel) StepTime

func (k *Kernel) StepTime(b kernel.Batch) kernel.StepEstimate

StepTime prices one forward pass. This is the hot path.

The returned StepEstimate carries a freshly allocated PerResource map. A simulator that calls this per simulated step, millions of times, can use StepTimeInto instead and reuse one map across calls, which makes the path allocation-free.

func (*Kernel) StepTimeInto

func (k *Kernel) StepTimeInto(b kernel.Batch,
	per map[kernel.Resource]time.Duration) kernel.StepEstimate

StepTimeInto prices one forward pass, writing the per-resource breakdown into per instead of allocating a map.

per is cleared first, so a caller reuses one map for every step. Passing nil is allowed and yields an estimate with no breakdown, for a caller that wants only the duration.

The estimate's own fields are values, so nothing else escapes: this form allocates nothing at all, which is what lets a long simulation avoid the garbage collector on its innermost loop.

func (*Kernel) TierTime

func (k *Kernel) TierTime(tier string, dir kernel.Direction, bytes int64,
	queueDepth int) time.Duration

TierTime prices one transfer between GPU and an offload tier.

type Repos

type Repos struct {
	Scenarios string // directory holding scenario files
	Catalog   string // a blis-catalog checkout, or a pinned copy of one
	Registry  string // a blis-registry checkout, or a pinned copy of one
}

Repos locates the roots a scenario's names resolve against.

A scenario names its model, hardware, fabric and coefficient sets without saying where they live — it describes a deployment, not a filesystem — so a caller supplies the roots.