latencykernel

README

blis-latency-kernel

The cost model of BLIS. Given a model, a GPU, a set of coefficients for that GPU and a deployment, it answers the two questions a serving simulator asks: how long one forward pass over a batch of requests takes, and how much GPU memory the deployment holds.

Documentation: https://inference-sim.github.io/blis-latency-kernel/

It is a Go library implementing the Kernel interface from blis-schemas. It reads the model and the hardware from blis-catalog and the coefficients from blis-registry, and holds no data of its own. inference-sim calls it once per simulated step in pull request #1851. It prices the engine behavior of vLLM v0.31.0.

Use

go get github.com/inference-sim/blis-latency-kernel@latest

k, err := latencykernel.Open("my-deployment.yaml", latencykernel.Repos{
	Scenarios: "scenarios", Catalog: "blis-catalog", Registry: "blis-registry",
})
if err != nil {
	panic(err)
}
e := k.StepTime(batch) // e.Overlap, e.NoOverlap, e.Bottleneck, e.PerResource

Price a batch from Go walks through it with a real deployment, and Pricing a step explains what the estimate means.

Develop

scripts/fetch-testdata.sh   # the catalog and registry, at the commits testdata/upstream.lock pins
go test ./...

The tests are behavioral: they check what the kernel does for an input, most often against a transcription of the vLLM source or an exact relation the answer must satisfy. testdata/README.md describes the test inputs, including the measurement corpora the scoring commands use, which are third-party data and not distributed here. Contributing describes the layout of the repository and the conventions a change is reviewed against.

License

Apache 2.0, matching the other BLIS repositories.

Documentation

Overview

Package latencykernel implements the blis-schemas Kernel interface: it prices GPU work for one resolved deployment.

Construction resolves everything — catalog facts, registry coefficients, the engine settings a layout overrides, and the model graph flattened to a per-kind plan. After that every method is a pure function of its arguments, so a discrete-event simulator can call StepTime at any simulated instant, from any goroutine, and get the same answer.

Two constructors, for the two shapes a caller comes in. Open takes a scenario filename and the roots its names resolve against, which is what a simulator or a scoring command has. New takes the documents already parsed, which is what a configuration search producing deployment variants that were never written to disk has. Open ends in New, so both validate identically and neither can price a deployment the other would refuse.

Speed is a requirement rather than a nicety: a simulator calls StepTime once per simulated step, millions of times over a run. So the hot path does arithmetic over a short pre-aggregated slice and allocates little: the per-resource breakdown map, which StepTimeInto lets a caller reuse, and a few scratch slices that grow with the number of requests. The plan holds one entry per distinct layer kind rather than one per layer, which for the deepest model in the catalog is three entries instead of 108.

Index

• Constants
• func LoadBundle(path string) (*scenario.Scenario, *deployment.Deployment, error)
• type Inputs
  • func OpenInputs(scenario string, r Repos, poolIndex int) (Inputs, error)
• type Kernel
  • func New(in Inputs) (*Kernel, error)
  • func Open(scenario string, r Repos) (*Kernel, error)
  • func OpenPool(scenario string, r Repos, poolIndex int) (*Kernel, error)
  • func (k *Kernel) AdmissionOverhead(promptTokens int) time.Duration
  • func (k *Kernel) CompletionOverhead() time.Duration
  • func (k *Kernel) DecodeContextParallelWidth() int
  • func (k *Kernel) Deployment() deployment.Pool
  • func (k *Kernel) Evidence() (measured, total int, assumed []string)
  • func (k *Kernel) FixedBytes() kernel.MemoryBreakdown
  • func (k *Kernel) OutputTokenOverhead() time.Duration
  • func (k *Kernel) PDTransferDemand(tokens int, from, to kernel.Placement) kernel.TransferDemand
  • func (k *Kernel) PDTransferTime(tokens int, from, to kernel.Placement) time.Duration
  • func (k *Kernel) PrefillContextParallelWidth() int
  • func (k *Kernel) Provenance() []kernel.CoefficientOrigin
  • func (k *Kernel) Resolved() kernel.Resolution
  • func (k *Kernel) SequenceFixedBytes() int64
  • func (k *Kernel) SequenceVariableBytes(tokens int) int64
  • func (k *Kernel) StepTime(b kernel.Batch) kernel.StepEstimate
  • func (k *Kernel) StepTimeInto(b kernel.Batch, per map[kernel.Resource]time.Duration) kernel.StepEstimate
  • func (k *Kernel) TierTime(tier string, dir kernel.Direction, bytes int64, queueDepth int) time.Duration
• type Repos

Examples

• Kernel.Evidence
• Kernel.FixedBytes
• New
• Open

Constants

const EngineBehaviourCommit = "db9527a46873454610df6dbedf79a36d6bf1a7f6"

EngineBehaviourCommit is the commit of vLLM's EngineBehaviourVersion tag. A comment citing vllm/<path>:<lines> means those lines at this commit.

const EngineBehaviourVersion = "0.31.0"

EngineBehaviourVersion is the vLLM release whose engine behaviour this kernel encodes.

const KernelAssumptionSet = "blis-latency-kernel"

KernelAssumptionSet is the Set a Provenance entry carries when the kernel supplied the value itself rather than reading it from a registry set or the deployment. No registry set can take this name, since registry sets are named for their coefficient family, so a consumer can tell the two apart without parsing anything else.

Functions

func LoadBundle

func LoadBundle(path string) (*scenario.Scenario, *deployment.Deployment, error)

LoadBundle reads a scenario and the deployment applied to it from one file, as two YAML documents separated by `---`.

blis-schemas keeps the two apart: a Scenario is the immutable problem (model, cluster inventory, coefficient and engine-version references) and a Deployment is the tunable configuration chosen against it (pools, each a parallelism layout with its own engine settings). One `blis run` is one of each.

They share a file here, rather than sitting in a scenario and a sibling deployment file, because a measurement row addresses a deployment by a single filename — `{"scenario": "glm-5-h200-fp8-sglang-tp8.yaml", ...}` — and testdata/measurements holds over three thousand such rows. A sibling-file layout would rewrite every one of them to say nothing it does not already say.

blis-schemas' own LoadScenario and LoadDeployment each open a path and read one document, so neither can reach the second half of a pair; this reproduces their strict decoding over a single stream instead. Strictness is the point: a misspelled key that parsed silently would leave a document that validates while omitting the setting its author intended, which is the failure mode hardest to notice.

Types

type Inputs

type Inputs struct {
	Scenario *scenario.Scenario
	// Deployment is the tunable configuration applied to the scenario: the pools that lay
	// the model out. It is a separate document because a scenario fixes the immutable
	// problem and an optimizer sweeps deployments against it.
	Deployment *deployment.Deployment
	// PoolIndex selects which pool of a disaggregated deployment this kernel prices. Each
	// pool runs its own engine with its own settings, so one kernel per pool.
	PoolIndex    int
	Model        *model.Graph
	Chip         *hardware.Chip
	Fabric       *hardware.Fabric
	Devices      []*hardware.StorageDevice
	Coefficients []*coefficient.Set
	// Rules carry the engine-version behaviour the resolver needs.
	Rules resolve.EngineRules
}

Inputs are the documents a kernel is built from. They arrive already parsed: loading belongs to blis-schemas, and New takes documents rather than paths so a caller holding structs — a config search producing deployment variants never written to disk, a test, a generator — does not have to serialize them first. Open is the path-based half of the pair.

New DOES validate them, at the field layer, which is the half of blis-schemas' two-layer validation that is version-independent and always the author's to fix. See New.

func OpenInputs

func OpenInputs(scenario string, r Repos, poolIndex int) (Inputs, error)

OpenInputs resolves one pool of a scenario file into the documents New prices it from, without building the kernel.

It is OpenPool's loading half, exported for a caller that needs the documents as well as the kernel. A scoring harness that configures a rival backend for the same deployment needs the model's name, the chip's memory and the model's expert geometry -- deployment identity, which the kernel deliberately does not re-export (blis-schemas#36 draws that line: identity for comparison is the harness's concern, not the cost model's). Reading them from these Inputs, and building the kernel with New(in), means the harness holds the SAME chip and graph the kernel priced, not a second load of files that could resolve differently.

What it must not be used for is reaching configuration the kernel resolved: widths and backends are Resolved's to report, and the pool's settings are Deployment's. Reading those from Inputs instead would rebuild the second answer to "which pool is this" that the interface exists to remove.

type Kernel

type Kernel struct {
	// contains filtered or unexported fields
}

Kernel prices one deployment. Obtain one from New; the zero value is not usable.

func New

func New(in Inputs) (*Kernel, error)

New resolves the inputs into a kernel.

Everything that can be decided once is decided here: which coefficients apply, which requested settings the layout overrides, and the model graph flattened to a per-kind plan. The result holds no mutable state, so its methods are pure and safe to call concurrently.

Construction is deliberately strict. A missing coefficient, an unresolvable condition or a non-positive rate is an error rather than a default, because each would otherwise produce a step time that looks plausible and is wrong by whatever the term contributes. validateDocuments extends that strictness to the documents themselves.

Example

A deployment variant that was never written to disk: load the documents once, change one setting, and build a kernel from the result.

package main

import (
	"fmt"

	latencykernel "github.com/inference-sim/blis-latency-kernel"
)

// --8<-- [start:repos]
// The roots a scenario's names resolve against: its model, chip, fabric and coefficient sets.
var repos = latencykernel.Repos{
	Scenarios: "testdata",
	Catalog:   "testdata/catalog",
	Registry:  "testdata/registry",
}

func main() {
	// --8<-- [start:new]
	in, err := latencykernel.OpenInputs("aisimulate/deepseek-v3-h200-fp8-sglang-tp8.yaml",
		repos, 0)
	if err != nil {
		panic(err)
	}
	base, err := latencykernel.New(in)
	if err != nil {
		panic(err)
	}

	// The same deployment with a 16-bit KV cache instead of fp8. New copied what it needed,
	// so changing the documents now leaves base as it was.
	in.Deployment.Pools[0].Engine.CacheDType = "bfloat16"
	wide, err := latencykernel.New(in)
	if err != nil {
		panic(err)
	}
	fmt.Printf("fp8:  %d bytes of KV per token per rank\n", base.SequenceVariableBytes(64)/64)
	fmt.Printf("bf16: %d bytes of KV per token per rank\n", wide.SequenceVariableBytes(64)/64)
	// --8<-- [end:new]

}

Output:
fp8:  35136 bytes of KV per token per rank
bf16: 70272 bytes of KV per token per rank

func Open

func Open(scenario string, r Repos) (*Kernel, error)

Open resolves one scenario file into a kernel.

The path-based half of the constructor pair. New takes documents a caller already holds; Open takes a filename and the roots its names resolve against, which is what a simulator or a scoring command has. Both end in New, so both inherit its validation and neither can price a deployment the other would reject.

It is exported because the alternative is every consumer reimplementing it. Go forbids importing another module's internal packages, so while this lived in internal/harness, inference-sim carried a copy of the sequence and said so in a comment, and blis-config-search and every future backend repo would have needed their own. The duplication was tolerable only because it was confined to loading — but "which files a scenario implies" is this package's answer to give, not each caller's to guess.

Every artifact is loaded through blis-schemas' own loaders, and the engine rules come from that package's version registry rather than from a caller, so a scenario pinned to an older release cannot silently get current behaviour.

It prices the deployment's FIRST pool, which is what a colocated deployment has. Use OpenPool to price one pool of a disaggregated one.

Example

Pricing a decode step: open a scenario, describe the batch, read the estimate.

package main

import (
	"fmt"
	"time"

	"github.com/inference-sim/blis-schemas/kernel"

	latencykernel "github.com/inference-sim/blis-latency-kernel"
)

// --8<-- [start:repos]
// The roots a scenario's names resolve against: its model, chip, fabric and coefficient sets.
var repos = latencykernel.Repos{
	Scenarios: "testdata",
	Catalog:   "testdata/catalog",
	Registry:  "testdata/registry",
}

func ms(d time.Duration) float64 { return d.Seconds() * 1e3 }

func main() {
	// --8<-- [start:open]
	// DeepSeek-V3, served in fp8 on eight H200s at tensor-parallel width 8.
	const scenario = "aisimulate/deepseek-v3-h200-fp8-sglang-tp8.yaml"
	k, err := latencykernel.Open(scenario, repos)
	if err != nil {
		panic(err)
	}

	// 32 requests, each decoding one token against a context of 4,096 tokens.
	batch := kernel.Batch{DecodeThreshold: 8}
	for range 32 {
		batch.Reqs = append(batch.Reqs, kernel.ReqShape{
			Scheduled: 1, Computed: 4095, PromptLen: 4096,
		})
	}

	e := k.StepTime(batch)
	fmt.Printf("between %.1f and %.1f ms, bottleneck %s\n",
		ms(e.Overlap), ms(e.NoOverlap), e.Bottleneck)
	for _, r := range []kernel.Resource{kernel.ResourceSM, kernel.ResourceHBM,
		kernel.ResourceNVLink, kernel.ResourceHost} {
		fmt.Printf("%-7s %5.1f ms\n", r, ms(e.PerResource[r]))
	}
	// --8<-- [end:open]

}

Output:
between 24.1 and 25.7 ms, bottleneck hbm
sm        4.5 ms
hbm      16.5 ms
nvlink    1.2 ms
host      3.5 ms

func OpenPool

func OpenPool(scenario string, r Repos, poolIndex int) (*Kernel, error)

OpenPool resolves ONE pool of a scenario file into a kernel.

A disaggregated deployment states a prefill pool and a decode pool, and the two differ in the quantities that set step time: tensor-parallel width, whether expert parallelism is on, the engine's token budget. One kernel prices one pool — each runs its own engine with its own settings — so a caller serving the roles separately opens one kernel per pool. Pricing both from pool 0 would charge the decode pool the prefill pool's parallelism, and the simulation would still run.

Open is this function at pool 0. Both exist because the common case should not have to name an index, and the disaggregated case cannot be served without one.

func (*Kernel) AdmissionOverhead

func (k *Kernel) AdmissionOverhead(promptTokens int) time.Duration

AdmissionOverhead returns host time before a request can be scheduled.

func (*Kernel) CompletionOverhead

func (k *Kernel) CompletionOverhead() time.Duration

CompletionOverhead returns fixed host time at request completion.

func (*Kernel) DecodeContextParallelWidth

func (k *Kernel) DecodeContextParallelWidth() int

DecodeContextParallelWidth returns how many ranks shard the decode KV cache.

A consumer sizing its own KV budget needs it and cannot derive it from anything else this kernel exposes: it is the only axis that shards a LATENT cache, because KVBytesPerToken floors at one KV head and a latent cache has exactly one, so no tensor-parallel width reduces it. A simulator that read Resolution's TensorParallelWidth alone would size an MLA deployment's cache as if DCP did nothing.

From the resolved layout rather than the deployment document: a Parallelism states a request and resolution settles it.

A method on this type rather than a field of kernel.Resolution only because blis-schemas v0.3.0's Resolution carries the tensor-, data- and expert-parallel widths and not the two context-parallel ones. The argument that put those three in Resolution -- a consumer that can read one width and not the others has to re-derive the rest -- applies to these two as well, so this method and its companion are interim, pending the same fields upstream.

func (*Kernel) Deployment

func (k *Kernel) Deployment() deployment.Pool

Deployment returns the pool this kernel prices, as the document stated it.

The request, not the resolution: a caller wanting a width or a backend reads Resolved, and reads this for the settings resolution does not touch -- the admission settings a scheduler sizes itself from. It is the pool New was given at PoolIndex, so a caller holding several kernels of one disaggregated deployment needs no index of its own to know which pool each prices.

A deep copy, so a caller writing through one of the engine block's pointer fields changes its copy and not the kernel. blis-schemas permits returning a shallow copy and asking callers not to write; this is not a hot path, and a promise the type enforces is worth more than one a reader has to keep.

func (*Kernel) Evidence

func (k *Kernel) Evidence() (measured, total int, assumed []string)

Evidence reports how much of the resolved coefficient set rests on measurement, so a caller can state a prediction's footing.

Example

Evidence: how much of what a kernel used was measured, and what it assumed.

package main

import (
	"fmt"
	"maps"
	"slices"

	latencykernel "github.com/inference-sim/blis-latency-kernel"
)

// --8<-- [start:repos]
// The roots a scenario's names resolve against: its model, chip, fabric and coefficient sets.
var repos = latencykernel.Repos{
	Scenarios: "testdata",
	Catalog:   "testdata/catalog",
	Registry:  "testdata/registry",
}

func main() {
	k, err := latencykernel.Open("aisimulate/deepseek-v3-h200-fp8-sglang-tp8.yaml", repos)
	if err != nil {
		panic(err)
	}
	// --8<-- [start:evidence]
	measured, total, assumed := k.Evidence()
	fmt.Printf("%d of %d values rest on evidence; %d were assumed\n",
		measured, total, len(assumed))

	// The same entries, counted by the method that produced each.
	byMethod := map[string]int{}
	for _, o := range k.Provenance() {
		byMethod[o.Method]++
	}
	for _, m := range slices.Sorted(maps.Keys(byMethod)) {
		fmt.Printf("  %-12s %3d\n", m, byMethod[m])
	}

	// The kernel's own assumptions, as distinct from registry values whose method is
	// "assumed", carry the set name KernelAssumptionSet.
	for _, o := range k.Provenance() {
		if o.Set == latencykernel.KernelAssumptionSet {
			fmt.Println("the kernel assumed", o.Name)
		}
	}
	// --8<-- [end:evidence]

}

Output:
109 of 151 values rest on evidence; 41 were assumed
  assumed       41
  measured      94
  not_charged    1
  vendor_spec   15
the kernel assumed max_cudagraph_capture_size
the kernel assumed engine_version

func (*Kernel) FixedBytes

func (k *Kernel) FixedBytes() kernel.MemoryBreakdown

FixedBytes returns occupancy independent of the request set, per rank.

Example

Memory: what one rank holds before any request arrives, and what each request adds.

package main

import (
	"fmt"

	latencykernel "github.com/inference-sim/blis-latency-kernel"
)

// --8<-- [start:repos]
// The roots a scenario's names resolve against: its model, chip, fabric and coefficient sets.
var repos = latencykernel.Repos{
	Scenarios: "testdata",
	Catalog:   "testdata/catalog",
	Registry:  "testdata/registry",
}

func main() {
	k, err := latencykernel.Open("aisimulate/deepseek-v3-h200-fp8-sglang-tp8.yaml", repos)
	if err != nil {
		panic(err)
	}
	// --8<-- [start:memory]
	const gib = 1 << 30
	m := k.FixedBytes()
	fmt.Printf("weights      %5.1f GiB\n", float64(m.Weights)/gib)
	fmt.Printf("activations  %5.1f GiB\n", float64(m.ActivationPeak)/gib)
	fmt.Printf("CUDA graphs  %5.1f GiB\n", float64(m.CUDAGraph)/gib)
	fmt.Printf("comm buffers %5.1f GiB\n", float64(m.CommBuffers)/gib)
	fmt.Printf("EPLB         %5.1f GiB\n", float64(m.EPLBRedundant)/gib)
	fmt.Printf("total        %5.1f GiB per rank\n", float64(m.Total())/gib)

	// KV cache for one request, rounded up to whole pages.
	fmt.Printf("a 4,096-token request: %.0f MiB of KV cache per rank\n",
		float64(k.SequenceVariableBytes(4096))/(1<<20))
	// --8<-- [end:memory]

}

Output:
weights       78.2 GiB
activations    1.1 GiB
CUDA graphs    0.8 GiB
comm buffers   3.9 GiB
EPLB           0.0 GiB
total         84.0 GiB per rank
a 4,096-token request: 137 MiB of KV cache per rank

func (*Kernel) OutputTokenOverhead

func (k *Kernel) OutputTokenOverhead() time.Duration

OutputTokenOverhead returns host time per emitted token.

func (*Kernel) PDTransferDemand

func (k *Kernel) PDTransferDemand(tokens int, from, to kernel.Placement) kernel.TransferDemand

PDTransferDemand is the handoff PDTransferTime prices, separated into payload, floor and the links it crosses, so a simulator running several handoffs at once can arbitrate the links they share. The kernel prices one transfer; dividing a shared link among several is the simulator's work.

Decode-context parallelism deliberately does NOT divide the payload. A prefill pool holds the whole cache for a request -- DCP shards the DECODE cache, and vLLM's own wording is "Number of ranks that shard the decode KV cache" (vllm/config/parallel.py:359-362 at v0.31.0) -- so the bytes that cross between pools are the whole request's, whatever the decode pool's dcp width is.

The links mirror the three placements a handoff can have:

same node                          that node's NVLink domain, at the NVLink rate
same rack, one-domain NVLink rack  that rack's NVLink domain, at the NVLink rate
otherwise                          the sender's NIC egress and the receiver's NIC
                                   ingress, each at the NIC rate

Link IDs are derived from the placement alone, so every kernel of one deployment names a physical link the same way: a prefill pool's kernel and a decode pool's kernel describe one handoff with the same IDs, and two handoffs that cross one NIC name it alike.

The floor is the expert-group all-to-all floor, which is what PDTransferTime has always charged. Whether that is the right floor for a KV handoff is a separate question: changing it would move every P/D price, so it is not changed here.

func (*Kernel) PDTransferTime

func (k *Kernel) PDTransferTime(tokens int, from, to kernel.Placement) time.Duration

PDTransferTime prices moving one request's KV between pools: its demand, run alone.

Computed from PDTransferDemand rather than beside it, so the two cannot drift: the blis-schemas contract is that a demand alone costs exactly this price, for every argument.

func (*Kernel) PrefillContextParallelWidth

func (k *Kernel) PrefillContextParallelWidth() int

PrefillContextParallelWidth returns how many ranks split a prefill sequence.

The companion to DecodeContextParallelWidth, and the two are genuinely independent: PCP splits prefill computation and expands the process world size while leaving the KV cache replicated, where DCP shards the cache without expanding the world size -- it reuses the tensor-parallel ranks when pcp is 1, and spans the PCP axis or the whole TP x PCP block otherwise (vllm/config/parallel.py:359-362 at v0.31.0). A consumer sizing a deployment needs both, and neither can be derived from the other.

Interim for the same reason as DecodeContextParallelWidth.

func (*Kernel) Provenance

func (k *Kernel) Provenance() []kernel.CoefficientOrigin

Provenance reports every coefficient this kernel used and its evidence.

func (*Kernel) Resolved

func (k *Kernel) Resolved() kernel.Resolution

Resolved reports the configuration after resolution, including overridden requests.

func (*Kernel) SequenceFixedBytes

func (k *Kernel) SequenceFixedBytes() int64

SequenceFixedBytes returns per-sequence bytes that do not vary with prompt length: recurrent state, where the model has it and the cache mode keeps it bounded, and the bounded caches a graph states -- window rings and fixed state (cacheFixedBytes).

func (*Kernel) SequenceVariableBytes

func (k *Kernel) SequenceVariableBytes(tokens int) int64

SequenceVariableBytes returns per-sequence KV bytes for a token count, quantized to the engine's page size.

func (*Kernel) StepTime

func (k *Kernel) StepTime(b kernel.Batch) kernel.StepEstimate

StepTime prices one forward pass. This is the hot path.

The returned StepEstimate carries a freshly allocated PerResource map. A simulator that calls this per simulated step, millions of times, can use StepTimeInto instead and reuse one map across calls, which removes that allocation.

func (*Kernel) StepTimeInto

func (k *Kernel) StepTimeInto(b kernel.Batch,
	per map[kernel.Resource]time.Duration) kernel.StepEstimate

StepTimeInto prices one forward pass, writing the per-resource breakdown into per instead of allocating a map.

per is cleared first, so a caller reuses one map for every step. Passing nil is the same as calling StepTime: a new map is allocated.

The map is the allocation a caller can remove. The step still allocates a few scratch slices that grow with the number of requests -- the prefill chunks, the decode contexts, the prefills a context gather packs -- and BenchmarkStepTimeIntoDecode reports how many.

func (*Kernel) TierTime

func (k *Kernel) TierTime(tier string, dir kernel.Direction, bytes int64,
	queueDepth int) time.Duration

TierTime prices one transfer between GPU and an offload tier.

type Repos

type Repos struct {
	Scenarios string // directory holding scenario files
	Catalog   string // a blis-catalog checkout, or a pinned copy of one
	Registry  string // a blis-registry checkout, or a pinned copy of one
}

Repos locates the roots a scenario's names resolve against.

A scenario names its model, hardware, fabric and coefficient sets without saying where they live — it describes a deployment, not a filesystem — so a caller supplies the roots.