blisschemas

README

blis-schemas

Go schemas and validators for the BLIS cost model: what a model is, what hardware can do, what a coefficient claims, how far speculative decoding advances a request, what a deployment specifies, what a measured run recorded, and the interface a cost model implements.

Documentation: https://inference-sim.github.io/blis-schemas/

This repository holds the shapes of documents and the checks on them, and no data. The data lives in blis-catalog (declared facts) and blis-registry (coefficient sets and acceptance profiles), which run this module's validators in CI. blis-latency-kernel implements the Kernel interface defined here, and inference-sim calls it in pull request #1851.

Use

go get github.com/inference-sim/blis-schemas@latest

s, err := blisschemas.LoadScenario("scenario.yaml")
if err != nil {
    return err
}
d, err := blisschemas.LoadDeployment("deployment.yaml")
if err != nil {
    return err
}

rep := blisschemas.Validate(blisschemas.Bundle{Scenario: s, Deployment: d})
for _, p := range rep.Problems() {
    fmt.Println(p) // field problems first, then rule problems
}

Validate a catalog or a registry checkout:

go run github.com/inference-sim/blis-schemas/cmd/validate-catalog@latest  /path/to/blis-catalog
go run github.com/inference-sim/blis-schemas/cmd/validate-registry@latest /path/to/blis-registry

Develop

go test ./...                                              # schemas, validators, rules, docs
VLLM_SOURCE=/path/to/vllm-0.29.0 go test -run MatchSource ./rules/v0_29/ # each pack against its own release
VLLM_SOURCE=/path/to/vllm-0.31.0 go test -run MatchSource ./rules/v0_31/
AIPERF_SOURCE=/path/to/aiperf-0.12.0/src/aiperf go test -run MatchSource ./spec/workload/ # replay constants

How changes are organized, how an engine release is added, and how the documentation is built and published: see Contributing in the documentation.

License

Apache 2.0. See LICENSE.

Documentation

Overview

Package blisschemas is the entry point: it composes field-level validation with version-scoped rule validation over a complete set of documents.

The two layers stay visible in the result. A Report separates field problems from rule problems, because the two need different responses: a field problem is a malformed document and always the author's to fix, where a rule problem may mean the document is right and the engine version is wrong, or that a rule needs updating for a release it has not seen.

Field validation runs first and rule validation only if it passes. A rule reading a malformed document produces findings that are artifacts of the malformation, and a reader cannot tell those from real ones.

Index

• func LoadAcceptanceProfileSet(path string) (*acceptance.ProfileSet, error)
• func LoadCatalogWorkload(path string) (*workload.CatalogWorkload, error)
• func LoadChip(path string) (*hardware.Chip, error)
• func LoadCoefficientSet(path string) (*coefficient.Set, error)
• func LoadDeployment(path string) (*deployment.Deployment, error)
• func LoadEvaluationRun(path string) (*evaluation.Run, error)
• func LoadFabric(path string) (*hardware.Fabric, error)
• func LoadMeasuredRun(path string) (*measurement.MeasuredRun, error)
• func LoadMeasurementSources(path string) (*measurement.Registry, error)
• func LoadModelGraph(path string) (*model.Graph, error)
• func LoadModelIdentity(path string) (*model.Identity, error)
• func LoadRoutingProfileSet(path string) (*routing.ProfileSet, error)
• func LoadScenario(path string) (*scenario.Scenario, error)
• func LoadStorageDevices(path string) ([]*hardware.StorageDevice, error)
• func LoadWorkload(path string) (*workload.Shape, error)
• func ResolveReplay(catalogRoot string, r *workload.Replay) (*workload.Replay, error)
• type Bundle
• type Report
  • func Validate(b Bundle) Report
  • func ValidateAgainstCatalog(b Bundle, catalogRoot, registryRoot string) Report
  • func (r Report) OK() bool
  • func (r Report) Problems() []validate.Problem

Examples

• Validate
• Validate (Problems)
• Validate (Rules)
• ValidateAgainstCatalog

Functions

func LoadAcceptanceProfileSet

func LoadAcceptanceProfileSet(path string) (*acceptance.ProfileSet, error)

LoadAcceptanceProfileSet reads an acceptance-profile set from blis-registry's acceptance namespace. It does not validate; call Validate.

func LoadCatalogWorkload

func LoadCatalogWorkload(path string) (*workload.CatalogWorkload, error)

LoadCatalogWorkload reads one catalog workloads/<name>.yaml in either form: a traffic shape, or a replay workload when the file's top-level key is replay:. Both are named from the path stem, and both decode strictly. A replay file may carry the catalog's `_comment` keys at its top level, as a shape may.

func LoadChip

func LoadChip(path string) (*hardware.Chip, error)

LoadChip reads a chip descriptor from blis-catalog's hardware namespace. The file carries no name — identity is the filename — so the loader stamps Chip.Name from the path stem. The Chip type tags Name `yaml:"-"`, so the stamp is the field's only source: there is no in-file name to prefer, hence no `if Name == ""` guard, and an in-file `name` is rejected under strict decode (#27).

func LoadCoefficientSet

func LoadCoefficientSet(path string) (*coefficient.Set, error)

LoadCoefficientSet reads a coefficient set from blis-registry.

func LoadDeployment

func LoadDeployment(path string) (*deployment.Deployment, error)

LoadDeployment reads a deployment document — the tunable configuration applied to a scenario. It does not validate; call Validate.

func LoadEvaluationRun

func LoadEvaluationRun(path string) (*evaluation.Run, error)

LoadEvaluationRun reads a measured run.

func LoadFabric

func LoadFabric(path string) (*hardware.Fabric, error)

LoadFabric reads a fabric descriptor from blis-catalog's networks namespace, naming it from the filename as LoadChip does: the stamp is Fabric.Name's only source, and an in-file `name` is a rejected unknown field (#27).

func LoadMeasuredRun

func LoadMeasuredRun(path string) (*measurement.MeasuredRun, error)

LoadMeasuredRun reads a measured run from blis-catalog's measurements/<source>/. It does not validate; call ValidateWithSources with the registry beside it.

func LoadMeasurementSources

func LoadMeasurementSources(path string) (*measurement.Registry, error)

LoadMeasurementSources reads the licence registry, blis-catalog's measurements/sources.yaml.

func LoadModelGraph

func LoadModelGraph(path string) (*model.Graph, error)

LoadModelGraph reads a derived model graph.

func LoadModelIdentity

func LoadModelIdentity(path string) (*model.Identity, error)

LoadModelIdentity reads a model entry's identity manifest, blis-catalog's models/<name>/model.yaml: the name the entry claims and the provenance of the vendor config beside it. Unlike a chip, a fabric or a workload, this file carries its name IN the document — the catalog requires that name to match the directory — so the loader does not stamp it; the caller checks it against the directory, as it does for a graph.

func LoadRoutingProfileSet

func LoadRoutingProfileSet(path string) (*routing.ProfileSet, error)

LoadRoutingProfileSet reads a routing-profile set from blis-registry's routing namespace. It does not validate; call Validate.

func LoadScenario

func LoadScenario(path string) (*scenario.Scenario, error)

LoadScenario reads a scenario document. It does not validate; call Validate.

func LoadStorageDevices

func LoadStorageDevices(path string) ([]*hardware.StorageDevice, error)

LoadStorageDevices reads the offload-tier classes from blis-catalog's devices namespace.

The file is a mapping of tier name to device facts rather than a list, because a tier is referred to by name from a deployment's offload block. So this loader is not a plain decodeStrict: it walks the mapping and stamps each device's Name from its key, which is the only place that key survives into the loaded value.

Sorted by name on return, so two loads of one file yield the same order and a caller comparing kernels built from it is comparing configuration rather than map iteration.

func LoadWorkload

func LoadWorkload(path string) (*workload.Shape, error)

LoadWorkload reads a traffic shape from blis-catalog's workloads namespace. The file carries no name — identity is the filename — so the loader stamps Shape.Name from the path stem. The Shape type tags Name `yaml:"-"`, so the stamp is the field's only source: there is no in-file name to prefer, hence no `if Name == ""` guard, and an in-file `name` is rejected under strict decode. Chip and Fabric now follow this same rule (#27), so the three by-filename entities are single-sourced identically.

A replay workload (a file whose top-level key is replay:) is not a shape; it is refused here with a pointer to LoadCatalogWorkload, which reads either form.

func ResolveReplay

func ResolveReplay(catalogRoot string, r *workload.Replay) (*workload.Replay, error)

ResolveReplay returns the inline replay a binding's replay arm describes: an inline replay as it is, or a reference merged with the catalog replay workload it names, read from <catalogRoot>/workloads/<name>.yaml. It refuses a missing entry, an entry that is a shape, and a merge that is not a valid replay.

Types

type Bundle

type Bundle struct {
	Scenario     *scenario.Scenario
	Deployment   *deployment.Deployment
	Model        *model.Graph
	Chip         *hardware.Chip
	Fabric       *hardware.Fabric
	Devices      []*hardware.StorageDevice
	Coefficients []*coefficient.Set
	// AcceptanceProfiles are registry acceptance-profile sets, validated field by field.
	AcceptanceProfiles []*acceptance.ProfileSet
	// RoutingProfiles are registry routing-profile sets, validated field by field.
	RoutingProfiles []*routing.ProfileSet
	Workload        *workload.Shape
	// ReplayWorkload is a catalog replay workload, validated field by field.
	ReplayWorkload *workload.CatalogReplay
	Evaluation     *evaluation.Run
}

Bundle is a complete set of documents describing one run. Optional members are nil when a caller is validating a subset: a catalog contributor checks a chip alone, where a CI job checking a run supplies everything. The Scenario fixes the immutable problem and the Deployment the mutable configuration against it; validating both together is what checks that a deployment fits its cluster.

type Report

type Report struct {
	// Field holds structural problems: malformed documents, unrecognized vocabulary,
	// arithmetic that does not close. These are version-independent.
	Field *validate.Problems
	// Rule holds version-scoped problems, each attributed to a named rule.
	Rule *validate.Problems
	// RulesApplied is the engine version whose pack ran, empty if none did.
	RulesApplied string
}

Report is the outcome of validating a bundle, with the two layers kept apart.

func Validate

func Validate(b Bundle) Report

Validate checks a bundle: every present document field-by-field, then the version-scoped rules if the field layer passed.

Example

ExampleValidate validates a scenario and a deployment together with the catalog documents they name.

package main

import (
	"fmt"

	blisschemas "github.com/inference-sim/blis-schemas"
)

func must[T any](v T, err error) T {
	if err != nil {
		panic(err)
	}
	return v
}

func main() {
	// --8<-- [start:validate]
	s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
	d := must(blisschemas.LoadDeployment("docs/examples/deployment.yaml"))
	g := must(blisschemas.LoadModelGraph("testdata/models/llama-3.1-8b-instruct/graph.yaml"))
	c := must(blisschemas.LoadChip("testdata/hardware/h100.yaml"))

	rep := blisschemas.Validate(blisschemas.Bundle{
		Scenario: s, Deployment: d, Model: g, Chip: c,
	})
	for _, p := range rep.Problems() {
		fmt.Println(p)
	}
	fmt.Println("ok:", rep.OK(), "rules applied:", rep.RulesApplied)
	// --8<-- [end:validate]

}

Output:
ok: true rules applied: 0.29.0

Example (Problems)

ExampleValidate_problems shows what a failing bundle reports: every problem at once, each at the path of the field that is wrong.

package main

import (
	"fmt"

	blisschemas "github.com/inference-sim/blis-schemas"
)

func must[T any](v T, err error) T {
	if err != nil {
		panic(err)
	}
	return v
}

func main() {
	s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
	d := must(blisschemas.LoadDeployment("docs/examples/deployment.yaml"))
	// --8<-- [start:problems]
	pool := &d.Pools[0]
	pool.Parallel.TP = 8  // 8 x 4 data-parallel ranks needs 32 GPUs
	pool.Parallel.DCP = 3 // does not divide tp
	pool.Engine.GPUMemoryUtilization = 1.5

	rep := blisschemas.Validate(blisschemas.Bundle{Scenario: s, Deployment: d})
	for _, p := range rep.Problems() {
		fmt.Println(p)
	}
	// --8<-- [end:problems]

}

Output:
error: deployment.pools[0].engine.gpu_memory_utilization: must lie in [0, 1] (0 means unset), got 1.5
error: deployment.pools[0].parallel: needs 32 GPUs (pp 1 x tp 8 x pcp 1 x dp 4, one per rank) but the pool's 1 node(s) of 8 GPUs provide 8
error: deployment.pools[0].parallel.dcp: decode-context parallelism reuses the tensor-parallel ranks when prefill-context parallelism is off, so tp 8 must be divisible by dcp 3

Example (Rules)

ExampleValidate_rules shows a rule problem: the documents are well formed, but the engine version they name would refuse them.

package main

import (
	"fmt"

	blisschemas "github.com/inference-sim/blis-schemas"
)

func must[T any](v T, err error) T {
	if err != nil {
		panic(err)
	}
	return v
}

func main() {
	s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
	d := must(blisschemas.LoadDeployment("docs/examples/deployment.yaml"))
	// --8<-- [start:rules]
	d.Pools[0].Engine.CacheDType = "fp8_e4m3fn" // not a name vLLM 0.29.0 accepts

	rep := blisschemas.Validate(blisschemas.Bundle{Scenario: s, Deployment: d})
	fmt.Println("field layer ok:", rep.Field.OK())
	for _, p := range rep.Rule.All() {
		fmt.Println(p)
	}
	// --8<-- [end:rules]

}

Output:
field layer ok: true
error [enum-values-known]: : pools[0].engine.cache_dtype: "fp8_e4m3fn" is not accepted by engine 0.29.0

func ValidateAgainstCatalog

func ValidateAgainstCatalog(b Bundle, catalogRoot, registryRoot string) Report

ValidateAgainstCatalog resolves every by-name reference a bundle's scenario makes against a catalog and registry on disk. It is separate from Validate on purpose: Validate is pure, and this repository deliberately does not vendor the catalog, so a caller without one checked out gets the pure checks and one with a catalog gets the references resolved too. The README's "two layers" split rests on that separation — keeping the shapes apart from the contents is what lets either side be validated in CI without the other being present — so resolution lives behind its own entry point rather than inside Validate.

It checks presence, not validity: that model names a directory the catalog has, that hardware, fabric and a workload shape name files it has, and that every storage tier, coefficient set and acceptance-profile set a scenario names exists. A replay arm that references a catalog replay workload is loaded and merged with its run choices (ResolveReplay), since a reference that resolves to an invalid replay is a miss too. The Trace arm and an inline Replay are excluded on purpose — they reference external captures by path or repository, not catalog identities. Validating the resolved documents is Validate's job, reached by loading them into a Bundle; this answers the prior question of whether the names resolve at all, which nothing else does — `hardware: nvidia-h200` validates clean today and fails later, elsewhere, as a missing file rather than an invalid scenario.

Every failure names the path it looked for and enumerates what the catalog actually has, so the error teaches the naming convention in one turn rather than leaving a generator to guess: `hardware: nvidia-h200` → "looked for hardware/nvidia-h200.yaml; the catalog has [a100-80 a100-sxm b200 h100 h200 ...]".

A nil scenario resolves nothing and is reported as such rather than passing silently: a bundle with no scenario names no references, so there is nothing to resolve, but a caller that meant to pass one should not read an empty green as success. An empty catalogRoot or registryRoot is only an error when a reference of the corresponding kind is actually made.

Problems are recorded under dotted paths below "references." — references.hardware, references.coefficients[1] — matching how spec/ validators report, so a reader of a mixed report sees resolution findings in the same shape as field findings.

Example

ExampleValidateAgainstCatalog resolves a scenario's by-name references against a catalog checkout and a registry checkout.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	blisschemas "github.com/inference-sim/blis-schemas"
)

func must[T any](v T, err error) T {
	if err != nil {
		panic(err)
	}
	return v
}

func main() {
	// A stand-in registry: resolution checks only that each named set's file exists.
	registry := must(os.MkdirTemp("", "registry"))
	defer os.RemoveAll(registry)
	must(0, os.MkdirAll(filepath.Join(registry, "coefficients"), 0o755))
	for _, name := range []string{"cost-model-primitives", "cost-model-collectives",
		"cost-model-host-overheads", "cost-model-attention", "cost-model-recurrent"} {
		must(0, os.WriteFile(filepath.Join(registry, "coefficients", name+".yaml"), nil, 0o644))
	}

	// --8<-- [start:resolve]
	s := must(blisschemas.LoadScenario("docs/examples/scenario.yaml"))
	s.Cluster.Hardware = "nvidia-h100" // the catalog calls it h100

	rep := blisschemas.ValidateAgainstCatalog(
		blisschemas.Bundle{Scenario: s}, "testdata", registry)
	for _, p := range rep.Problems() {
		fmt.Println(p)
	}
	// --8<-- [end:resolve]

}

Output:
error: references.hardware: "nvidia-h100": looked for hardware/nvidia-h100.yaml; the catalog has [a100-80 a100-sxm b200 b300 gb200-nvl72 h100 h200 l40s]

func (Report) OK

func (r Report) OK() bool

OK reports whether nothing failed in either layer.

func (Report) Problems

func (r Report) Problems() []validate.Problem

Problems returns every finding from both layers, field problems first.