README
¶
inference-plane
Reference implementation of the Control Plane built throughout Inference Is All You Need: A Systems Engineer's Guide (Apress, 2026).
This repo and the book are paired in the Tanenbaum / MINIX style — the book teaches principles with readable code that shows the shape; this repo is the production-grade implementation. See ARCHITECTURE.md for the design rationale and the deployment topology.
The repo evolves alongside the book. Each Part of the book corresponds to a versioned release of the control plane:
| Version | Chapter | Adds |
|---|---|---|
| v0.1 | 6 | Backend abstraction, gRPC + OpenAI HTTP, observability |
| v0.2 | 7–10 | Auth, rate limiting, caching, queueing |
| v0.3 | 12–15 | Multi-tenancy, billing, multi-backend routing |
| v1.0 | 18–19 | Production hardening, autoscaling, full platform |
Each version is published as a maintained branch (release/vX.Y) with
fixes cherry-picked from main, plus an immutable tag (vX.Y.0)
marking the chapter's text-match snapshot. Check out the branch by
default; pin to the tag if you need byte-identical match to the book.
v0.1 (this branch)
Single-backend control plane proxying to vLLM serving Qwen 3.5 8B, exposed on two transport surfaces:
- gRPC + Connect-RPC at
/inferenceplane.v1.{Service}/{Method} - OpenAI-compatible HTTP at
/v1/completions,/v1/chat/completions,/health
Observability via the OpenTelemetry Collector exporting to a local Grafana stack (Tempo, Loki, Mimir).
Quick start
Local dev (any host, no GPU) — control plane + full observability stack with the mock backend:
git checkout release/v0.1
make up # builds controlplane + brings up obs stack (skips vllm)
make load # synthetic traffic so dashboards populate
make dashboards # open Grafana at http://localhost:3000
Mock backend returns synthetic OpenAI-shaped responses with realistic latency and token counts. Runs on Mac, Linux, CI — no GPU required.
Real inference (NVIDIA GPU host) — switch the engine to vllm and bring the stack up with the gpu profile:
# In deploy/config.yaml: backend.engine: "vllm"
docker compose --profile gpu --env-file pinned-versions.env -f deploy/docker-compose.yaml up -d --build
make smoke # Go integration tests against the live stack
See Makefile for the full target list, or run make help.
Repository layout
cmd/iplane/ single binary with subcommands:
iplane serve run the control plane
iplane instance create / list / describe / destroy GPU instances
iplane load fire synthetic traffic
iplane gen-names regenerate OTel name vocabulary
protos/ buf-managed proto sources
inferenceplane/v1/ service + type definitions
gen/ generated proto code (committed; regenerate via
`cd protos && make gen`)
internal/
backends/ Backend interface + vLLM implementation
config/ YAML + env config loading
services/ gRPC service implementations
telemetry/ OTel SDK setup + generated names.go
web/server/ HTTP layer: connect-rpc adapters + grpc-gateway
deploy/
config.yaml reference deployment config (mounted into compose)
docker-compose.yaml vLLM + control plane + OTel collector + Grafana stack
grafana/ provisioned dashboards and datasources
*-config.yaml per-component configuration
tests/smoke/ Go integration tests (build-tag gated)
metric-names.yaml canonical OTel name vocabulary (paired with book)
pinned-versions.env model + engine + stack versions (paired with book)
providers.yaml multi-provider rate table for cost gauges
Topology
One process, two listeners. The gRPC server is the source of truth for the API; HTTP surfaces (gRPC-gateway and Connect-RPC) are bindings on top, both calling the same in-process gRPC server. See ARCHITECTURE.md for details.
License
MIT. See LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
iplane
command
Command iplane is the operator and developer CLI for the inference control plane.
|
Command iplane is the operator and developer CLI for the inference control plane. |
|
iplane/cmd
Package cmd contains the cobra commands for the iplane CLI.
|
Package cmd contains the cobra commands for the iplane CLI. |
|
gen
|
|
|
internal
|
|
|
backends
Package backend defines the abstraction every inference engine must satisfy and the OpenAI-shaped request and response types that flow through the control plane.
|
Package backend defines the abstraction every inference engine must satisfy and the OpenAI-shaped request and response types that flow through the control plane. |
|
config
Package config holds the typed Config struct that the iplane serve subcommand assembles from viper (file + env + flags + defaults).
|
Package config holds the typed Config struct that the iplane serve subcommand assembles from viper (file + env + flags + defaults). |
|
deployments/sshdocker
Package sshdocker implements iplane's deployment executor for v0.1: SSH into a provisioned instance and run the engine container via docker CLI commands over the SSH session.
|
Package sshdocker implements iplane's deployment executor for v0.1: SSH into a provisioned instance and run the engine container via docker CLI commands over the SSH session. |
|
engineagent
Package engineagent is the agent half of the control channel: the thing that runs next to an engine on a rented box and tells the control plane that engine exists.
|
Package engineagent is the agent half of the control channel: the thing that runs next to an engine on a rented box and tells the control plane that engine exists. |
|
engines
Package engines is the control plane's record of what data planes exist, learned from what announced itself rather than from what the control plane remembers renting.
|
Package engines is the control plane's record of what data planes exist, learned from what announced itself rather than from what the control plane remembers renting. |
|
healthcheck
Package healthcheck runs a single goroutine inside iplane serve that probes each replica's /health endpoint on a tick and calls Quarantine/Restore on the deployment service when a replica crosses the failure or success threshold.
|
Package healthcheck runs a single goroutine inside iplane serve that probes each replica's /health endpoint on a tick and calls Quarantine/Restore on the deployment service when a replica crosses the failure or success threshold. |
|
imagearch
Package imagearch answers which CPU architectures a container image can run on, by asking the registry that holds it.
|
Package imagearch answers which CPU architectures a container image can run on, by asking the registry that holds it. |
|
metrics
Package metrics holds the OTel meter instruments emitted by the control plane and helpers that record into them.
|
Package metrics holds the OTel meter instruments emitted by the control plane and helpers that record into them. |
|
modelstores
Package modelstores defines how iplane transforms operator-supplied model specifications (HF ids, future: URLs, local paths, network- volume references) into the concrete form the engine actually consumes.
|
Package modelstores defines how iplane transforms operator-supplied model specifications (HF ids, future: URLs, local paths, network- volume references) into the concrete form the engine actually consumes. |
|
modelstores/huggingface
Package huggingface implements modelstores.ModelStore against the public Hugging Face Hub API.
|
Package huggingface implements modelstores.ModelStore against the public Hugging Face Hub API. |
|
modelstores/volumecache
Package volumecache implements a warm-cache ModelStore that mounts a pre-staged volume so the engine loads weights off it instead of re-downloading from HuggingFace on every deploy.
|
Package volumecache implements a warm-cache ModelStore that mounts a pre-staged volume so the engine loads weights off it instead of re-downloading from HuggingFace on every deploy. |
|
provisioners
Package provisioners hosts the operator-facing ProvisionerService (CLI-callable today as an in-process struct, registerable with a connect/grpc mux later), the Go-internal Provider interface every per-provider adapter satisfies, and the helpers (ID validation, tag-key constants, error wrapping) shared across both.
|
Package provisioners hosts the operator-facing ProvisionerService (CLI-callable today as an in-process struct, registerable with a connect/grpc mux later), the Go-internal Provider interface every per-provider adapter satisfies, and the helpers (ID validation, tag-key constants, error wrapping) shared across both. |
|
provisioners/enginewait
Package enginewait holds the poll loop an image-native provider runs between renting a box and the engine answering /health.
|
Package enginewait holds the poll loop an image-native provider runs between renting a box and the engine answering /health. |
|
provisioners/external
Package external implements a non-owning Provider: it registers a RUNNING replica pointing at an engine URL the operator runs themselves (on-prem vLLM, a k8s Service, or a local `iplane mock-engine`), rather than provisioning anything.
|
Package external implements a non-owning Provider: it registers a RUNNING replica pointing at an engine URL the operator runs themselves (on-prem vLLM, a k8s Service, or a local `iplane mock-engine`), rather than provisioning anything. |
|
provisioners/fabric
Package fabric holds the cross-provider knowledge of what GPU interconnect a card is capable of, and the rules for deciding whether a candidate satisfies an operator's fabric requirement.
|
Package fabric holds the cross-provider knowledge of what GPU interconnect a card is capable of, and the rules for deciding whether a candidate satisfies an operator's fabric requirement. |
|
provisioners/lambdalabs
Package lambdalabs implements the Provider interface against Lambda Labs Cloud's REST API (https://cloud.lambdalabs.com/api/v1).
|
Package lambdalabs implements the Provider interface against Lambda Labs Cloud's REST API (https://cloud.lambdalabs.com/api/v1). |
|
provisioners/lifecycle
Package lifecycle holds long-running goroutines the daemon launches to maintain provisioner state-of-record beyond per-request handlers.
|
Package lifecycle holds long-running goroutines the daemon launches to maintain provisioner state-of-record beyond per-request handlers. |
|
provisioners/local
Package local implements the Provider interface against the operator's laptop.
|
Package local implements the Provider interface against the operator's laptop. |
|
provisioners/providertest
Package providertest carries the conformance suites every Provider adapter has to pass, so a contract the Service depends on is asserted in each adapter's own test package rather than being discovered from the Service's behaviour.
|
Package providertest carries the conformance suites every Provider adapter has to pass, so a contract the Service depends on is asserted in each adapter's own test package rather than being discovered from the Service's behaviour. |
|
provisioners/runpod
Package runpod implements the Provider interface against RunPod's REST API v2 (https://api.runpod.io/v2).
|
Package runpod implements the Provider interface against RunPod's REST API v2 (https://api.runpod.io/v2). |
|
provisioners/skucatalog
Package skucatalog holds the resolver every provider adapter runs over its own GPU catalog: filter the rows by the operator's numeric constraints, drop what cannot carry the requested fabric, order by price, cap the list.
|
Package skucatalog holds the resolver every provider adapter runs over its own GPU catalog: filter the rows by the operator's numeric constraints, drop what cannot carry the requested fabric, order by price, cap the list. |
|
provisioners/stores/file
Package file is the file-backed impl of provisioners.Store.
|
Package file is the file-backed impl of provisioners.Store. |
|
provisioners/vast
Package vast implements the Provider interface against Vast.ai's REST API (https://console.vast.ai/api/).
|
Package vast implements the Provider interface against Vast.ai's REST API (https://console.vast.ai/api/). |
|
router
Package router is the v0.2 data-plane primitive: an HTTP handler that puts iplane back into the inference request path.
|
Package router is the v0.2 data-plane primitive: an HTTP handler that puts iplane back into the inference request path. |
|
router/policy
Package policy is the routing-decision seam that decouples per-request replica selection from the rest of the router.
|
Package policy is the routing-decision seam that decouples per-request replica selection from the rest of the router. |
|
scheduler
Package scheduler is the v0.2 Beat 2.4 dequeue-and-dispatch layer: it sits between the router's queue lanes (Beat 2.3) and the engine, picking which entry runs next and enforcing a per- deployment in-flight cap.
|
Package scheduler is the v0.2 Beat 2.4 dequeue-and-dispatch layer: it sits between the router's queue lanes (Beat 2.3) and the engine, picking which entry runs next and enforcing a per- deployment in-flight cap. |
|
series
Package series loads and validates the Part IV measurement registry.
|
Package series loads and validates the Part IV measurement registry. |
|
services
Package services implements the gRPC service interfaces generated from the protos in inferenceplane.v1.
|
Package services implements the gRPC service interfaces generated from the protos in inferenceplane.v1. |
|
sshkeys
Package sshkeys handles iplane-managed SSH key pairs.
|
Package sshkeys handles iplane-managed SSH key pairs. |
|
stores/queue
Package queue provides a bounded FIFO interface and an M/M/k worker pool that drains it.
|
Package queue provides a bounded FIFO interface and an M/M/k worker pool that drains it. |
|
stores/queue/inmem
Package inmem is the in-process FIFO implementation of queue.BoundedQueue.
|
Package inmem is the in-process FIFO implementation of queue.BoundedQueue. |
|
telemetry
Package telemetry initializes the OpenTelemetry SDK: tracer provider, meter provider, and resource attributes describing this deployment.
|
Package telemetry initializes the OpenTelemetry SDK: tracer provider, meter provider, and resource attributes describing this deployment. |
|
version
Package version carries the build's identity.
|
Package version carries the build's identity. |
|
vrambudget
Package vrambudget computes whether a model will fit on the cards you are about to rent, before you rent them.
|
Package vrambudget computes whether a model will fit on the cards you are about to rent, before you rent them. |
|
web/server
Package server wires the HTTP-side bindings on the daemon's HTTP listener.
|
Package server wires the HTTP-side bindings on the daemon's HTTP listener. |
|
tests
|
|
|
smoke-providers
Package smokeproviders holds the shared real-API smoke-test core that every per-provider package (tests/smoke-runpod, smoke-vast, future smoke-lambdalabs, ...) wraps thinly.
|
Package smokeproviders holds the shared real-API smoke-test core that every per-provider package (tests/smoke-runpod, smoke-vast, future smoke-lambdalabs, ...) wraps thinly. |
Click to show internal directories.
Click to hide internal directories.