inference-plane

module

v0.1.0 Latest Latest Go to latest Published: Jun 1, 2026 License: MIT

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/inference-book/inference-plane

Links

Open Source Insights

README ¶

inference-plane

Reference implementation of the Control Plane built throughout Inference Is All You Need: A Systems Engineer's Guide (Apress, 2026).

This repo and the book are paired in the Tanenbaum / MINIX style — the book teaches principles with readable code that shows the shape; this repo is the production-grade implementation. See ARCHITECTURE.md for the design rationale and the deployment topology.

The repo evolves alongside the book. Each Part of the book corresponds to a versioned release of the control plane:

Version	Chapter	Adds
v0.1	6	Backend abstraction, gRPC + OpenAI HTTP, observability
v0.2	7–10	Auth, rate limiting, caching, queueing
v0.3	12–15	Multi-tenancy, billing, multi-backend routing
v1.0	18–19	Production hardening, autoscaling, full platform

Each version is published as a maintained branch (release/vX.Y) with fixes cherry-picked from main, plus an immutable tag (vX.Y.0) marking the chapter's text-match snapshot. Check out the branch by default; pin to the tag if you need byte-identical match to the book.

v0.1 (this branch)

Single-backend control plane proxying to vLLM serving Qwen 3.5 8B, exposed on two transport surfaces:

gRPC + Connect-RPC at /inferenceplane.v1.{Service}/{Method}
OpenAI-compatible HTTP at /v1/completions, /v1/chat/completions, /health

Observability via the OpenTelemetry Collector exporting to a local Grafana stack (Tempo, Loki, Mimir).

Quick start

Local dev (any host, no GPU) — control plane + full observability stack with the mock backend:

git checkout release/v0.1
make up         # builds controlplane + brings up obs stack (skips vllm)
make load       # synthetic traffic so dashboards populate
make dashboards # open Grafana at http://localhost:3000

Mock backend returns synthetic OpenAI-shaped responses with realistic latency and token counts. Runs on Mac, Linux, CI — no GPU required.

Real inference (NVIDIA GPU host) — switch the engine to vllm and bring the stack up with the gpu profile:

# In deploy/config.yaml: backend.engine: "vllm"
docker compose --profile gpu --env-file pinned-versions.env -f deploy/docker-compose.yaml up -d --build
make smoke      # Go integration tests against the live stack

See Makefile for the full target list, or run make help.

Repository layout

cmd/iplane/           single binary with subcommands:
                        iplane serve      run the control plane
                        iplane instance   create / list / describe / destroy GPU instances
                        iplane load       fire synthetic traffic
                        iplane gen-names  regenerate OTel name vocabulary
protos/               buf-managed proto sources
  inferenceplane/v1/  service + type definitions
gen/                  generated proto code (committed; regenerate via
                       `cd protos && make gen`)
internal/
  backends/           Backend interface + vLLM implementation
  config/             YAML + env config loading
  services/           gRPC service implementations
  telemetry/          OTel SDK setup + generated names.go
  web/server/         HTTP layer: connect-rpc adapters + grpc-gateway
deploy/
  config.yaml         reference deployment config (mounted into compose)
  docker-compose.yaml vLLM + control plane + OTel collector + Grafana stack
  grafana/            provisioned dashboards and datasources
  *-config.yaml       per-component configuration
tests/smoke/          Go integration tests (build-tag gated)
metric-names.yaml     canonical OTel name vocabulary (paired with book)
pinned-versions.env   model + engine + stack versions (paired with book)
providers.yaml        multi-provider rate table for cost gauges

Topology

One process, two listeners. The gRPC server is the source of truth for the API; HTTP surfaces (gRPC-gateway and Connect-RPC) are bindings on top, both calling the same in-process gRPC server. See ARCHITECTURE.md for details.

License

Apache 2.0. See LICENSE.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
cmd
iplane command Command iplane is the operator and developer CLI for the inference control plane.	Command iplane is the operator and developer CLI for the inference control plane.
iplane/cmd Package cmd contains the cobra commands for the iplane CLI.	Package cmd contains the cobra commands for the iplane CLI.
examples
01-end-to-end command Example: end-to-end provisioning walkthrough.	Example: end-to-end provisioning walkthrough.
02-cli-end-to-end command Example: end-to-end provisioning walkthrough driving the iplane CLI binary (no separate server, no gRPC client construction).	Example: end-to-end provisioning walkthrough driving the iplane CLI binary (no separate server, no gRPC client construction).
03-deploy-end-to-end command Example: end-to-end deployment walkthrough.	Example: end-to-end deployment walkthrough.
gen
go/inferenceplane/v1
go/inferenceplane/v1/inferenceplanev1connect
go/provisioner/v1
go/provisioner/v1/provisionerv1connect
internal
backends Package backend defines the abstraction every inference engine must satisfy and the OpenAI-shaped request and response types that flow through the control plane.	Package backend defines the abstraction every inference engine must satisfy and the OpenAI-shaped request and response types that flow through the control plane.
config Package config holds the typed Config struct that the iplane serve subcommand assembles from viper (file + env + flags + defaults).	Package config holds the typed Config struct that the iplane serve subcommand assembles from viper (file + env + flags + defaults).
deployments/sshdocker Package sshdocker implements iplane's deployment executor for v0.1: SSH into a provisioned instance and run the engine container via docker CLI commands over the SSH session.	Package sshdocker implements iplane's deployment executor for v0.1: SSH into a provisioned instance and run the engine container via docker CLI commands over the SSH session.
metrics Package metrics holds the OTel meter instruments emitted by the control plane and helpers that record into them.	Package metrics holds the OTel meter instruments emitted by the control plane and helpers that record into them.
modelstores Package modelstores defines how iplane transforms operator-supplied model specifications (HF ids, future: URLs, local paths, network- volume references) into the concrete form the engine actually consumes.	Package modelstores defines how iplane transforms operator-supplied model specifications (HF ids, future: URLs, local paths, network- volume references) into the concrete form the engine actually consumes.
modelstores/huggingface Package huggingface implements modelstores.ModelStore against the public Hugging Face Hub API.	Package huggingface implements modelstores.ModelStore against the public Hugging Face Hub API.
provisioners Package provisioners hosts the operator-facing ProvisionerService (CLI-callable today as an in-process struct, registerable with a connect/grpc mux later), the Go-internal Provider interface every per-provider adapter satisfies, and the helpers (ID validation, tag-key constants, error wrapping) shared across both.	Package provisioners hosts the operator-facing ProvisionerService (CLI-callable today as an in-process struct, registerable with a connect/grpc mux later), the Go-internal Provider interface every per-provider adapter satisfies, and the helpers (ID validation, tag-key constants, error wrapping) shared across both.
provisioners/local Package local implements the Provider interface against the operator's laptop.	Package local implements the Provider interface against the operator's laptop.
provisioners/runpod Package runpod implements the Provider interface against RunPod's REST API (https://rest.runpod.io/v1).	Package runpod implements the Provider interface against RunPod's REST API (https://rest.runpod.io/v1).
provisioners/state Package state holds the iplane state file and the read-modify-write loop the Service runs under.	Package state holds the iplane state file and the read-modify-write loop the Service runs under.
services Package services implements the gRPC service interfaces generated from the protos in inferenceplane.v1.	Package services implements the gRPC service interfaces generated from the protos in inferenceplane.v1.
sshkeys Package sshkeys handles iplane-managed SSH key pairs.	Package sshkeys handles iplane-managed SSH key pairs.
telemetry Package telemetry initializes the OpenTelemetry SDK: tracer provider, meter provider, and resource attributes describing this deployment.	Package telemetry initializes the OpenTelemetry SDK: tracer provider, meter provider, and resource attributes describing this deployment.
web/server Package server wires the HTTP-side bindings on top of the in-process gRPC server.	Package server wires the HTTP-side bindings on top of the in-process gRPC server.