kueue-bench

README

kueue-bench

A CLI tool for creating and managing local Kueue test environments using kind and kwok.

Prerequisites

To use kueue-bench:

• Docker

To build from source:

• Go (v1.21+)
• Make

kueue-bench uses kind and Helm as Go libraries (no external CLI dependencies required). Kind and Helm versions are compiled into the binary.

Quick Start

Create a Topology

Use one of the example configurations:

kueue-bench topology create single-cluster \
  --file examples/topologies/single-cluster.yaml

This creates a single kind cluster with:

• 10 simulated CPU nodes (via Kwok)
• Kueue controller and CRDs installed
• A simple CPU-only ResourceFlavor, ClusterQueue, and LocalQueue

List Topologies

List currently running topologies (topology metadata is stored in ~/.kueue-bench/topologies/

kueue-bench topology list

Test with a sample job

Node pools in the cluster are tainted with kwok.x-k8s.io/node to prevent real workloads from running on them (e.g. the Kueue controller), so be sure to add a toleration. Pod lifecycle is completely simulated and managed by Kwok stages, so any logic will not actually run.

cat <<EOF | kubectl create -f -
apiVersion: batch/v1
kind: Job
metadata:
  generateName: test-job-
  namespace: default
spec:
  parallelism: 2
  completions: 2
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default-lq
    spec:
      containers:
      - name: sleep
        image: busybox
        command: ["sleep", "10"]
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
      tolerations:
      - key: kwok.x-k8s.io/node
        operator: Equal
        value: "fake"
        effect: NoSchedule
      restartPolicy: Never
EOF

# Check that Kueue admitted the workload
kubectl get workloads -A

Watch with the TUI (experimental)

Launch an interactive terminal UI connected to a running topology:

kueue-bench tui --topology single-cluster

Shows live ClusterQueue utilization, workload admission status, and Kueue events. Key bindings:

• 1/2 — switch between Queues and Workloads tabs
• Enter — drill into a queue or workload for detail
• s — submit an ad-hoc job
• c — switch clusters (multi-cluster topologies)
• Esc / q — go back / quit

Delete a Topology

Clean up when you're done:

kueue-bench topology delete single-cluster

Installation

Pre-built Binaries

Download the latest release from the GitHub Releases page. Binaries are available for Linux and macOS (amd64/arm64).

Build from Source

git clone https://github.com/jhwagner/kueue-bench.git
cd kueue-bench
make install

This will install kueue-bench to your Go binaries directory, typically $HOME/go/bin/ (make sure this is in your $PATH).

Alternatively, for local development builds, use:

make build

This will place the binary at ./bin/kueue-bench.

Examples

See the examples/ directory for topology and workload configurations:

Topologies (examples/topologies/):

• single-cluster.yaml — Basic single cluster setup
• single-cluster-gpu.yaml — Multi-pool setup with CPU and GPU nodes
• single-cluster-with-extensions.yaml — Installing additional components (e.g. JobSet) via Helm
• multikueue.yaml — MultiKueue topology with WorkerSets and automatic quota derivation
• basic-queue.yaml — Single-queue CPU cluster; paired with the basic-queue workload profile
• cohort-borrowing.yaml — Two-tenant GPU cohort demonstrating idle quota lending
• fair-share-contention.yaml — Three-tenant GPU cohort with fair-sharing under sustained contention

Workload Profiles (examples/workloads/):

• basic-queue.yaml — CPU jobs targeting a single queue; ~61% steady-state utilization
• cohort-borrowing.yaml — GPU jobs showing Team B bursting into Team A's idle quota
• fair-share-contention.yaml — GPU jobs showing proportional borrowing under oversubscription

Configuration

See Topology Schema for the full configuration reference.

Development

# Build binary to ./bin/
make build

# Run tests
make test

# Format and verify
make verify

Project Structure

kueue-bench/
├── cmd/kueue-bench/    # CLI entry point and commands
├── pkg/                # Core library packages
│   ├── config/         # Topology schema and parsing
│   ├── cluster/        # kind cluster management
│   ├── kwok/           # Kwok installation and nodes
│   ├── kueue/          # Kueue installation and resources
│   └── topology/       # Topology orchestration
├── examples/           # Example topology and workload files
│   ├── topologies/     # Topology configuration examples
│   └── workloads/      # Workload profile examples
└── docs/               # Documentation

Planned Features

• Metrics and dashboards for visualizing results
• Benchmark runner and automated reports for quantitative analysis of Kueue configurations