kueue-bench

README

kueue-bench

A CLI tool for creating and managing local Kueue test environments using kind and kwok.

Prerequisites

To use kueue-bench:

• Docker

To build from source:

• Go (v1.21+)
• Make

kueue-bench uses kind and Helm as Go libraries (no external CLI dependencies required). Kind and Helm versions are compiled into the binary.

Quick Start

Create a Topology

Use one of the example configurations:

kueue-bench topology create single-cluster \
  --file examples/single-cluster.yaml

This creates a single kind cluster with:

• 10 simulated CPU nodes (via Kwok)
• Kueue controller and CRDs installed
• A simple CPU-only ResourceFlavor, ClusterQueue, and LocalQueue

List Topologies

List currently running topologies (topology metadata is stored in ~/.kueue-bench/topologies/

kueue-bench topology list

Test with a sample job

Node pools in the cluster are tainted with kwok.x-k8s.io/node to prevent real workloads from running on them (e.g. the Kueue controller), so be sure to add a toleration. Pod lifecycle is completely simulated and managed by Kwok stages, so any logic will not actually run.

cat <<EOF | kubectl create -f -
apiVersion: batch/v1
kind: Job
metadata:
  generateName: test-job-
  namespace: default
spec:
  parallelism: 2
  completions: 2
  template:
    metadata:
      labels:
        kueue.x-k8s.io/queue-name: default-lq
    spec:
      containers:
      - name: sleep
        image: busybox
        command: ["sleep", "10"]
        resources:
          requests:
            cpu: "1"
            memory: "1Gi"
      tolerations:
      - key: kwok.x-k8s.io/node
        operator: Equal
        value: "fake"
        effect: NoSchedule
      restartPolicy: Never
EOF

# Check that Kueue admitted the workload
kubectl get workloads -A

Delete a Topology

Clean up when you're done:

kueue-bench topology delete single-cluster

Installation

Pre-built Binaries

Download the latest release from the GitHub Releases page. Binaries are available for Linux and macOS (amd64/arm64).

Build from Source

git clone https://github.com/jhwagner/kueue-bench.git
cd kueue-bench
make install

This will install kueue-bench to your Go binaries directory, typically $HOME/go/bin/ (make sure this is in your $PATH).

Alternatively, for local development builds, use:

make build

This will place the binary at ./bin/kueue-bench.

Examples

See the examples/ directory:

• single-cluster.yaml — Basic single cluster setup
• single-cluster-gpu.yaml — Multi-pool setup with CPU and GPU nodes
• single-cluster-with-extensions.yaml — Installing additional components (e.g. JobSet) via Helm
• multikueue.yaml — MultiKueue topology with WorkerSets and automatic quota derivation

Configuration

See Topology Schema for the full configuration reference.

Development

# Build binary to ./bin/
make build

# Run tests
make test

# Format and verify
make verify

Project Structure

kueue-bench/
├── cmd/kueue-bench/    # CLI entry point and commands
├── pkg/                # Core library packages
│   ├── config/         # Topology schema and parsing
│   ├── cluster/        # kind cluster management
│   ├── kwok/           # Kwok installation and nodes
│   ├── kueue/          # Kueue installation and resources
│   └── topology/       # Topology orchestration
├── examples/           # Example topology files
└── docs/               # Documentation

Planned Features

• Multi-cluster topologies
• Workload generation for load simulation
• Metrics and dashboards for visualizing results
• Benchmark runner and automated reports for quantitative analysis of Kueue configurations