test-scale

README

test-scale

A distributed load testing platform that generates realistic, distributed load against a target system on demand — without manual provisioning of load-generation infrastructure.

test-scale is built for repeatable performance regression testing: submit a small YAML plan, pre-scale workers from the requested throughput, watch histogram-correct metrics, and compare the result with a previous run.

It is not a scripting engine. v0.1 supports one HTTP request per plan, including method, headers, body, timeout, expected statuses, and constant, ramp, step, or spike traffic. Multi-step journeys, browser testing, geo-distributed workers, and multi-tenant hosting are intentionally out of scope.

Architecture

flowchart LR
    CLI --> CP["Control Plane<br/>(Gin API + Orchestrator)"]
    CP --> PG["Postgres<br/>(plans / runs)"]
    CP --> NATS["NATS<br/>(job queue)"]
    CP --> AS["Autoscaler<br/>(Brain + Scaler)"]
    AS --> WP["Worker Pool<br/>(scalable containers)"]
    NATS --> WP
    WP --> Target["Target Under Test"]
    WP --> AGG["Aggregator<br/>(OTel receiver + histogram merge)"]
    AGG --> PROM["Prometheus"]
    PROM --> GF["Grafana<br/>(live + historical dashboards)"]

Installation

Homebrew (macOS / Linux)

brew install MohamedBenTaher/tap/test-scale

Docker (one-liner)

curl -sSL https://raw.githubusercontent.com/MohamedBenTaher/test-scale/main/setup.sh | bash

Binary download

Tagged releases publish binaries, checksums, SBOMs, and signatures on GitHub Releases.

Existing Kubernetes cluster

helm install test-scale \
  oci://ghcr.io/mohamedbentaher/charts/test-scale \
  --version 0.1.0 \
  --namespace test-scale --create-namespace

The chart defaults are a self-contained demo. For a shared cluster, start with values-production.yaml, use managed Postgres and NATS, and create the referenced database Secret first. Release images are multi-architecture, keylessly signed, and published at ghcr.io/mohamedbentaher/test-scale.

From source

git clone https://github.com/MohamedBenTaher/test-scale.git
cd test-scale
make build                    # builds bin/test-scale and server binaries
# or install CLI globally:
go install github.com/MohamedBenTaher/test-scale/cmd/cli@latest

Quick start

Prerequisites

• Go 1.26.5+
• Docker with Compose v2
• curl and jq

1. Run the first test

git clone https://github.com/MohamedBenTaher/test-scale.git
cd test-scale
make quickstart

quickstart builds current application images, starts the platform, waits for health, and runs examples/smoke.yaml against the bundled target. It automatically chooses free host ports and prints the UI and Grafana URLs.

Success ends with output similar to:

PASS: run run-0711-2042-abcd completed with 1 worker(s), 10 RPS, 0 errors
First run complete.

For a remote one-liner:

curl -fsSL https://raw.githubusercontent.com/MohamedBenTaher/test-scale/main/setup.sh | bash

2. Run your own plan

Workers scale automatically when a run starts (Docker Compose + AUTOSCALER_BACKEND=docker).

make docker-run PLAN=examples/http-post.yaml
make docker-open                          # opens the web UI

See the example ladder for public GET, JSON POST, JWT/Bearer, session-cookie, and multi-worker plans.

3. Stop or reset

make docker-down   # stop containers; keep Postgres data
make docker-reset  # stop containers; delete local volumes and run history

Full walkthrough: First-run guide. CLI reference: CLI guide.

Embedded UI

The control plane serves an embedded web UI on the URL printed by make docker-up. The HTML is compiled into the Go binary; Grafana remains a separate service.

Tab	Purpose
Submit	Create a new test plan or re-use a deduplicated saved plan; optional plan/run names; live cost estimate
Runs	Auto-refreshes every 5s — status, slug, RPS, p95, errors, SLO, cost; Re-run, Stop, Dashboard, Compare
Compare	Side-by-side run comparison with RPS, p95, error, and cost deltas; regression flag when p95 > 20% vs baseline
Dashboard	Embedded Grafana with a run picker (slug labels); defaults to the running or latest run

Runs are identified by human-readable slugs (e.g. run-0708-0843-b2ab) instead of UUIDs in the UI.

CLI reference

Command	Description
submit -f plan.yaml	Create test plan + start a run; live watch on TTY
status --run-id <slug|uuid>	Get run status; streams until complete on TTY
plans	List all saved test plans
rerun --plan-id <id>	Re-run a saved test plan
compare --current … --baseline …	Side-by-side comparison with deltas
open	Open the web UI in your browser
--version	Print version and exit

Aliases: run/start → submit · watch/get → status · ls/list → plans · retry → rerun · diff → compare · ui/dashboard → open

Watch flags (on submit, status, rerun): --watch · --no-watch · --json

Test plan format

# test-plan.yaml
target_url: http://mock-target   # use mock-target with Docker Compose
protocol: http
ramp_pattern: constant
target_rps: 100
duration_sec: 30

Worker count is computed automatically by the autoscaler (ceil(target_rps / per_worker_rps)).

Auth and payloads

Protected routes and request bodies are first-class plan fields:

method: POST
auth:
  type: jwt                    # bearer | jwt | cookie | session | header
  token: eyJhbGciOiJIUzI1NiJ9…  # Authorization: Bearer <token>
# or session cookie:
# auth:
#   type: session
#   name: sessionid             # defaults to "session"
#   value: abc123
json:                          # structured payload (sets Content-Type)
  query: health
  limit: 10
# or raw body:
# body: '{"query":"health"}'
timeout_ms: 3000
expected_statuses: [200, 202]

Examples: examples/http-auth-jwt.yaml, examples/http-auth-session.yaml, examples/http-post.yaml.

auth and json are normalized into headers/body when the plan is created. Do not set both body and json.

Autoscaling

Worker count is computed at run time from target_rps and autoscaler.per_worker_rps (default 500). The control plane pre-scales workers before dispatch and drains them when no runs are active.

Backend	Behavior
noop	Returns scaled count immediately (local dev, manual workers)
docker	Scales worker service via docker compose up --scale
keda / k8s	Writes testscale_desired_workers; KEDA scales the worker Deployment

See docs/05-phase3-kubernetes.md for Kubernetes + KEDA setup (make k8s-verify). For external access with TLS, see docs/07-k8s-ingress-tls.md (make k8s-ingress-verify).

Configure in test-scale.yaml or via env: AUTOSCALER_BACKEND, AUTOSCALER_PER_WORKER_RPS.

Cost estimation

Estimated run cost is shown in the Submit tab (before you start), in the Runs table (after completion), and in Compare (with delta vs baseline).

cost:
  provider: static   # or azure
  hourly_rate: 0.01
  per_worker_rps: 500

Target verification

Set SAFETY_VERIFICATION_TOKEN in .env to require targets to serve the token at {target_url}/.well-known/test-scale-verification. Leave empty to skip (localhost always bypasses).

Docker Compose: mock-target serves the default token (demo-verification-token) at http://mock-target.

Limitations & when to trust results

Local stress tests are powerful for regression and plumbing — but they are not a substitute for production capacity planning unless you deliberately scale load generators the same way real clients behave.

What a local run actually measures

With Docker Compose on one machine, all workers share one host’s CPU, memory, and network. Traffic often comes from one IP. You are answering:

“How does my backend behave when load is generated from this environment?”

That is not the same as geo-distributed clients, edge TLS, CDN paths, or production-scale concurrency patterns.

Good uses (trust these)

Use	Why it’s valid
Regression checks	Same setup, same plan — compare run A vs run B with diff
CI / pre-merge smoke	Did we break the loop? Does SLO logic fire?
Relative capacity	“Where does p95 climb on this hardware?”
Platform validation	Autoscaler, metrics merge, worker coordination

Misleading uses (be careful)

Pitfall	What goes wrong
Absolute prod claims	“10k RPS locally → prod handles 10k” without multi-node load gen
Generator saturation	Workers max CPU/network before the backend does — you measured test-scale, not the app
Single remote target, local workers	One fat pipe vs many thin client connections
No baseline	One number with no historical comparison on the same setup

How to get more trustworthy numbers

1. Compare, don’t absolutize — test-scale diff -c today -b baseline on the same worker topology.
2. Watch active workers & RPS — if workers can’t hit target_rps, add workers or calibrate per_worker_rps.
3. Scale the generator — use Kubernetes + KEDA (make k8s-verify) to spread workers across nodes.
4. Label runs — use plan/run names that record environment (e.g. local-1node-2w, k8s-10w).
5. Match production later — Phase 4 stretch: geo-distributed workers; until then, treat local results as directional.

See Functional overview — Interpreting results for the full write-up.

Troubleshooting

Problem	Fix
command not found: test-scale	Use ./bin/test-scale after make build, or go install ./cmd/cli
Default UI port is in use	Run make docker-up; it reuses the active stack port or selects the next free port
Workers stay at 0 (Compose)	Ensure Docker is running and its socket is mounted into ctrl-plane
insufficient worker capacity (K8s)	Install kube-state-metrics; check KEDA ScaledObject and Prometheus scrape
Live CLI metrics show —	Run make docker-up; Compose configures ctrl-plane to reach the internal Prometheus service

Documentation

See docs/README.md for the full index.

Doc	Contents
Functional overview	Goals, use cases, limitations & interpreting results
Technical overview	Architecture, data flow, implementation status
Deployment models	Docker Compose vs Kubernetes
Kubernetes + KEDA	Helm install, kind smoke test
Ingress + TLS	cert-manager, external UI/API access
CLI guide	Commands, aliases, env vars, examples

Development

go mod tidy
make test
make lint
make build          # all binaries → bin/

Make targets

Two deployment modes — use one or the other:

make help               # print all targets

Docker Compose

make quickstart
make docker-verify
make docker-run PLAN=examples/http-post.yaml
make docker-down

Kubernetes + Ingress (your current setup)

make k8s-ingress-up
make k8s-ingress-verify
make k8s-ingress-run PLAN=deploy/kubernetes/ci-smoke.yaml
make k8s-ingress-down

Kubernetes without Ingress (port-forward)

make k8s-up
kubectl port-forward -n test-scale svc/test-scale-ctrl-plane 8080:8080
make k8s-verify

See CHANGELOG.md for release history. For long-running deployments, see the operator guide. The proposed custom-worker boundary is documented separately and is not a v0.1 API.

Security and contributing

Docker Compose is intended for trusted local development because the control plane uses the Docker socket to scale workers. Use Kubernetes/KEDA for shared environments. See SECURITY.md, CONTRIBUTING.md, and LICENSE.