README
¶
Khaos: Kafka Load Testing & Chaos Engineering
Khaos is an open-source Kafka traffic generator, load-testing tool, and chaos engineering CLI for reproducing realistic Kafka workloads and failure scenarios (consumer lag, hot partitions, rebalances, and broker failures) on demand, instead of waiting for production to find them.
Documentation · Quick Start · Scenario Reference
What it does
- Generate realistic Kafka test data: structured, faker-backed records in JSON, Avro, or Protobuf.
- Simulate producer and consumer traffic: configurable throughput, key distributions, and consumer group topology.
- Load test Kafka clusters and the applications that consume from them, including Kafka Streams and Flink jobs.
- Reproduce failure conditions on purpose: consumer lag, hot partitions, rebalances, and broker failures, scheduled on a timeline.
Scenarios are plain YAML. No code, no client library, no instrumentation in the system under test.
Quick start
go install github.com/aleksandarskrbic/khaos/cmd/khaos@latest
khaos list # see the bundled scenarios
khaos run traffic/high-throughput # auto-starts a local 3-broker Kafka cluster
khaos run manages its own local Kafka cluster via Docker Compose. To target a cluster you
already have, including managed clusters needing SASL/SSL, use khaos simulate instead. See the
Quick Start guide and
Installation for release binaries and Docker.
A few scenarios
khaos run traffic/hot-partition # skewed key distribution overloads one partition
khaos run traffic/consumer-lag # producer rate outpaces slow consumers
khaos run chaos/broker-chaos # brokers stop and restart while traffic keeps flowing
khaos run chaos/rebalance-storm # a consumer group rebalances repeatedly
khaos validate checks a scenario file's structure without running it: the same command Khaos's
own CI runs against every bundled scenario. See the
Scenarios and
Guides sections of the docs for what
each one actually configures and why.
Documentation
Full documentation, including the CLI reference, the scenario YAML schema, and guides for load testing, data generation, and each failure scenario, lives at getkhaos.dev/docs:
Architecture
Khaos is a single Go binary. The scenario engine is independent of any user interface: it exposes
one read method, Snapshot(), and the terminal UI, the headless log loop, and the final summary
table all poll it. Nothing in the engine knows about terminals, so a headless run in CI behaves
identically to an interactive one, and a stalled UI can't stall a run.
Kafka access is franz-go, a pure-Go client, which is what makes
CGO_ENABLED=0, cross-compilation, go install, and a distroless/static image all work without
a C toolchain. See the Concepts page for the full picture.
Rewritten in Go
Khaos was rewritten from Python to Go: a single static binary, no runtime dependencies, and a pure Go Kafka client that cross-compiles cleanly. See the release notes for details.
Contributing
Issues and pull requests are welcome. See CONTRIBUTING.md. If Khaos is useful to you, a star helps others find it.
License
Apache 2.0. See LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
khaos
command
Command khaos generates Kafka traffic and simulates failure scenarios.
|
Command khaos generates Kafka traffic and simulates failure scenarios. |
|
internal
|
|
|
codec
Package codec turns generated message documents into Kafka record bytes and back, for the four wire shapes khaos supports: JSON, Avro (inline schema or Schema Registry) and Protobuf (inline schema or Schema Registry).
|
Package codec turns generated message documents into Kafka record bytes and back, for the four wire shapes khaos supports: JSON, Avro (inline schema or Schema Registry) and Protobuf (inline schema or Schema Registry). |
|
generate
Package generate builds the message payloads khaos produces: single field values, whole documents from a `fields:` schema, synthetic size-padded JSON when no schema is given, Kafka keys, and correlated multi-step flow messages.
|
Package generate builds the message payloads khaos produces: single field values, whole documents from a `fields:` schema, synthetic size-padded JSON when no schema is given, Kafka keys, and correlated multi-step flow messages. |
|
kafka
Package kafka builds the franz-go clients khaos runs its traffic through, and the admin helpers that prepare the topics those clients use.
|
Package kafka builds the franz-go clients khaos runs its traffic through, and the admin helpers that prepare the topics those clients use. |
|
localcluster
Package localcluster manages the bundled single-host Kafka cluster that khaos uses for local runs, by driving the docker CLI against embedded compose files.
|
Package localcluster manages the bundled single-host Kafka cluster that khaos uses for local runs, by driving the docker CLI against embedded compose files. |
|
scenario
Package scenario holds the khaos scenario domain model together with the YAML decoding and validation that produces it.
|
Package scenario holds the khaos scenario domain model together with the YAML decoding and validation that produces it. |
|
telemetry
Package telemetry provides khaos's structured logger, its Prometheus metric set and the small HTTP server that exposes /healthz and /metrics.
|
Package telemetry provides khaos's structured logger, its Prometheus metric set and the small HTTP server that exposes /healthz and /metrics. |
|
theme
Package theme is the one palette every khaos surface draws from: cyan for names, green for produced, yellow for consumed, magenta for failures, blue for DLQ.
|
Package theme is the one palette every khaos surface draws from: cyan for names, green for produced, yellow for consumed, magenta for failures, blue for DLQ. |
|
tui
Package tui renders live run state in the terminal.
|
Package tui renders live run state in the terminal. |