kafka-dlq

README

kafka-dlq

English | Türkçe

An inbox for your Kafka dead letter topics: see why messages failed, and replay them safely.

A consumer cannot process a message, so it lands in a DLQ topic. What happens next is the same everywhere: nobody looks at that topic, and when someone finally does, they find raw bytes. Why it failed, how many times it was tried, whether it was already replayed once — none of that is written down anywhere. It ends with the topic being deleted, or with somebody pasting the message back into production and starting the same incident a second time.

your consumer ──▶ orders.DLQ ──▶ kafka-dlq ──▶ indexed, filterable, replayable
                                     │
                                  SQLite  (payload, headers, reason, attempts,
                                           replay history — one binary, no infra)

• Indexed by reason, not by bytes. Free-form error text is unfilterable: it embeds ids, offsets and hostnames, so every message is a unique string. kafka-dlq reduces it to a short label — schema-error, timeout, conflict — with ordered rules you can replace. Text matching nothing stays unknown rather than being forced into the nearest bucket.
• Deduplicated, so a loop is visible. The same message failing again updates one row instead of adding one. Replay it, watch it come back, and it is marked looping — the fault was never fixed.
• Replayed safely. An explicit filter is required, a range replay must quote a matching dry-run plan, records go out in original order per key, anything at its replay limit is blocked, the rate is capped, and every attempt is written down. Details below — this is the part worth reading.
• One binary. No Postgres, no Redis, no JVM. The store is embedded SQLite via a pure-Go driver, so there is no cgo and cross-compilation just works.

The list, and the signal that matters

cust-1011 is marked looping: it was replayed once and then landed in the DLQ again. That is the difference between a message worth replaying and one that will just repeat the incident.

Built for teams running Kafka outside the JVM — Go, Python, Node. Spring Kafka's @RetryableTopic and DeadLetterPublishingRecoverer give JVM teams a lot of this; everyone else writes shell scripts. (kafka-dlq reads Spring's headers too, so it works during a migration.)

Quick start

go install github.com/YusufDrymz/kafka-dlq/cmd/kafka-dlq@latest

cp kafka-dlq.example.yaml kafka-dlq.yaml   # edit brokers and your DLQ topic
kafka-dlq serve                            # indexes the topic, serves the UI

Then, in another terminal:

$ kafka-dlq list
ID        KEY        REASON            ATTEMPTS  REPLAYS  LAST SEEN  STATUS
fe2345dc  cust-1010  schema-error      3         0        5m ago     new
9a0153fe  cust-1011  timeout           3         1        7m ago     new (looping)
489a7479  cust-1010  schema-error      3         0        9m ago     new
54383f07  cust-1012  conflict          3         0        21m ago    new
9f6caf3f  cust-1013  timeout           3         0        44m ago    new
cedc98db  cust-1014  upstream-error    3         0        58m ago    new

$ kafka-dlq list --summary
REASON            COUNT  LAST SEEN
schema-error      3      5m ago
timeout           2      7m ago
conflict          1      21m ago
upstream-error    1      58m ago
validation-error  1      2h ago

serve is not required for anything except the UI. Every command talks to the store directly, so replay works at 3am when the service that would have served the UI is the thing you are trying to fix.

Try the loop in two minutes

The demo is not a happy path. It walks the incident this tool exists for: a message fails, you replay it without fixing anything, and it comes back.

cd demo
docker compose up -d --build
./run.sh

Six steps: an application drops poison messages into orders.DLQ, kafka-dlq indexes them by reason, an operator previews a replay, the replay runs with the fault still in place, the messages fail again — and the same rows come back marked looping, with the second replay refused at max_replays.

The UI is at http://localhost:8085 (bearer token demo-token — the compose file sets it, and config refuses to serve an unauthenticated API on anything but loopback, so even the demo has to pass one).

The replay safety model

Replay is the most dangerous button in a DLQ tool. Five things stand between an operator and a repeated incident.

1. An explicit filter — there is no --all. Replay requires at least one of --dlq, --reason, --key, --id or a time window. Draining an entire dead letter topic into production cannot be one command.

2. A dry run the real replay must match. Preview first:

$ kafka-dlq replay --dlq orders.DLQ --reason timeout --dry-run
plan 019f762e-3e16-71b5-baf4-03ab9de6ce22
target topic      orders
records           1
distinct keys     1
oldest / newest   44m ago / 44m ago
estimated time    under a second at the configured rate
blocked           1 at max_replays (use --force to override; it is audited)

first records in publish order:
  ID        KEY        REASON   LAST SEEN
  9f6caf3f  cust-1013  timeout  44m ago

nothing has been published. To execute this exact plan:
  kafka-dlq replay --plan 019f762e-3e16-71b5-baf4-03ab9de6ce22 --dlq orders.DLQ --reason timeout

The execute button does not exist until a dry run has produced a plan, and it disappears again if any filter changes.

The plan id is not a formality. The execution hashes its selection criteria and compares them to the plan's — without that, "I previewed it" and "I replayed what I previewed" are unrelated claims, and you could preview a narrow window and execute a wide one. A plan is single-use. A run that died halfway is resumed rather than restarted, so the records it already published are not published twice.

3. Original order per key. Kafka orders within a partition, and the partition comes from the key. Replaying two records of the same key concurrently reorders them — for a stream of order updates, that means the final state is wrong. kafka-dlq groups by key, publishes each group in origin_offset order from a single goroutine, waits for the broker's acknowledgement between records, and runs different keys in parallel. A failure stops its own key and nothing else, because continuing past it would let later records overtake the one that never landed.

4. A poison-pill limit. max_replays defaults to 1. The counter exists to break a loop, and a loop comes from republishing without fixing the fault; a second automatic chance delays the incident rather than reducing it. Blocked records are listed in the dry run, not hidden. --force overrides the limit and is written to the audit trail.

5. A rate limit and a ledger. Publishing is capped at rate_limit (default 50/s), in evenly spaced slots rather than bursts — a token bucket that releases 50 at once and then waits would drown the very consumers the limit protects. Every run and every attempt lands in replay_runs and replay_attempts, which outlive the records themselves: after prune, "was this message ever replayed, by whom, to where" is still answerable.

Replayed messages carry x-kafka-dlq-replay-of and x-kafka-dlq-replay-count, so a consumer can behave differently on a replay and a second failure is traceable without this tool's database.

Replay is at-least-once. Your consumers must be idempotent; kafka-dlq does not and cannot make them so.

The five-second question

CLI

kafka-dlq serve      consume the configured DLQ topics and serve the UI and API
kafka-dlq list       list indexed dead letters (--summary groups by reason)
kafka-dlq show       one dead letter with headers, payload and replay history
kafka-dlq replay     republish dead letters (see the safety model above)
kafka-dlq discard    mark a record as never-to-be-replayed (--note required)
kafka-dlq prune      delete records older than a cutoff; keeps replay history
kafka-dlq runs       list past replay runs
kafka-dlq version

Filters shared by list and replay: --dlq, --reason, --key, --status, --id (repeatable), --since 24h, --from, --to-time. Times accept a duration meaning "ago" or a timestamp.

The short ids list prints can be passed to show, discard and replay --id; an ambiguous one is an error rather than a guess.

discard insists on --note. Six months later that note is the only record of why a message was written off.

Exit codes are a contract:

code	meaning
0	the command did what it said
1	a business outcome: nothing matched, a safety gate refused, a replay partially failed
2	the tool could not run: bad flags, broken config, unreachable broker

A partial replay exits 1. A script treating "9 of 10 replayed" as success would move on with a gap.

Configuration

See kafka-dlq.example.yaml for the annotated version. The essentials:

version: 1
kafka:
  brokers: [localhost:9092]
dlqs:
  - topic: orders.DLQ
    origin_topic: orders     # where replay publishes by default
    max_replays: 1
server:
  listen: "127.0.0.1:8085"
replay:
  rate_limit: 50/s
  require_dry_run: true
retention: 30d

Three deliberate choices:

• An unknown field fails the load. A typo in max_replays that silently left the poison-pill gate at its default is exactly the quiet misconfiguration this tool exists to prevent, and a warning in a log nobody reads is not a defence.
• The server binds loopback by default. This process holds every payload that failed in production plus a button that republishes them. Exposing it is a deliberate edit — and then ui_token_env becomes required.
• Secrets are named, never written. The file carries environment variable names, so it stays safe to commit. An unset variable is an error rather than an empty token, which would quietly leave the API open.

Error reasons

Reason labels come from ordered rules; first match wins. The built-in set covers timeout, rate-limited, auth-error, not-found, conflict, connection-error, upstream-error, db-error, validation-error, schema-error and panic. Overlaps are resolved by order and locked in by tests — a duplicate key is a conflict, not a generic db-error, because that one is usually safe to discard rather than replay.

Supplying your own rules replaces the built-in set entirely, so what you read in your config is every rule that can fire.

Where the error text comes from

kafka-dlq checks, in order: your configured header, your configured fallback, its own dlq-error-reason, then Spring Kafka's kafka_dlt-exception-message and kafka_dlt-exception-fqcn. Origin coordinates come from x-dlq-origin-{topic,partition,offset} or Spring's kafka_dlt-original-*.

Spring writes the original partition as a 4-byte big-endian int and the offset as an 8-byte long rather than as text, so both encodings are read. A DLQ whose producer writes no headers at all still works — it just deduplicates on content instead of coordinates and every reason is unknown. That degradation is the honest limit of the tool, and it is the biggest reason to check your producers before adopting it.

HTTP API

serve exposes the same operations the CLI uses, so the API is not a way around the safety model.

GET  /healthz                              liveness; queries the store, no auth
GET  /api/dead-letters                     list; filters mirror the CLI flags
GET  /api/dead-letters/{id}                detail with headers, payload, history
POST /api/dead-letters/{id}/discard        {"note": "..."}
GET  /api/summary                          counts grouped by reason
GET  /api/runs, /api/runs/{id}             replay audit trail
POST /api/replay/plan                      dry run; returns a plan id
POST /api/replay                           execute a plan
GET  /api/config                           topics, limits — no secrets

Authentication is a static bearer token when ui_token_env is set, compared in constant time. OIDC is not implemented.

What kafka-dlq is not

Deliberate non-goals, so you can pick the right tool:

• Not a Kafka UI. It does not create topics, manage ACLs, edit consumer group offsets, show broker metrics or rebalance partitions. Use AKHQ or Kafbat UI for that — they are good at it, and this is not an attempt to compete with them.
• Not a retry framework. It does not retry or back off on your behalf. You put the message in the DLQ; this manages what happens afterwards.
• Not the source of truth. Kafka is. This store is an index and an operations ledger; losing it costs you reason labels and replay history, not messages.
• No exactly-once. Replay is at-least-once and your consumers must be idempotent.
• No multi-tenancy or RBAC. It is a team tool. The actor recorded on a replay is the OS user or ui — an audit breadcrumb, not an authenticated identity.
• No Avro/Protobuf decoding yet. Payloads are shown as JSON when they parse and as bytes otherwise. Schema Registry support is on the roadmap.
• No internet-scale claim. One process, one embedded store, one writer.

Neighbours

Tool	DLQ as a concept	Reason index	Replay	License	Infra
kafka-dlq	yes	yes	ordered, dry-run gated, audited	MIT	1 binary
AKHQ	no — just a topic	no	no	Apache-2.0	JVM
Kafbat UI	no	no	prefill a produce form	Apache-2.0	JVM
Redpanda Console	no	no	no	BSL (source-available)	1 binary
Kpow	yes	partly	"Clone to Topic"	commercial	JVM
Dead Letter Society	yes	yes	review then replay	none stated	Quarkus + Postgres

Two honest notes on that table. Dead Letter Society is the same idea, built before this one — multi-cluster DLQ monitoring with schema decoding and a review flow. It is a single-author project with no license file, and it needs Postgres and a JVM; kafka-dlq's differences are the single binary, Go, and the replay safety model. And Kpow already solves this commercially, which is the clearest evidence the problem is real.

If one of those fits your constraints better, use it.

How it behaves

• Indexing is at-least-once. Offsets are committed after the record is stored, and the fingerprint turns a redelivery into an update rather than a duplicate. Auto-commit would risk dropping a dead letter on restart, and losing the record of a lost message is a poor trade.
• Deduplication is on origin_topic + partition + offset when the headers are there, and on dlq_topic + key + payload when they are not. The fallback is weaker in a specific way: two genuinely distinct messages with identical key and bytes collapse into one row. For a DLQ that is usually right — identical bytes failing repeatedly is one problem, not many — but it means the row count is a count of distinct failures, not of records.
• Payloads over 1 MiB are stored truncated and flagged. Replay re-reads the full record from Kafka, and fails loudly if retention has expired rather than publishing half a message.
• One writer. SQLite in WAL mode, a single connection. At DLQ volumes this is not a bottleneck, and it removes a category of concurrency bug from a tool whose job is not to lose messages.

See it before you install it

The Kafka lab at lab.yusufdariyemez.com/kafka is an interactive, deterministic model of the behaviour this tool operates on — including a "Dead letters" section that shows why replaying a DLQ before fixing the fault loops the incident. It runs entirely in the browser.

Development

go test ./...                # unit tests; no Docker needed
go test -race -count=2 ./...  # what CI runs
gofmt -l . && go vet ./...

See CONTRIBUTING.md for what the odder-looking tests are protecting — several of them encode a bug that has already been paid for once.

The UI lives in web/ and its build output is committed, so go install produces a binary with the UI inside and Node is only needed to change the UI. CI verifies npm run build succeeds but does not diff the output — Tailwind's native minifier emits different bytes on macOS and Linux, so a byte-comparison gate would fail permanently for reasons unrelated to correctness.

Roadmap

1. Alerting — a webhook when a DLQ grows or a new reason appears. This is the real answer to "nobody looks at the DLQ".
2. Schema Registry with Avro and Protobuf decoding.
3. PostgreSQL store and multi-instance serve.
4. Multi-cluster.
5. Prometheus metrics, automatic pruning, config hot-reload.
6. A consumer-side helper library so Go services write standard DLQ headers in the first place — the actual antidote to the "no headers" limitation.

License

MIT