FireDrill

README

FireDrill 🔥

Fire drills for your backups. Prove recovery before you need it.

FireDrill restores your real backups into a disposable, isolated sandbox, verifies the data actually came back intact, measures the true recovery time, and emits signed, audit-grade evidence — then destroys the sandbox. It answers the question every backup tool quietly dodges: if today were the disaster, would you actually get your data back — and how long would it take?

FireDrill does not back anything up. It is the verification layer on top of whatever backup you already run (pg_dump, mysqldump, Velero, pgBackRest, RDS snapshots, …): backup-agnostic recovery verification with audit-grade proof. Postgres, MySQL and Velero (Kubernetes namespaces) are supported today; the driver interface is built for more.

Demo

Install

Download a binary from releases, or build from source:

git clone https://github.com/kirilurbonas/FireDrill && cd FireDrill && make build

Quickstart

Requirements: Docker running locally. No Postgres client needed on the host — restore tooling runs inside the sandbox container.

make build                        # builds bin/firedrill
./bin/firedrill keygen            # one-time: create the evidence-signing keypair
./examples/make-demo-backup.sh    # generate a realistic demo pg_dump
./bin/firedrill validate -f examples/firedrill.yaml
./bin/firedrill run payments-db -f examples/firedrill.yaml
./bin/firedrill verify-evidence evidence/payments-db-*.json

Exit codes: 0 recovery verified · 1 drill ran but recovery not verified · 2 drill could not execute.

How it works

1. Declare recovery targets in firedrill.yaml — source backup, restore method, RTO/RPO objectives, checks.
2. Provision a throwaway sandbox — a Docker container (own bridge network, loopback-only port) or a Kubernetes pod (dedicated namespace, deny-all-egress NetworkPolicy) — with random one-off credentials and a hard TTL.
3. Restore the latest backup into it (local file or s3://…), timed — that's your measured RTO.
4. Verify — restore succeeded, freshness vs RPO, row counts, order-independent checksums, custom smoke SQL.
5. Report — a JSON evidence record signed with ed25519, mapped to compliance controls (ISO27001-A.8.13, SOC2-A1.2, …).
6. Destroy the sandbox — guaranteed by both a deferred teardown and an in-process TTL watchdog.

See examples/firedrill.yaml (Postgres) and examples/firedrill-mysql.yaml (MySQL) for the spec format, and docs/architecture.md for design.

Evidence

Every drill writes evidence/<drill>-<timestamp>.json plus a detached .sig envelope. Any tampering with the evidence fails verification:

$ firedrill verify-evidence evidence/payments-db-2026-07-08T19-25-12Z.json
✓ signature valid — evidence is intact

Auditors can verify with --public-key ~/.config/firedrill/firedrill.pub to additionally pin the signing key.

With report.html: true, a self-contained HTML report (<evidence>.html) is written next to the JSON — shareable with anyone who won't read JSON.

Compliance-control export

Drills declare which controls they evidence (report.controls: [ISO27001-A.8.13, SOC2-A1.2]). firedrill controls aggregates an evidence directory into an auditor-ready matrix — per control: every run, its result, measured restore time, RTO/RPO status, and whether the evidence signature validates:

firedrill controls                          # markdown to stdout
firedrill controls --format json -o controls.json

Hand the markdown (or JSON) straight to your GRC team at audit time instead of screenshots and a Confluence page.

Kubernetes

Two levels of Kubernetes support:

Sandbox provider — set sandbox.provider: kubernetes and the drill provisions the sandbox as a pod (dedicated namespace, deny-all-egress NetworkPolicy, random credentials, TTL force-delete) instead of a Docker container. The CLI reaches it through a port-forward; in-cluster it uses the pod IP.

Operator — declare drills as RecoveryDrill custom resources and let the operator run them on a cron schedule:

kubectl apply -f deploy/crd.yaml
kubectl apply -f deploy/operator.yaml      # or run `firedrill operator` with a kubeconfig
kubectl apply -f deploy/example-recoverydrill.yaml
kubectl get drills -n firedrill-system     # NAME  PHASE  VERIFIED  LAST RUN  SCHEDULE

The CR's spec: block is exactly the firedrill.yaml spec — the operator validates and runs it with the same code as the CLI, and records the outcome (phase, verified, measured RTO/RPO) in .status.

The operator image is published to ghcr.io/kirilurbonas/firedrill (multi-arch) by the release workflow — deploy/operator.yaml uses it out of the box; pin a version tag in production.

Velero drills — if your backups are Velero backups, FireDrill can drill whole namespaces: it restores the backup into an ephemeral namespace via a Velero Restore with namespaceMapping (production is never touched), verifies the workloads actually came back, and deletes the namespace:

source:
  driver: velero
  from: { type: velero, backup: shop-nightly, namespace: shop }
sandbox: { provider: kubernetes, ttl: 20m }
verify:
  - restoreSucceeded: {}
  - podsReady: { timeout: 5m }                     # every restored pod reaches Ready
  - resourceCount: { kind: deployments, min: 1 }   # objects actually came back

Requires Velero installed in the cluster. Try it locally: examples/velero/setup-velero-kind.sh stands up Velero + MinIO + a demo backup in a kind cluster, then firedrill run shop-ns -f examples/firedrill-velero.yaml.

Metrics

Drill results export as Prometheus metrics via report.sinks:

report:
  sinks:
    - { type: prometheus, textfileDir: /var/lib/node_exporter/textfile }  # node_exporter textfile collector
    - { type: pushgateway, url: http://pushgateway:9091 }                 # for scrape-based setups

Exported (per drill): firedrill_drill_verified, firedrill_restore_duration_seconds (measured RTO), firedrill_backup_age_seconds (RPO), firedrill_rto_met, firedrill_rpo_met, firedrill_check_passed{check=…}, firedrill_drill_timestamp_seconds. Alert on firedrill_drill_verified == 0 or a rising restore_duration trend. Sink failures are warnings — they never fail a drill.

A ready-made Grafana dashboard (verification history, RTO/RPO trends, time-since-last-drill) ships at deploy/grafana-dashboard.json — import it and point it at your Prometheus datasource.

Slack notifications

Add a slack sink to get drill outcomes in a channel. The webhook URL is read from an environment variable — it never appears in the spec:

report:
  sinks:
    - { type: slack, webhookEnv: SLACK_WEBHOOK_URL, onlyFailures: true }

onlyFailures: true keeps the channel quiet until a drill actually fails — usually what you want for the 3 a.m. pager channel.

Guardrails

Risk	Mitigation
Accidentally touching production	Docker: own network, published to 127.0.0.1 only. Kubernetes: deny-all-egress NetworkPolicy. Sources are read-only (FireDrill only downloads)
Sandbox left running	Deferred destroy on every code path and a TTL watchdog that force-removes the container/pod past the deadline
"Restore ran" ≠ "data is back"	Data-level checks: row counts, checksums, user smoke SQL — not just exit codes
Secrets leaking into evidence	Credentials referenced by name (credentialsRef → AWS profile), never inlined or persisted
Corrupt/garbage backups passing	A failed restore fails the drill; dependent checks report SKIP, never false PASS
Secrets in process lists	Database passwords reach in-sandbox tooling via environment, never argv

See SECURITY.md for the full security model and how to report vulnerabilities.

How is this different from …?

Tool	What it does	What it doesn't
pgbackrest_auto	Automated restore + validate for pgBackRest	pgBackRest-only, bash, no signed evidence
AWS Backup restore testing	Managed periodic restore tests	AWS resources only, evidence stays in AWS
Backup tools with verify (pgBackRest, pg_probackup, …)	Checksum their own backups	Verify writes, not end-to-end recovery
FireDrill	Backup-agnostic recovery drills with measured RTO/RPO and signed, control-mapped evidence	Doesn't back anything up — by design

A backup that has never been restored is just hope stored on disk. FireDrill turns that hope into a signed record an auditor can check.

Development

make test    # unit tests
make e2e     # full drill loops against real Docker + a Kubernetes cluster (kind); k8s tests skip if no cluster is reachable
make lint    # golangci-lint (incl. gosec)

CI runs all of it — lint (with e2e files), govulncheck, unit tests, and the Docker/Kubernetes/operator e2e suites against a kind cluster. Dependabot keeps dependencies current.

Roadmap

Next up: cloud sandboxes (Terraform/RDS), sigstore/cosign attestations. See firedrill-plan.md.

License

Apache-2.0