catacomb

README

Regression testing for Claude Code agents.
Change a prompt, a skill, or an MCP tool — then let statistics,
not vibes, tell you whether your agent got worse.

    

---

You spent weeks tuning your agent — prompts, skills, MCP tools, CLAUDE.md. Now every change to that setup is a gamble: agents are nondeterministic, so one good run after a prompt tweak proves nothing, and one bad run proves nothing either. Catacomb settles the question with statistics. catacomb bench runs the same tasks repeatedly under the old setup and the new one, recording each run locally as secret-redacted evidence; catacomb regress compares the two groups and maps the verdict to an exit code, so CI blocks the regression before it merges. No daemon, no service, no network — the whole loop is plain local files.

✨ Features

• A gate, not a vibe check. Repeated runs per variant, compared with statistics built for small samples (why this matters); the verdict maps to the exit code your CI already understands.
• Plain local files. Evidence directories plus an optional SQLite store for baselines — nothing listens, nothing phones home.
• Evidence you can share. Recorded transcripts pass through best-effort secret redaction before they ever touch disk (what is and isn't caught).
• Comparisons survive prompt rewrites. The agent can name phases of its own run (checkpoints), giving regress a stable axis even when prompt churn re-keys every step (concepts).
• Longitudinal memory. Pin golden groups as named baselines; every recorded comparison accumulates into a history that trends replays (workflows).
• Checks the answer, not just the path. Declare a per-task verifier and its pass/fail verdict rides the same statistical gate (verifying task outcomes).

---

🧰 Requirements

Claude Code installed and claude on your PATH. Catacomb itself is a single static binary — no runtime, no dependencies, no config file.

---

📦 Installation

brew tap realkarych/tap
brew trust realkarych/tap      # newer Homebrew requires trusting third-party taps
brew install --cask catacomb

Docker

Package: https://github.com/realkarych/catacomb/pkgs/container/catacomb.

docker run --rm ghcr.io/realkarych/catacomb:latest version

Debian / Ubuntu (APT)

# Import the signing key
curl -fsSL https://realkarych.github.io/catacomb-apt/public.key \
  | sudo tee /etc/apt/trusted.gpg.d/catacomb.asc

# Add the repository
echo "deb [arch=$(dpkg --print-architecture)] \
  https://realkarych.github.io/catacomb-apt stable main" \
  | sudo tee /etc/apt/sources.list.d/catacomb.list

# Install / update
sudo apt update
sudo apt install catacomb

Go install / from source (Go ≥ 1.26)

go install github.com/realkarych/catacomb/cmd/catacomb@latest
# make sure $GOBIN (default ~/go/bin) is on your PATH

Or clone the repo and make build.

Windows / other distros

Download the pre-built archive from the Releases page, unpack it, and add the binary to your PATH. On Windows, you may need Unblock-File .\catacomb.exe before first run.

Upgrading from Homebrew: brew upgrade --cask catacomb. Migrating from the old formula: brew uninstall catacomb && brew install --cask catacomb.

---

🚀 Tutorial

Ten minutes, two small files, one caught regression. The scenario: someone on your team wants to add a chain-of-thought instruction to a shared prompt, and you want CI to tell you what that does to the agent's behavior — before it merges.

1. Create it

Make an empty folder with two files. agent.sh is the agent command under test — any command works as long as it emits stream-json, which is how catacomb finds the session transcript:

#!/usr/bin/env bash
# agent.sh — the agent under test. bench runs this once per cell.
set -euo pipefail
exec claude -p "${PROMPT}" \
  --model claude-haiku-4-5 \
  --output-format stream-json \
  --verbose \
  --strict-mcp-config \
  --setting-sources project

(The last two flags keep your user-scope plugins and hooks out of the benchmark, so runs are comparable across machines.)

demo.yaml declares the experiment — a basket: the matrix of tasks × variants × reps. Each combination is one cell, run as a plain local process:

# demo.yaml — the same question, with and without a chain-of-thought
# instruction someone wants to add to the shared prompt.
basket: demo
reps: 5
tasks:
  - id: answer
    cmd: ["./agent.sh"]
    dir: .
variants:
  - id: main
    env:
      PROMPT: "In one short sentence, what is a hash function?"
  - id: candidate
    env:
      PROMPT: "In one short sentence, what is a hash function? Think step by step and write out your full reasoning before you answer."

chmod +x agent.sh

2. Run it

catacomb bench demo.yaml --runs-dir runs

Ten cells run sequentially — 2 variants × 5 reps, about a minute and a few cents on haiku. Each cell streams the agent's raw stream-json output to your terminal; the block below shows only the tail:

…
marked 10/10 cells
Next steps:
  catacomb regress --runs-dir runs --baseline label:basket=demo,variant=main --candidate label:basket=demo,variant=candidate

What just happened:

• each cell ran agent.sh as a plain local process and catacomb resolved its transcripts from ~/.claude/projects;
• each run was written to runs/<run-id>/ as a secret-redacted evidence directory — transcripts plus a meta.json with labels, exit code, and cost;
• the manifest (demo.yaml.manifest.jsonl) records every cell, so an interrupted basket resumes with --resume.

3. Gate it

Compare the candidate group against the baseline group — the epilogue above already printed this command for you:

catacomb regress --runs-dir runs \
  --baseline label:basket=demo,variant=main \
  --candidate label:basket=demo,variant=candidate

(If your Claude Code is newer than the versions catacomb has been tested against, a few harmless warning: transcript … newer than tested … lines may precede the table.)

baseline runs 5  candidate runs 5
coverage steps 1.00  phases 1.00  steps_trusted true  overall regression
sensitivity: gate cannot fire at this support (paired gate needs k>=5 tasks)
VERDICT       SCOPE   KEY                               NAME         METRIC       BASELINE  CANDIDATE  BAND                DETAIL
regression    total   -                                 -            cost_usd     0.00      0.01       [0.00, 0.00]        -
regression    total   -                                 -            duration_ms  5059.00   7165.00    [3755.50, 6362.50]  -
ok            total   -                                 -            error_rate   0.00      0.00       [0.00, 0.35]        -
ok            total   -                                 -            nodes        4.00      4.00       [3.00, 5.00]        -
ok            total   -                                 -            tokens_in    10.00     10.00      [7.50, 12.50]       -
regression    total   -                                 -            tokens_out   147.00    465.00     [91.50, 202.50]     -
insufficient  paired  -                                 -            cost_usd     0.00      0.00       -                   matched 1 task below paired min 5
insufficient  paired  -                                 -            duration_ms  0.00      0.00       -                   matched 1 task below paired min 5
insufficient  paired  -                                 -            tokens_in    0.00      0.00       -                   matched 1 task below paired min 5
insufficient  paired  -                                 -            tokens_out   0.00      0.00       -                   matched 1 task below paired min 5
regression    phase   eb1d7eb24fc38d7838cf7b81664c90e6  task:answer  cost_usd     0.00      0.01       [0.00, 0.00]        -
regression    phase   eb1d7eb24fc38d7838cf7b81664c90e6  task:answer  duration_ms  5059.00   7165.00    [3755.50, 6362.50]  -
regression    phase   eb1d7eb24fc38d7838cf7b81664c90e6  task:answer  tokens_out   147.00    465.00     [91.50, 202.50]     -
audit: baseline run bench-demo-answer-main-r1 (task answer) cost_usd 0.010954549999999999 vs group median 0.0033167 (band 0.00165835)

The chain-of-thought instruction made every run roughly 3× chattier (median tokens_out 147 → 465) and about 40% slower (median duration 5.1 s → 7.2 s), and even at haiku pennies the cost crossed its noise band — the overall verdict is regression and the exit code is 1. The sensitivity: line refers only to the paired per-task axis, which needs five tasks where this demo has one — the run-total metrics are what fired the gate. That exit code is the gate:

catacomb regress --runs-dir runs \
  --baseline label:basket=demo,variant=main \
  --candidate label:basket=demo,variant=candidate \
  && echo "safe to merge"
# 0 = ok · 1 = regression · 2 = operational error

Two runs of an agent are two samples from a distribution — a single side-by-side diff cannot tell a real regression from sampling noise, which is why every comparison here is group-vs-group (methodology).

4. Upgrade it

Pin a baseline and keep history. Label selectors churn; names don't. Pin the golden group once, then --record every CI comparison and replay the drift later:

catacomb baseline set demo-main --label basket=demo,variant=main \
  --runs-dir runs --db demo.db
catacomb regress --runs-dir runs --db demo.db \
  --baseline name:demo-main \
  --candidate label:basket=demo,variant=candidate --record
catacomb trends demo-main --db demo.db

SEQ  CREATED               CANDIDATE                            VERDICT     REGRESSIONS  INSUFFICIENT  DURATION_MS  COST_USD  ERROR_RATE
1    2026-07-13T10:59:37Z  label:basket=demo,variant=candidate  regression  6            4             7165.00      0.01      0.00

Let the agent mark its own phases. Prompt rewrites re-key steps, which degrades step-level alignment. Checkpoints survive that: give the agent the shipped MCP marker tool, tell it when to mark, and declare what you expect:

{"mcpServers":{"catacomb":{"command":"catacomb","args":["mcp"]}}}

Pass that file via claude --mcp-config, add one line to your CLAUDE.md — e.g. "call mcp__catacomb__mark with name: plan before planning and after tests pass" — and declare checkpoints: [plan, tests.pass] on the task. Declared phases become stable comparison rows in the regress table, robust to prompt churn (concepts).

Verify the answer itself. Deterministic observables catch behavioral drift; whether the agent produced the right answer is task-specific. Declare the files a task produces and a command that scores them:

tasks:
  - id: sql
    cmd: ["./agent.sh"]
    artifacts: ["out/result.csv"]
    verify:
      cmd: ["python3", "./verify_sql.py"]
      env: { GOLDEN: "/fixtures/golden.csv" }   # ground truth, OUTSIDE the workdir

The verifier reads the captured artifacts and emits one pass/fail line — two calls with the shipped Python SDK:

import os

from catacomb_verifier import Cell, emit, compare_tables

cell = Cell.from_env()
res = compare_tables(cell.artifact("out/result.csv"), os.environ["GOLDEN"], ordered=False)
emit(passed=res.equal, tool="verify_sql", tool_version="1")

bench runs it after every cell, and the resulting verifier.pass rate gates through regress by default — same Wilson bounds as everything else (verifying task outcomes).

Recap

You now have: a declarative basket, redacted local evidence for every run, a statistical gate wired to CI exit codes, a named baseline accumulating history, and two quality axes (checkpoints, verifier) that survive prompt rewrites. Recipes for all of it — sensitivity at small k, the paired sign test, external scores, the accuracy-vs-cost Pareto table — live in the workflows guide.

---

🧠 How it works

Catacomb reduces each session's transcripts (the main session plus every subagent sub-transcript) into one deterministic execution graph:

session
  └─ user_prompt
       └─ assistant_turn
            ├─ tool_call
            ├─ mcp_call
            └─ subagent
                 └─ user_prompt
                      └─ ...

Cross-run identity rides two keys: a step key hashes each call's redacted, salient input, so "the same logical step" aligns across runs even though every node ID differs; a phase key names checkpoint windows, so comparisons survive even when prompt churn re-keys the steps. Groups of runs are then aggregated and compared per ADR-0022: Wilson bounds for rates, IQR noise bands for metrics, an exact sign test for paired per-task drift. The full pipeline — bench runner, redacted capture, reducer, keys, gate, baselines, history — is implemented end-to-end under a 100%-test-coverage TDD gate; catacomb deliberately ships no live viewer (ADR-0026). The deeper story is in concepts.

---

🔬 Methodology

The gate's design follows the published eval literature, not house heuristics:

• Repeated, isolated trials; tasks from real failures; outcome-over-path scoring — Anthropic, Demystifying evals for AI agents (2026).
• Group comparison with error bars; paired designs as free variance reduction — Anthropic, A statistical approach to model evals; Miller, Adding Error Bars to Evals.
• Wilson bounds and the exact sign test are the right small-n tools — naive CLT-based intervals undercover below a few hundred datapoints: Bowyer et al., Don't use the CLT for LLM evals (ICML 2025).
• pass^k reporting and deterministic final-state verification — τ-bench (ICLR 2025), which catacomb's reliability block and verifier model follow.
• The harness and its transcripts are a first-class reliability concern — transcript inspection catches shortcuts that pass outcome verifiers: Holistic Agent Leaderboard (ICLR 2026).
• Verifiers must themselves be validated — annotators flagged 61.1% of sampled SWE-bench tasks for unit tests that may unfairly reject valid solutions (SWE-bench Verified), and vetting reduces but does not eliminate the bias (SWE-Bench+). Hence catacomb's verifier contract keeps comparators total, offline-re-runnable, and out of the agent's reach.
• LLM judges do not replace deterministic checks — even expert-designed graders trail human inter-rater agreement (GDPval); judge protocol discipline per OpenAI's evaluation best practices. Catacomb gates on deterministic observables and lets external scorers ride the same mechanism instead of baking a judge in.

Further reading on domain benchmarks: Spider 2.0, ELT-Bench, the BIRD family.

---

🔒 Privacy

Catacomb runs no daemon and opens no sockets — everything is local files. Evidence transcripts pass through secret redaction on the write path (API keys, tokens, private keys, connection strings, high-entropy values → typed markers). It is best-effort, not a guarantee — the classes it deliberately does not catch are listed under known residuals. Graphs and step keys hash only post-redaction content; the SQLite store holds baselines and reports, never transcripts or payloads.

---

📚 Documentation & Development

Reading order:

1. This README — install, tutorial, the mental model
2. Concepts — the action graph, step keys, phases
3. Workflows — recipes: baselines, trends, verifiers, external scores, diff, export
4. CLI reference — every command, flag, and exit code
5. Configuration · Ingestion · Privacy & operations

Design depth: design spec · ADRs · implementation plans · release process · contributor & agent guide.

make build   # build bin/catacomb
make test    # tests with -race + coverage profile
make cover   # enforce the 100% coverage gate
make lint    # golangci-lint

---

🙏 Contribution

• Found a bug? Open an issue with reproduction steps and expected vs. actual behavior.
• Have a question? Ping @karych on Telegram, or open an issue.
• Want a feature? Open an issue describing the use case.
• Ready to contribute code? Read AGENTS.md first — the repo runs under a 100%-test-coverage, TDD-first gate. Fork, branch, PR (tag @realkarych).

Your feedback and contributions are always welcome 💙.

---

⚖️ License

Apache-2.0.