dbtrail

module

v0.112.0 Latest Latest Go to latest Published: Oct 21, 2026 License: Apache-2.0

Details

Valid go.mod file
Redistributable license
Tagged version
Stable version
Learn more about best practices

Repository

github.com/dbtrail/dbtrail

Links

Open Source Insights

README ¶

DBTrail keeps your MySQL tables as Parquet files you own, minutes behind the source, with every version of every row. Query them with DuckDB. Production never sees the query.

$ duckdb -init views.sql
D SELECT c.tier, count(*) AS orders, round(sum(o.total), 2) AS revenue
  FROM demo.orders o
  JOIN demo.customers c ON c.id = o.customer_id
  GROUP BY c.tier ORDER BY revenue DESC;
┌──────────┬────────┬───────────────┐
│   tier   │ orders │    revenue    │
│ varchar  │ int64  │ decimal(38,2) │
├──────────┼────────┼───────────────┤
│ platinum │ 539848 │   59426197.40 │
│ gold     │ 539752 │   59353658.54 │
│ silver   │ 539056 │   59295443.48 │
│ bronze   │ 538385 │   59242795.95 │
└──────────┴────────┴───────────────┘

That query read Parquet files. The MySQL primary never saw it.

Minutes behind the source. Not high availability, not a backup. Limits

Why

Reports, dashboards and ad-hoc analysis compete with your application when they run on the primary: a full scan pushes the working set out of the buffer pool, a long read holds back purge, and a GROUP BY over millions of rows spills temp tables to disk. A reporting replica is one more MySQL to pay for and patch, and it still answers at row-store speed.

DBTrail gives those queries a copy of their own, in a columnar format, outside mysqld.

How it works

The first copy reads your tables once (with mydumper).
After that, only the binlog. DBTrail connects over the replication protocol, the way a replica does. No plugin, no agent on the database host, no triggers. That is why Amazon RDS and Aurora work.
On your schedule, the copy is updated. DBTrail folds the recorded changes into a new Parquet snapshot, from every 5 minutes to once a day. The source is not read again, unless a table changes shape (see Limits).
You query it with DuckDB. A generated views.sql gives you one view per table, named like the source table (shop.orders), that follows the newest snapshot.

One Docker Compose stack, and a folder or an S3 bucket. No pipeline to build, no warehouse to run, no Kafka. See Analytics with DuckDB.

The numbers

Source: RDS MySQL 8.4, TPC-C dataset, 73 million rows, about 10 GB, under a sysbench-tpcc load of 180 transactions per second. Measured 18 to 20 September 2026 with DBTrail 0.84 refreshing every 5 minutes. Six report queries, total time:

Where the queries ran	Total time
MySQL	22 min 28 s
ClickHouse fed by Airbyte CDC	12.6 s
DBTrail + DuckDB	4.6 s
MyDuck Server	2.7 s

One of the six, a two-table join grouped by district and month, took 3 min 52 s on MySQL and 0.9 s on the copy. MyDuck was faster on this set: it keeps data in DuckDB's own format, which only DuckDB reads.

Freshness, measured separately under about 340 transactions per second with a 5-minute schedule: commit to visible in DuckDB took 2.7 to 13.2 minutes, median 5.9.

Method, cost comparison and ClickBench results: Introducing the MySQL Analytical Replica.

What you get

An analytical copy

Heavy queries off your MySQL. They run in DuckDB, in its own process, on the copy. Only the binlog leaves mysqld.
DuckDB, ready to open. One views.sql, one view per table. Your DuckDB runs it on your machine: a laptop, a notebook, a BI box.
Open files you own. Plain Parquet on disk or in S3, with a Hive-partitioned change history. Nothing proprietary.
Other engines too. Export a snapshot to Iceberg for Spark, Trino or Athena with bintrail export iceberg. See Iceberg export.

Open the copy through views.sql. An engine that reads the Parquet files directly sees each table as of its last full write and can miss the change files stored beside it. The views apply them for you.

Time travel and recovery, from the same stream

DBTrail keeps every change on your MySQL server, before and after, and writes the SQL that undoes the ones you didn't want. It reads them from the same binlog as the copy, from the day you install it:

Every version of every row. See any row or table as it was at a past moment, from the web interface or the reconstruct CLI. An optional MySQL port (beta) answers the same questions with AS OF SQL from any mysql client. See Time-Travel SQL.
Undo precisely. Generate the SQL that reverses just the damaged rows. recover-cascade also rebuilds the child rows an ON DELETE CASCADE removed below the binary log. DBTrail writes the SQL. You review it and you run it. See Query & Recovery.
Prove it holds. bintrail verify checks, from two snapshots and the index and without touching the source, that a recovery would reproduce it. bintrail status flags any gap the capture could not fill. See Verify.
Web interface and MCP. Add servers, schedule snapshots and browse changes in the browser. Connect Claude Desktop or any MCP client to ask in plain English. Every tool is read-only. See the 5-minute guide.

Try time travel in 30 seconds, with nothing of yours connected:

docker run --rm -p 6033:6033 ghcr.io/dbtrail/bintrail-demo

See the demo image.

Limits

A copy you can trust is one whose edges you know.

Not high availability. Your application never points at the copy. Nothing fails over to it.
Not your backup. History starts the day you install it. Keep your physical backups.
Minutes behind, not seconds, for the files. The copy refreshes on the schedule you set. Through the SQL port, a heavy read also sees what capture has indexed since, usually seconds behind.
Needs a ROW binlog with full row images. binlog_format=ROW and binlog_row_image=FULL. bintrail doctor checks both and prints the fix (on RDS and Aurora, set them in the parameter group).
Some changes mean a full read. Adding, dropping, renaming or retyping a column, or a TRUNCATE, DROP or RENAME of a table, pauses the scheduled update until DBTrail reads the database again, at most once a day. That read takes a short lock: a global read lock while it starts, or, on RDS and Aurora, a lock on the tables it reads. A new table joins with a read of its own. Index and other everyday changes do not trigger one.
History needs a bucket. Without S3, row history keeps 48 hours by default (--rotate-retain changes it). The copy itself stays: the newest 3 snapshots per table on local disk.
Cascaded deletes before MySQL 9.6, and on MariaDB. Rows deleted by a foreign key never reach the binlog. recover-cascade rebuilds most of them, and the copy keeps them until DBTrail next reads the database in full.
Not a MySQL server. Reports run in DuckDB, not from your MySQL client. The optional MySQL port (beta, no TLS) only answers what a row or table looked like at a past moment, or what changed between two.
Yours to operate. DBTrail and its small index MySQL run on your machines. Disk and backups are yours.

Full list: Limitations.

What it works with

Source	Analytical copy	Row history and undo
MySQL 8.0, 8.4	Yes	Yes
Percona Server for MySQL 8.0, 8.4	Yes	Yes
Amazon RDS for MySQL	Yes (verified)	Yes (verified)
Amazon Aurora MySQL	Yes (verified)	Yes (verified)
Google Cloud SQL for MySQL	Should work, please report issues	Should work
MariaDB 10.11+, including Amazon RDS for MariaDB	Yes	Yes. See MariaDB source
PostgreSQL 14+	One-time copy only, no scheduled update yet	Beta. See PostgreSQL source

DBTrail never needs the binlog files on disk, which is why managed cloud databases work.

Install

You need Docker with Compose, and DuckDB on the machine where you query.

curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/main/install.sh | sh

This downloads the Docker Compose stack, starts it, waits until DBTrail answers, and prints the next steps.

Sign in. Open http://127.0.0.1:8090, create a username and password, and press Create & sign in.
Connect. Click + Add server, enter the host and port of your MySQL and press Find it. The form shows the SQL that creates DBTrail's user. Run it on your MySQL, then press I ran it. DBTrail checks the rest and starts on its own.
First change. Change a row on your MySQL. It shows under Recent changes on the Overview within a minute, with an Undo that writes the SQL to reverse it.
First copy. On Snapshots, press Read database now. Normally this is the only time DBTrail reads your tables directly.
Set a schedule. On the Settings tab of Snapshots, under Update the copy, pick how often (every 5 minutes, for example) and press Turn on. Each run after the first folds the recorded changes instead of reading the source.
Query it. On Snapshots, press Download, then Download the data. Unpack the .tar.gz and run duckdb -init views.sql inside the snapshot folder. To read the copy that keeps updating, see Query in DuckDB.

Prefer the command line? See the command-line quickstart.

Who builds it

DBTrail is built by Daniel Guzman-Burgos, former MySQL Technical Lead at Percona.

Documentation

Start here	Reference	Operations
Install	Analytics with DuckDB	Deployment · Capacity
Start page · Command-line quickstart	Dump & Baseline · Iceberg export	Rotation & Status
DBA guide	Query & Recovery · Time-Travel SQL	Docker
30-second demo	Verify recoveries · Web interface	Upload to S3 · S3 IAM policy · Upgrading
Limitations	Streaming · Indexing · DDL tracking	Server identity
	Connect an AI assistant · MCP server	Parquet debugging
	MariaDB source · PostgreSQL source

Privacy

bintrail is DBTrail's command-line tool. It runs entirely in your infrastructure. Your database data never leaves it: not your rows, schemas, table names, queries, hostnames, DSNs, or file paths.

Official release builds do report metadata-only usage statistics: which command ran, whether it succeeded, a coarse error class, and your version and platform. It is on by default and turns off in one line:

bintrail telemetry off      # or: DO_NOT_TRACK=1, BINTRAIL_TELEMETRY=off
bintrail telemetry show     # prints what would be sent; sends nothing

No identifier of any kind is stored or transmitted, so nothing ties those statistics to you or your machine. A binary you build yourself has no reporting address compiled in and cannot send anything. TELEMETRY.md documents every field, every control, and the CI tests that enforce both.

The privacy policy covers the Claude Desktop extension (.mcpb), which reports nothing at all, and how data moves when an AI client queries your deployment.

License

Apache-2.0: free for any use, including commercial and production. Contributions are welcome; see CONTRIBUTING.md (a CLA is required, prompted automatically on your first PR).

Want the index server operated for you (sized, backed up, upgraded, kept alive on-call) instead of running it yourself? That is the managed service at dbtrail.com. SUPPORT.md draws the ship-vs-operate line.

Directories ¶

?	: This menu
/	: Search site
f or F	: Jump to
y or Y	: Canonical URL

Path	Synopsis
assurance Package assurance exposes read-only access to the backup-assurance signals — restore coverage, the capture-continuity verdict, baseline staleness, and the scheduled-verify run history — for tooling and embedding distributions that import the core as a module.	Package assurance exposes read-only access to the backup-assurance signals — restore coverage, the capture-continuity verdict, baseline staleness, and the scheduled-verify run history — for tooling and embedding distributions that import the core as a module.
cliapp
cmd
bintrail command Command bintrail is the OSS binary.	Command bintrail is the OSS binary.
bintrail-console command Command bintrail-console serves the read-only Bintrail web console as a standalone binary, decoupled from the core `bintrail` CLI.	Command bintrail-console serves the read-only Bintrail web console as a standalone binary, decoupled from the core `bintrail` CLI.
bintrail-mcp command Bridge mode (--connect): a stdio ↔ Streamable-HTTP proxy.	Bridge mode (--connect): a stdio ↔ Streamable-HTTP proxy.
bintrail-pg command Command bintrail-pg is the PostgreSQL-source sibling of the core bintrail binary.	Command bintrail-pg is the PostgreSQL-source sibling of the core bintrail binary.
consoleapp Package consoleapp is the importable command layer of the bintrail-console binary — the console sibling of the cliapp seam.	Package consoleapp is the importable command layer of the bintrail-console binary — the console sibling of the cliapp seam.
ext Package ext exposes the extension points that embedding distributions — builds that import cliapp and wrap the OSS core — use to inject behavior: an audit sink recording data-access operations, extra doctor checks, agent WebSocket commands, and daemon source jobs.	Package ext exposes the extension points that embedding distributions — builds that import cliapp and wrap the OSS core — use to inject behavior: an audit sink recording data-access operations, extra doctor checks, agent WebSocket commands, and daemon source jobs.
mcpext Package mcpext is the extension seam for MCP tools: an embedding distribution registers a provider here and its tools appear on every MCP server the core builds — the standalone bintrail-mcp binary and the web console's /mcp endpoint alike.	Package mcpext is the extension seam for MCP tools: an embedding distribution registers a provider here and its tools appear on every MCP server the core builds — the standalone bintrail-mcp binary and the web console's /mcp endpoint alike.
indexquery Package indexquery exposes read-only programmatic access to the bintrail index — live MySQL partitions and Parquet archives merged — for tooling and embedding distributions that import the core as a module.	Package indexquery exposes read-only programmatic access to the bintrail index — live MySQL partitions and Parquet archives merged — for tooling and embedding distributions that import the core as a module.
internal
accessprofiles Package accessprofiles is the ONE implementation of authoring access profiles: the flags on tables and columns (table_flags), the named profiles (profiles) and the allow/deny rules that join them (access_rules).	Package accessprofiles is the ONE implementation of authoring access profiles: the flags on tables and columns (table_flags), the named profiles (profiles) and the allow/deny rules that join them (access_rules).
agent Package agent implements the outbound communication channel between the bintrail agent (running in customer infrastructure) and the dbtrail service.	Package agent implements the outbound communication channel between the bintrail agent (running in customer infrastructure) and the dbtrail service.
archive Package archive writes a binlog_events partition to a Parquet file before it is dropped.	Package archive writes a binlog_events partition to a Parquet file before it is dropped.
audittest Package audittest holds the canonical list of audit emissions the core must produce, plus the recording sink the contract tests drive it with.	Package audittest holds the canonical list of audit emissions the core must produce, plus the recording sink the contract tests drive it with.
baseline Package baseline converts mydumper output into Parquet files, enabling full audit reconstruction when combined with binlog change events.	Package baseline converts mydumper output into Parquet files, enabling full audit reconstruction when combined with binlog change events.
baselineintegrity Package baselineintegrity provides at-rest integrity for baseline Parquet files: a CRC-32C _MANIFEST sidecar written next to the _SUCCESS marker, validated on every baseline read path — local (#636) and S3 (#698).	Package baselineintegrity provides at-rest integrity for baseline Parquet files: a CRC-32C _MANIFEST sidecar written next to the _SUCCESS marker, validated on every baseline read path — local (#636) and S3 (#698).
buffer Package buffer provides an in-memory event buffer for BYOS mode.	Package buffer provides an in-memory event buffer for BYOS mode.
byos Package byos implements the Bring Your Own Storage (BYOS) data separation layer.	Package byos implements the Bring Your Own Storage (BYOS) data separation layer.
cascade Package cascade reconstructs the side effects of an InnoDB foreign-key cascade that InnoDB applied but never wrote to the binary log — ON DELETE CASCADE (deleted child rows), ON DELETE SET NULL (nulled child FKs), and their ON UPDATE siblings (child FKs rewritten to the parent's new key, or nulled, when the parent's referenced key is UPDATEd; #1002).	Package cascade reconstructs the side effects of an InnoDB foreign-key cascade that InnoDB applied but never wrote to the binary log — ON DELETE CASCADE (deleted child rows), ON DELETE SET NULL (nulled child FKs), and their ON UPDATE siblings (child FKs rewritten to the parent's new key, or nulled, when the parent's referenced key is UPDATEd; #1002).
cascadebaseline Package cascadebaseline implements cascade.BaselineProvider over internal/reconstruct — the Phase-2 fallback that lets cascade recovery recover a child row that has no binlog event in the lookback window by reading it out of a `bintrail baseline` Parquet snapshot.	Package cascadebaseline implements cascade.BaselineProvider over internal/reconstruct — the Phase-2 fallback that lets cascade recovery recover a child row that has no binlog event in the lookback window by reading it out of a `bintrail baseline` Parquet snapshot.
cascaderecover Package cascaderecover assembles the cascade-recovery SQL script (the documented preamble, the SET FOREIGN_KEY_CHECKS=0/1 wrapper, the CASCADE re-INSERTs, and the idempotent SET NULL restorations) from a synthesized cascade result.	Package cascaderecover assembles the cascade-recovery SQL script (the documented preamble, the SET FOREIGN_KEY_CHECKS=0/1 wrapper, the CASCADE re-INSERTs, and the idempotent SET NULL restorations) from a synthesized cascade result.
cli Package cli holds command-layer building blocks intended to be shared across bintrail binaries.	Package cli holds command-layer building blocks intended to be shared across bintrail binaries.
cliutil Package cliutil provides shared filter-parsing and output helpers used by both cmd/bintrail/ commands and cmd/bintrail-mcp/.	Package cliutil provides shared filter-parsing and output helpers used by both cmd/bintrail/ commands and cmd/bintrail-mcp/.
config
consistency Package consistency provides primitives that detect, with overwhelming probability, whether a Parquet snapshot diverges from the MySQL table it was taken from.	Package consistency provides primitives that detect, with overwhelming probability, whether a Parquet snapshot diverges from the MySQL table it was taken from.
console Package console serves an embedded, read-only, single-operator web UI over a bintrail index.	Package console serves an embedded, read-only, single-operator web UI over a bintrail index.
copymirror Package copymirror keeps a LOCAL mirror of the snapshots a server publishes to S3, so SQL on the copy can read them (#2302).	Package copymirror keeps a LOCAL mirror of the snapshots a server publishes to S3, so SQL on the copy can read them (#2302).
ddltext Package ddltext reads the text of a table DDL statement: its verb, the tables it names and, for an ALTER TABLE that only adds columns, the columns.	Package ddltext reads the text of a table DDL statement: its verb, the tables it names and, for an ALTER TABLE that only adds columns, the columns.
doctor
duckdbtuning Package duckdbtuning holds the shared DuckDB resource-tuning flags used by the offline query/recover/reconstruct/verify commands (#510/#511): the --ultrafast / --duckdb-threads / --duckdb-memory-limit flag set, the precedence resolution into a duckdbutil.Tuning, and the archive-fetcher adapter that carries that budget into parquetquery.	Package duckdbtuning holds the shared DuckDB resource-tuning flags used by the offline query/recover/reconstruct/verify commands (#510/#511): the --ultrafast / --duckdb-threads / --duckdb-memory-limit flag set, the precedence resolution into a duckdbutil.Tuning, and the archive-fetcher adapter that carries that budget into parquetquery.
duckdbutil Package duckdbutil holds small helpers shared by the DuckDB sessions bintrail opens (baseline reads, snapshot queries, S3 footer probes).	Package duckdbutil holds small helpers shared by the DuckDB sessions bintrail opens (baseline reads, snapshot queries, S3 footer probes).
event Package event holds the source-agnostic row-event type that flows from a capture backend into the indexer and the whole downstream value stack (query, recover, reconstruct, shim, console).	Package event holds the source-agnostic row-event type that flows from a capture backend into the indexer and the whole downstream value stack (query, recover, reconstruct, shim, console).
icebergexport Package icebergexport writes the current state of indexed tables to Apache Iceberg tables, incrementally (#1466).	Package icebergexport writes the current state of indexed tables to Apache Iceberg tables, incrementally (#1466).
indexer Package indexer consumes parsed binlog events and batch-inserts them into the binlog_events table in the index MySQL database.	Package indexer consumes parsed binlog events and batch-inserts them into the binlog_events table in the index MySQL database.
installid Package installid derives the replication server-id capture uses: the source's own triple (internal/serverid) and what tells this installation apart from another one capturing the same source.	Package installid derives the replication server-id capture uses: the source's own triple (internal/serverid) and what tells this installation apart from another one capturing the same source.
mcptools Package mcptools implements the read-only Bintrail MCP tools — query, recover, status, list_schema_changes, and the opt-in reconstruct (#953) and recover_cascade (#1128) — decoupled from how the index connection is obtained.	Package mcptools implements the read-only Bintrail MCP tools — query, recover, status, list_schema_changes, and the opt-in reconstruct (#953) and recover_cascade (#1128) — decoupled from how the index connection is obtained.
metadata
mydumperlock Package mydumperlock checks whether a source grants the privileges mydumper's point-consistent sync mode needs, BEFORE mydumper is launched.	Package mydumperlock checks whether a source grants the privileges mydumper's point-consistent sync mode needs, BEFORE mydumper is launched.