README
¶
DBTrail keeps your MySQL tables as Parquet in your bucket, minutes behind the source, and every version of every row. Query them with DuckDB. Production never sees the query.
$ duckdb -init views.sql
D SELECT status, count(*) FROM state_shop_orders GROUP BY 1;
That query reads Parquet in your bucket. The MySQL primary never saw it.
What you get
Reports, dashboards and ad-hoc analysis compete with your application when they run on the primary: full scans push the working set out of the buffer pool, and a long read holds back purge. A reporting replica is one more MySQL to pay for and patch, answering at MySQL speed. DBTrail gives those queries a copy of their own, one that also remembers the past.
An analytical copy
- Zero ETL. One full copy of each table, then only the binary log. DBTrail folds the changes into a new Parquet snapshot on your schedule, from every 5 minutes to once a day, without reading the source again. No pipeline, no warehouse, no Kafka. See Analytics with DuckDB.
- DuckDB, ready to open. One
views.sqlgives you one view per table,state_<schema>_<table>, that follows the newest snapshot on its own. Your DuckDB runs it, on your machine: a laptop, a notebook, a BI box. - Open files, your bucket. Plain Parquet on disk or in S3, Hive-partitioned
change history, nothing proprietary. Hand a table to Spark, Trino or Athena
with
bintrail export iceberg. See Iceberg export. - Nothing inside the server. No plugin, no fork, no agent on the database host. DBTrail connects the way a replica does, so it works where you cannot load an engine: Amazon RDS and Aurora.
Time travel and recovery, from the same stream
DBTrail keeps every change on your MySQL server, before and after, and writes the SQL that undoes the ones you didn't want.
- Every version of every row. Query any row or table as it was at a
past moment, from the web interface or the
reconstructCLI; the live SQLAS OFinterface also needs ProxySQL. See Time-Travel SQL. - Undo precisely. Generate the SQL that reverses just the damaged rows,
including the child rows an
ON DELETE CASCADEremoved below the binary log. See Query & Recovery. - Prove it holds.
bintrail verifychecks offline, without touching the source, that a recovery would reproduce it.bintrail statusflags any gap in the captured stream. See Verify. - Web interface and MCP. Add servers, schedule snapshots and browse changes in the browser. Connect Claude Desktop or any MCP client to ask in plain English (5-minute guide).
Works with MySQL, Percona Server for MySQL, Amazon RDS for MySQL,
and Amazon Aurora MySQL (verified). Google Cloud SQL for MySQL should
work too; please report issues. MariaDB is supported as an alpha source
(MariaDB source). DBTrail connects over the replication
protocol and never needs the binlog files on disk, which is why managed cloud
databases work. It requires MySQL 8.0+ with binlog_format=ROW and
binlog_row_image=FULL; bintrail doctor checks both and prints the exact fix.
What it is not. Not high availability: your application never points at the copy. Not a replacement for your physical backups: history starts the day you install it. Minutes behind the source, not seconds: the snapshot interval is your setting. See Analytics with DuckDB.
Try it in 30 seconds
One container, zero setup, time-travel SQL:
docker run --rm -p 6033:6033 ghcr.io/dbtrail/bintrail-demo
See the demo image.
Install
curl -fsSL https://raw.githubusercontent.com/dbtrail/dbtrail/main/install.sh | sh
This downloads the Docker Compose stack, starts it, waits until DBTrail answers, and prints the next steps. The first four are the same as the start page:
- Sign in. Open http://127.0.0.1:8090 and create a username and password.
- Connect. Click + Add server, give the server a name, and fill in the host and port of your MySQL server. The form suggests a user and password for DBTrail and shows the SQL that creates that user; run it on your MySQL with a login that can create users and grant them privileges, then press Save.
- First change. Change a row on your MySQL. It shows on the Overview within a minute, with an Undo that writes the SQL to reverse it.
- First copy. The first snapshot is the analytical copy. Today it takes a folder created first, then on the Snapshots page: type it in Backup dir, Save, and press Create backup. The start page has the command that creates the folder. Set a schedule on the same page (every 5 minutes, for example) and each run after the first updates the copy from the recorded changes instead of reading the source.
- Query it. On the MCP Server page, Download a DuckDB schema, then open it with
duckdb -init views.sql. See Analytics with DuckDB.
Prefer the command line? See the command-line quickstart.
Documentation
Privacy
bintrail runs entirely in your infrastructure. Your database data never leaves it: not your rows, schemas, table names, queries, hostnames, DSNs, or file paths.
Official release builds do report metadata-only usage statistics: which command ran, whether it succeeded, a coarse error class, and your version and platform. It is on by default and turns off in one line:
bintrail telemetry off # or: DO_NOT_TRACK=1, BINTRAIL_TELEMETRY=off
bintrail telemetry show # prints what would be sent; sends nothing
No identifier of any kind is stored or transmitted, so nothing ties those statistics to you or your machine. A binary you build yourself has no reporting address compiled in and cannot send anything. TELEMETRY.md documents every field, every control, and the CI tests that enforce both.
The privacy policy covers the Claude Desktop extension (.mcpb),
which reports nothing at all, and how data moves when an AI client queries
your deployment.
License
Apache-2.0: free for any use, including commercial and production. Contributions are welcome; see CONTRIBUTING.md (a CLA is required, prompted automatically on your first PR).
Want the index server operated for you (sized, backed up, upgraded, kept alive on-call) instead of running it yourself? That is the managed service at dbtrail.com. SUPPORT.md draws the ship-vs-operate line.
Directories
¶
| Path | Synopsis |
|---|---|
|
Package assurance exposes read-only access to the backup-assurance signals — restore coverage, the capture-continuity verdict, baseline staleness, and the scheduled-verify run history — for tooling and embedding distributions that import the core as a module.
|
Package assurance exposes read-only access to the backup-assurance signals — restore coverage, the capture-continuity verdict, baseline staleness, and the scheduled-verify run history — for tooling and embedding distributions that import the core as a module. |
|
cmd
|
|
|
bintrail
command
Command bintrail is the OSS binary.
|
Command bintrail is the OSS binary. |
|
bintrail-console
command
Command bintrail-console serves the read-only Bintrail web console as a standalone binary, decoupled from the core `bintrail` CLI.
|
Command bintrail-console serves the read-only Bintrail web console as a standalone binary, decoupled from the core `bintrail` CLI. |
|
bintrail-mcp
command
Bridge mode (--connect): a stdio ↔ Streamable-HTTP proxy.
|
Bridge mode (--connect): a stdio ↔ Streamable-HTTP proxy. |
|
bintrail-pg
command
Command bintrail-pg is the PostgreSQL-source sibling of the core bintrail binary.
|
Command bintrail-pg is the PostgreSQL-source sibling of the core bintrail binary. |
|
Package consoleapp is the importable command layer of the bintrail-console binary — the console sibling of the cliapp seam.
|
Package consoleapp is the importable command layer of the bintrail-console binary — the console sibling of the cliapp seam. |
|
Package ext exposes the extension points that embedding distributions — builds that import cliapp and wrap the OSS core — use to inject behavior: an audit sink recording data-access operations, extra doctor checks, agent WebSocket commands, and daemon source jobs.
|
Package ext exposes the extension points that embedding distributions — builds that import cliapp and wrap the OSS core — use to inject behavior: an audit sink recording data-access operations, extra doctor checks, agent WebSocket commands, and daemon source jobs. |
|
mcpext
Package mcpext is the extension seam for MCP tools: an embedding distribution registers a provider here and its tools appear on every MCP server the core builds — the standalone bintrail-mcp binary and the web console's /mcp endpoint alike.
|
Package mcpext is the extension seam for MCP tools: an embedding distribution registers a provider here and its tools appear on every MCP server the core builds — the standalone bintrail-mcp binary and the web console's /mcp endpoint alike. |
|
Package indexquery exposes read-only programmatic access to the bintrail index — live MySQL partitions and Parquet archives merged — for tooling and embedding distributions that import the core as a module.
|
Package indexquery exposes read-only programmatic access to the bintrail index — live MySQL partitions and Parquet archives merged — for tooling and embedding distributions that import the core as a module. |
|
internal
|
|
|
accessprofiles
Package accessprofiles is the ONE implementation of authoring access profiles: the flags on tables and columns (table_flags), the named profiles (profiles) and the allow/deny rules that join them (access_rules).
|
Package accessprofiles is the ONE implementation of authoring access profiles: the flags on tables and columns (table_flags), the named profiles (profiles) and the allow/deny rules that join them (access_rules). |
|
agent
Package agent implements the outbound communication channel between the bintrail agent (running in customer infrastructure) and the dbtrail service.
|
Package agent implements the outbound communication channel between the bintrail agent (running in customer infrastructure) and the dbtrail service. |
|
archive
Package archive writes a binlog_events partition to a Parquet file before it is dropped.
|
Package archive writes a binlog_events partition to a Parquet file before it is dropped. |
|
audittest
Package audittest holds the canonical list of audit emissions the core must produce, plus the recording sink the contract tests drive it with.
|
Package audittest holds the canonical list of audit emissions the core must produce, plus the recording sink the contract tests drive it with. |
|
baseline
Package baseline converts mydumper output into Parquet files, enabling full audit reconstruction when combined with binlog change events.
|
Package baseline converts mydumper output into Parquet files, enabling full audit reconstruction when combined with binlog change events. |
|
baselineintegrity
Package baselineintegrity provides at-rest integrity for baseline Parquet files: a CRC-32C _MANIFEST sidecar written next to the _SUCCESS marker, validated on every baseline read path — local (#636) and S3 (#698).
|
Package baselineintegrity provides at-rest integrity for baseline Parquet files: a CRC-32C _MANIFEST sidecar written next to the _SUCCESS marker, validated on every baseline read path — local (#636) and S3 (#698). |
|
buffer
Package buffer provides an in-memory event buffer for BYOS mode.
|
Package buffer provides an in-memory event buffer for BYOS mode. |
|
byos
Package byos implements the Bring Your Own Storage (BYOS) data separation layer.
|
Package byos implements the Bring Your Own Storage (BYOS) data separation layer. |
|
cascade
Package cascade reconstructs the side effects of an InnoDB foreign-key cascade that InnoDB applied but never wrote to the binary log — ON DELETE CASCADE (deleted child rows), ON DELETE SET NULL (nulled child FKs), and their ON UPDATE siblings (child FKs rewritten to the parent's new key, or nulled, when the parent's referenced key is UPDATEd; #1002).
|
Package cascade reconstructs the side effects of an InnoDB foreign-key cascade that InnoDB applied but never wrote to the binary log — ON DELETE CASCADE (deleted child rows), ON DELETE SET NULL (nulled child FKs), and their ON UPDATE siblings (child FKs rewritten to the parent's new key, or nulled, when the parent's referenced key is UPDATEd; #1002). |
|
cascadebaseline
Package cascadebaseline implements cascade.BaselineProvider over internal/reconstruct — the Phase-2 fallback that lets cascade recovery recover a child row that has no binlog event in the lookback window by reading it out of a `bintrail baseline` Parquet snapshot.
|
Package cascadebaseline implements cascade.BaselineProvider over internal/reconstruct — the Phase-2 fallback that lets cascade recovery recover a child row that has no binlog event in the lookback window by reading it out of a `bintrail baseline` Parquet snapshot. |
|
cascaderecover
Package cascaderecover assembles the cascade-recovery SQL script (the documented preamble, the SET FOREIGN_KEY_CHECKS=0/1 wrapper, the CASCADE re-INSERTs, and the idempotent SET NULL restorations) from a synthesized cascade result.
|
Package cascaderecover assembles the cascade-recovery SQL script (the documented preamble, the SET FOREIGN_KEY_CHECKS=0/1 wrapper, the CASCADE re-INSERTs, and the idempotent SET NULL restorations) from a synthesized cascade result. |
|
cli
Package cli holds command-layer building blocks intended to be shared across bintrail binaries.
|
Package cli holds command-layer building blocks intended to be shared across bintrail binaries. |
|
cliutil
Package cliutil provides shared filter-parsing and output helpers used by both cmd/bintrail/ commands and cmd/bintrail-mcp/.
|
Package cliutil provides shared filter-parsing and output helpers used by both cmd/bintrail/ commands and cmd/bintrail-mcp/. |
|
consistency
Package consistency provides primitives that detect, with overwhelming probability, whether a Parquet snapshot diverges from the MySQL table it was taken from.
|
Package consistency provides primitives that detect, with overwhelming probability, whether a Parquet snapshot diverges from the MySQL table it was taken from. |
|
console
Package console serves an embedded, read-only, single-operator web UI over a bintrail index.
|
Package console serves an embedded, read-only, single-operator web UI over a bintrail index. |
|
ddltext
Package ddltext reads the text of a table DDL statement: its verb, the tables it names and, for an ALTER TABLE that only adds columns, the columns.
|
Package ddltext reads the text of a table DDL statement: its verb, the tables it names and, for an ALTER TABLE that only adds columns, the columns. |
|
duckdbtuning
Package duckdbtuning holds the shared DuckDB resource-tuning flags used by the offline query/recover/reconstruct/verify commands (#510/#511): the --ultrafast / --duckdb-threads / --duckdb-memory-limit flag set, the precedence resolution into a duckdbutil.Tuning, and the archive-fetcher adapter that carries that budget into parquetquery.
|
Package duckdbtuning holds the shared DuckDB resource-tuning flags used by the offline query/recover/reconstruct/verify commands (#510/#511): the --ultrafast / --duckdb-threads / --duckdb-memory-limit flag set, the precedence resolution into a duckdbutil.Tuning, and the archive-fetcher adapter that carries that budget into parquetquery. |
|
duckdbutil
Package duckdbutil holds small helpers shared by the DuckDB sessions bintrail opens (baseline reads, snapshot queries, S3 footer probes).
|
Package duckdbutil holds small helpers shared by the DuckDB sessions bintrail opens (baseline reads, snapshot queries, S3 footer probes). |
|
event
Package event holds the source-agnostic row-event type that flows from a capture backend into the indexer and the whole downstream value stack (query, recover, reconstruct, shim, console).
|
Package event holds the source-agnostic row-event type that flows from a capture backend into the indexer and the whole downstream value stack (query, recover, reconstruct, shim, console). |
|
icebergexport
Package icebergexport writes the current state of indexed tables to Apache Iceberg tables, incrementally (#1466).
|
Package icebergexport writes the current state of indexed tables to Apache Iceberg tables, incrementally (#1466). |
|
indexer
Package indexer consumes parsed binlog events and batch-inserts them into the binlog_events table in the index MySQL database.
|
Package indexer consumes parsed binlog events and batch-inserts them into the binlog_events table in the index MySQL database. |
|
mcptools
Package mcptools implements the read-only Bintrail MCP tools — query, recover, status, list_schema_changes, and the opt-in reconstruct (#953) and recover_cascade (#1128) — decoupled from how the index connection is obtained.
|
Package mcptools implements the read-only Bintrail MCP tools — query, recover, status, list_schema_changes, and the opt-in reconstruct (#953) and recover_cascade (#1128) — decoupled from how the index connection is obtained. |
|
mydumperlock
Package mydumperlock checks whether a source grants the privileges mydumper's point-consistent sync mode needs, BEFORE mydumper is launched.
|
Package mydumperlock checks whether a source grants the privileges mydumper's point-consistent sync mode needs, BEFORE mydumper is launched. |
|
notify
Package notify is the console's outbound webhook notification channel: a generic JSON POST for the events that mean "your safety net has a hole" — continuity gap_lost, verify problems, rotation making no progress — plus edge-triggering so a persistent condition alerts on its transition (and once on recovery), not on every cycle.
|
Package notify is the console's outbound webhook notification channel: a generic JSON POST for the events that mean "your safety net has a hole" — continuity gap_lost, verify problems, rotation making no progress — plus edge-triggering so a persistent condition alerts on its transition (and once on recovery), not on every cycle. |
|
observe
Package observe provides shared observability setup for Bintrail: structured logging via log/slog and Prometheus metrics for the stream command.
|
Package observe provides shared observability setup for Bintrail: structured logging via log/slog and Prometheus metrics for the stream command. |
|
parquetquery
Package parquetquery implements a DuckDB-backed query engine for Parquet archive files.
|
Package parquetquery implements a DuckDB-backed query engine for Parquet archive files. |
|
parser
Package parser reads MySQL ROW-format binlog files and emits typed events.
|
Package parser reads MySQL ROW-format binlog files and emits typed events. |
|
pgbaseline
Package pgbaseline produces Parquet baseline snapshots directly from a live PostgreSQL source (#593 slice C) — the PG sibling of the mydumper→Parquet converter in internal/baseline, with no dump step: it COPYs each published table inside one REPEATABLE READ snapshot and streams the rows into internal/baseline.Writer.
|
Package pgbaseline produces Parquet baseline snapshots directly from a live PostgreSQL source (#593 slice C) — the PG sibling of the mydumper→Parquet converter in internal/baseline, with no dump step: it COPYs each published table inside one REPEATABLE READ snapshot and streams the rows into internal/baseline.Writer. |
|
pgcapture
Package pgcapture decodes a PostgreSQL logical-replication (pgoutput) stream into the source-neutral event.Event consumed by the indexer and the rest of bintrail's value stack — the PostgreSQL analog of the go-mysql binlog parser in internal/parser.
|
Package pgcapture decodes a PostgreSQL logical-replication (pgoutput) stream into the source-neutral event.Event consumed by the indexer and the rest of bintrail's value stack — the PostgreSQL analog of the go-mysql binlog parser in internal/parser. |
|
pgshim
Package pgshim is the PostgreSQL wire-protocol front-end for bintrail's time-travel engine (#1008).
|
Package pgshim is the PostgreSQL wire-protocol front-end for bintrail's time-travel engine (#1008). |
|
pgstreamrun
Package pgstreamrun is the PostgreSQL streaming CONSUMER: it drives the internal/pgcapture capturer (the producer) into the shared indexer, persisting a durable LSN checkpoint to stream_state.
|
Package pgstreamrun is the PostgreSQL streaming CONSUMER: it drives the internal/pgcapture capturer (the producer) into the shared indexer, persisting a durable LSN checkpoint to stream_state. |
|
pgverifysource
Package pgverifysource is the PostgreSQL live-source provider for internal/verify (#1024): the PG-native table checksum plus the pinned connection and the verify.PGSourceChecksum factory that wires it.
|
Package pgverifysource is the PostgreSQL live-source provider for internal/verify (#1024): the PG-native table checksum plus the pinned connection and the verify.PGSourceChecksum factory that wires it. |
|
query
Package query implements the binlog_events query engine — dynamic SQL construction from filter options and multi-format result rendering.
|
Package query implements the binlog_events query engine — dynamic SQL construction from filter options and multi-format result rendering. |
|
reconstruct
Package reconstruct implements row state reconstruction by combining baseline Parquet snapshots with indexed binlog change events.
|
Package reconstruct implements row state reconstruction by combining baseline Parquet snapshots with indexed binlog change events. |
|
recovery
Package recovery generates reversal SQL from indexed binlog events.
|
Package recovery generates reversal SQL from indexed binlog events. |
|
serverid
Package serverid manages persistent server identities for bintrail.
|
Package serverid manages persistent server identities for bintrail. |
|
shim
Package shim implements an in-process MySQL-protocol server that answers time-travel queries by translating them into bintrail's existing query engine.
|
Package shim implements an in-process MySQL-protocol server that answers time-travel queries by translating them into bintrail's existing query engine. |
|
status
Package status — this file implements the operator acknowledgement of a capture-skip tally (#1314).
|
Package status — this file implements the operator acknowledgement of a capture-skip tally (#1314). |
|
storage
Package storage defines an abstract storage backend interface for managing Parquet files and other binary objects.
|
Package storage defines an abstract storage backend interface for managing Parquet files and other binary objects. |
|
streamdeps
Package streamdeps wires the production helper functions into a streamrun.Deps — the host side of streamrun's dependency-injection seam.
|
Package streamdeps wires the production helper functions into a streamrun.Deps — the host side of streamrun's dependency-injection seam. |
|
telemetry
Package telemetry implements bintrail's usage telemetry: a metadata-only, spool-based reporter for command mix and error classes.
|
Package telemetry implements bintrail's usage telemetry: a metadata-only, spool-based reporter for command mix and error classes. |
|
telemetry/telemetrytest
Package telemetrytest provides shared helpers for the per-daemon telemetry WIRING tests (#1362) — the guards that drive a REAL daemon run function (runStream, runServe, runShim, …) and fail when that daemon's `go client.RunDaemon(ctx, name)` line is deleted.
|
Package telemetrytest provides shared helpers for the per-daemon telemetry WIRING tests (#1362) — the guards that drive a REAL daemon run function (runStream, runServe, runShim, …) and fail when that daemon's `go client.RunDaemon(ctx, name)` line is deleted. |
|
testutil
Package testutil provides shared helpers for integration tests that require a live MySQL connection.
|
Package testutil provides shared helpers for integration tests that require a live MySQL connection. |
|
testutil/fakemysql
Package fakemysql is a test-only MySQL server that answers every statement with one chosen server error number.
|
Package fakemysql is a test-only MySQL server that answers every statement with one chosen server error number. |
|
verify
Package verify reconstructs each table's state from baseline + binlog and compares its content fingerprint against the live source, proving that a recovery would reproduce the source.
|
Package verify reconstructs each table's state from baseline + binlog and compares its content fingerprint against the live source, proving that a recovery would reproduce the source. |
|
verify/verdict
Package verdict is the one rule that turns a verify run's per-table counts into its run-level outcome.
|
Package verdict is the one rule that turns a verify run's per-table counts into its run-level outcome. |
|
views
Package views generates DuckDB view definitions over a bintrail archive layout.
|
Package views generates DuckDB view definitions over a bintrail archive layout. |
|
test
|
|
|
e2e
Package e2e holds the full-pipeline end-to-end test for the bintrail CLI (init → snapshot → index → query → recover → status → rotate).
|
Package e2e holds the full-pipeline end-to-end test for the bintrail CLI (init → snapshot → index → query → recover → status → rotate). |
|
installer
Package installer tests install.sh, the one-command installer, by running it for real against stand-ins for docker, curl and lsof: the port handling is decided by the script, so it is tested as a script (#1768).
|
Package installer tests install.sh, the one-command installer, by running it for real against stand-ins for docker, curl and lsof: the port handling is decided by the script, so it is tested as a script (#1768). |
|
shim
Package shim_e2e holds the docker-compose-driven end-to-end test for `bintrail shim`.
|
Package shim_e2e holds the docker-compose-driven end-to-end test for `bintrail shim`. |
Click to show internal directories.
Click to hide internal directories.