README
¶
Urutau
Tupi–Guaraní for the potoo — a nightjar that stands motionless through the night, watching. A fitting name for a process that spends its life quietly watching a binlog.
v0.3.0 · Go ≥ 1.26 · license: Apache-2.0
Urutau replicates MySQL, Postgres, and Kafka into Apache Iceberg, ClickHouse, and Couchbase reflecting source state — upsert by primary key, first-class UPDATE/DELETE — with the CDC position committed alongside the data it describes, never in a store that could drift from it.
This repository holds the Go engine (coordinator, workers, CLI, operator). The Python SDK/planner lives in its own repository.
Why
For the common case — one sink, no multi-consumer replay — Urutau reads from MySQL, Postgres, or an existing Kafka/Redpanda topic and writes straight to the destination, without standing up any new broker or relay in between.
It also writes natively. Some CDC-to-lakehouse tools hand the actual write off to a JVM sidecar process — a JAR, a gRPC hop, a second runtime to keep alive. Urutau writes in the same Go binary that reads the log: one process, one failure domain, no second runtime in the build.
Recovery follows from the same idea. Nothing durable lives in what can
die — the coordinator and workers are replaceable; state lives only in the
source's own log, the destination table, and the pipeline definition, all
of which survive a total restart. Recovering from a dead cluster means
kubectl apply and reading the committed position back out of the sink,
not replaying a separate checkpoint log.
The engine is closed; the driver seam is open. Sources and sinks are
public Go contracts at the module root — a source or sink is a package
that implements a handful of small interfaces and registers itself, never
touching an internal/ path. See Writing a plugin.
Get started
# pipeline.yaml
pipeline: orders-demo
source:
kind: mysql
uri: mysql://user:pass@localhost:3306/shop
serverId: "1101"
sink:
type: iceberg+rest
uri: http://localhost:8181/api/catalog
warehouse: quickstart_catalog
namespace: bronze
tables:
- source: shop.orders
target: bronze.orders
primaryKey: [id]
partitionBy: [day(created_at)]
writeMode: upsert
urutau run -f pipeline.yaml --metrics-addr :9090
run is the collapsed mode: coordinator and worker in one process against
the sink — no Kubernetes required to try it. The --metrics-addr flag
enables the embedded monitoring dashboard at http://localhost:9090
alongside the Prometheus /metrics endpoint.
→ Quickstart walks through this end to end on your machine — build the binary, stand up a real MySQL + Iceberg locally with Docker Compose, run the pipeline, read the result back through Trino, and watch a live change replicate.
Status
Released. The engine runs end to end — MySQL/Postgres/Kafka into Iceberg, ClickHouse, or Couchbase, single-process or distributed, with the k8s operator — and the commit path has been verified by reading back through Trino rather than trusting a successful write.
v0.2.0 adds background Iceberg table maintenance (compaction, snapshot expiry, orphan cleanup) and an embedded monitoring dashboard with real-time SSE updates.
It is not yet software with production mileage: correctness-critical paths are still being actively hardened, so read Known limitations and roadmap before relying on this for anything you can't afford to lose.
Documentation
Documentation is hosted at maltzsama.github.io/urutau.
| Page | What's in it |
|---|---|
| Quickstart | Run a real pipeline on your machine, step by step |
| Concepts | The mental model — reflect source state, coordinator/worker, snapshot→stream, position-in-sink |
| Deploy on Kubernetes | Submit one CDCPipeline; the operator runs the coordinator and its workers |
| Distributed mode | Coordinator + worker without Kubernetes — the same engine, by hand |
| Monitoring | Metrics, /statusz, logs, what to alert on |
| Dashboard | Embedded monitoring UI with real-time updates |
| Table maintenance | Background compaction, snapshot expiry, orphan cleanup |
| Reliability | Audit trail, checkpoints, supervision, resume |
| Troubleshooting | Symptom → cause → fix, grouped by area |
| CLI reference | All four binaries and every flag |
| Pipeline specification | The pipeline YAML schema, field by field |
| Delivery guarantees | The behavior contract — at-least-once, per-table atomicity, ordering, deletes, poison-batch policy. Read this before depending on any behavior not shown in an example. |
| Sources | MySQL, Postgres, Kafka — what each needs, Kafka's decoder formats |
| Sinks | Iceberg, ClickHouse, Couchbase — commit mechanics, nested-column support, atomicity trade-offs |
| Dashboard API | REST and SSE reference for the embedded dashboard |
| Enrichment | Broadcast reference join — grammar, cold start, examples |
| Known limitations and roadmap | What's genuinely missing today, kept current |
| What's new in v0.2.0 | Release highlights — maintenance, dashboard, metrics |
| Plugin contract | The normative Arrow Flight subprocess plugin contract |
| Writing a plugin | The Arrow Flight subprocess contract (normative), plus the narrower in-process Go .so alternative |
| Architecture | Package boundaries, the dependency diagram, repository map, E2E spike findings |
| Operator | How a CDCPipeline becomes a coordinator StatefulSet plus worker Deployments |
| EncodeKey | Design note: the collapse-stage key encoding |
| State position | Design note: where a committed position lives, and the sink-vs-store arbitration rule |
Companion repository
The Python authoring SDK and planner (the .py pipeline definitions this
engine's operator resolves) live in a separate repository. This repo never
imports Python and never executes user code directly — the planner runs in
an init container, ahead of the coordinator.
Contributing
Issues and PRs are welcome. All code, comments, commit messages, and documentation in this repository are English. See CONTRIBUTING.md for the workflow and CODE_OF_CONDUCT.md for the community expectations. The contributor sign-off policy (DCO vs CLA) is still open — tracked in the roadmap.
Development
make bootstrap # buf, golangci-lint, setup-envtest pinned into ./bin
make envtest-setup # install the operator envtest control plane
make build # bin/urutau, bin/urutau-coordinator, bin/urutau-worker, bin/urutau-operator
make test # go test -race ./... (operator envtest skipped without assets)
make lint # golangci-lint
make proto # buf lint + generate (generated code is committed)
make docs-site # install + serve docs at localhost:3000
make k8s-load # build the image into minikube's docker daemon
make k8s-deploy # apply the operator + CRD (needs cert-manager)
make k8s-load/k8s-deploy are the local Kubernetes loop; see
Deploy on Kubernetes.
No protoc needed — generation uses buf with the protoc-gen-go/
protoc-gen-go-grpc plugins pinned as go tool.
License
This project is licensed under the Apache License 2.0. See the LICENSE file for details.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
v1alpha1
Package v1alpha1 holds the CDCPipeline CR types (apiVersion urutau.io/v1alpha1) and the machine-written status.
|
Package v1alpha1 holds the CDCPipeline CR types (apiVersion urutau.io/v1alpha1) and the machine-written status. |
|
cmd
|
|
|
coordinator
command
|
|
|
operator
command
|
|
|
urutau
command
|
|
|
worker
command
|
|
|
Arrow canonical extension types — registration, not implementation.
|
Arrow canonical extension types — registration, not implementation. |
|
Package dataplane defines the columnar zero-copy data plane seam type for the CDC pipeline.
|
Package dataplane defines the columnar zero-copy data plane seam type for the CDC pipeline. |
|
Package driver is the plugin registry.
|
Package driver is the plugin registry. |
|
internal
|
|
|
builtin
Package builtin blank-imports the built-in source and sink drivers so their init() functions register with the driver registry.
|
Package builtin blank-imports the built-in source and sink drivers so their init() functions register with the driver registry. |
|
coordinator
Package coordinator drives the source side of the split pipeline: it owns the replication reader and the DBLog snapshot, serves the control plane (Session/Assignment) and the Arrow Flight data plane, and streams change batches to the connected workers.
|
Package coordinator drives the source side of the split pipeline: it owns the replication reader and the DBLog snapshot, serves the control plane (Session/Assignment) and the Arrow Flight data plane, and streams change batches to the connected workers. |
|
dashboard
Package dashboard serves the coordinator's embedded monitoring UI and its JSON API.
|
Package dashboard serves the coordinator's embedded monitoring UI and its JSON API. |
|
enrich
Package enrich is the table-level enrichment stage: a columnar broadcast hash join against small reference tables (CR-069).
|
Package enrich is the table-level enrichment stage: a columnar broadcast hash join against small reference tables (CR-069). |
|
errors
Package errors decides whether a source failure is worth retrying.
|
Package errors decides whether a source failure is worth retrying. |
|
eventlog
Package eventlog implements the per-run-id JSONL event log in S3 — one event per lifecycle cycle or commit, never per row.
|
Package eventlog implements the per-run-id JSONL event log in S3 — one event per lifecycle cycle or commit, never per row. |
|
grpctls
Package grpctls builds mutual-TLS gRPC credentials for the coordinator (server) and worker (client) control plane.
|
Package grpctls builds mutual-TLS gRPC credentials for the coordinator (server) and worker (client) control plane. |
|
logging
Package logging builds the process-wide slog logger from CLI flags, so every binary in the tree configures its handler and level the same way.
|
Package logging builds the process-wide slog logger from CLI flags, so every binary in the tree configures its handler and level the same way. |
|
maintenance
Package maintenance schedules the engine's table-maintenance operations.
|
Package maintenance schedules the engine's table-maintenance operations. |
|
observability
Package observability exposes the lean Prometheus metrics (§13.1) and the live status endpoint (§13.4): a small set of series, no per-event logging.
|
Package observability exposes the lean Prometheus metrics (§13.1) and the live status endpoint (§13.4): a small set of series, no per-event logging. |
|
operator
Package operator reconciles CDCPipeline CRs into a coordinator StatefulSet: one coordinator per pipeline, owning the workers it creates.
|
Package operator reconciles CDCPipeline CRs into a coordinator StatefulSet: one coordinator per pipeline, owning the workers it creates. |
|
pipeline
Package pipeline defines the stage abstraction that bridges external plugin processes to the runner's source/sink interfaces.
|
Package pipeline defines the stage abstraction that bridges external plugin processes to the runner's source/sink interfaces. |
|
plugin
Package plugin implements adapters that wrap the Flight client into the standard source.Source and sink.Sink interfaces.
|
Package plugin implements adapters that wrap the Flight client into the standard source.Source and sink.Sink interfaces. |
|
plugin/flightserver
Package flightserver serves the plugin Arrow Flight contract (internal/ plugin/contract) in-process, backed by a real source.Source or sink.Sink.
|
Package flightserver serves the plugin Arrow Flight contract (internal/ plugin/contract) in-process, backed by a real source.Source or sink.Sink. |
|
plugin/flightwrap
Package flightwrap implements driver.PluginWrap: it turns a .so's raw registered source/sink factories into factories that serve the real driver over an in-process Arrow Flight server (internal/plugin/ flightserver) and hand back a Flight client adapter (internal/plugin) — the SAME contract a subprocess plugin speaks.
|
Package flightwrap implements driver.PluginWrap: it turns a .so's raw registered source/sink factories into factories that serve the real driver over an in-process Arrow Flight server (internal/plugin/ flightserver) and hand back a Flight client adapter (internal/plugin) — the SAME contract a subprocess plugin speaks. |
|
plugin/proc
Package proc manages external plugin processes: spawn, readiness, log pumping, and lifecycle.
|
Package proc manages external plugin processes: spawn, readiness, log pumping, and lifecycle. |
|
rowchange
Package change defines the change event that flows from source decoding to sink writing: one row-level operation with its before/after images, primary key, and source position.
|
Package change defines the change event that flows from source decoding to sink writing: one row-level operation with its before/after images, primary key, and source position. |
|
runner
Package runner wires the collapsed process: one binary runs the source reader, the DBLog snapshot, and the worker in a single process (local mode).
|
Package runner wires the collapsed process: one binary runs the source reader, the DBLog snapshot, and the worker in a single process (local mode). |
|
sink/clickhouse
Package clickhouse implements the sink contract on ClickHouse.
|
Package clickhouse implements the sink contract on ClickHouse. |
|
sink/couchbase
Package couchbase implements the sink contract on Couchbase: key-value document writes where upsert-by-key IS the native operation — no equality deletes, no eventual merges.
|
Package couchbase implements the sink contract on Couchbase: key-value document writes where upsert-by-key IS the native operation — no equality deletes, no eventual merges. |
|
sink/iceberg
Package iceberg wraps the Apache Iceberg REST catalog and the write paths exercised by the spike: create/load, append, and the data+equality-delete transaction that mirrors upsert.
|
Package iceberg wraps the Apache Iceberg REST catalog and the write paths exercised by the spike: create/load, append, and the data+equality-delete transaction that mirrors upsert. |
|
snapshot
Package dblog holds the source-agnostic DBLog snapshot orchestrator: chunking by primary key, low/high watermarks, and the caught-up proof that closes each window — never a timer.
|
Package dblog holds the source-agnostic DBLog snapshot orchestrator: chunking by primary key, low/high watermarks, and the caught-up proof that closes each window — never a timer. |
|
source/kafka
Package kafka implements the Kafka source adapter.
|
Package kafka implements the Kafka source adapter. |
|
source/kafka/decoder
Package decoder implements the debezium-json decoder for Kafka CDC messages.
|
Package decoder implements the debezium-json decoder for Kafka CDC messages. |
|
source/mysql
Package source.mysql implements the MySQL replication source on top of go-mysql/canal: a single binlog reader that decodes row events into rowchange.Change, positions them at their transaction GTID, and exposes the synced and master positions for the DBLog watermark logic.
|
Package source.mysql implements the MySQL replication source on top of go-mysql/canal: a single binlog reader that decodes row events into rowchange.Change, positions them at their transaction GTID, and exposes the synced and master positions for the DBLog watermark logic. |
|
source/mysql/charsetgen
command
Command charsetgen turns the .tsv tables in this directory into charset_tables.go, the single-byte decoding tables for the 7 character sets that have no decoder in golang.org/x/text (see issue #113).
|
Command charsetgen turns the .tsv tables in this directory into charset_tables.go, the single-byte decoding tables for the 7 character sets that have no decoder in golang.org/x/text (see issue #113). |
|
source/postgres
Package source.postgres implements the PostgreSQL replication source: a single logical-decoding reader (pgoutput) that decodes row changes into rowchange.Change, positions them at their commit LSN, and exposes the synced and confirmed positions for the DBLog watermark logic.
|
Package source.postgres implements the PostgreSQL replication source: a single logical-decoding reader (pgoutput) that decodes row changes into rowchange.Change, positions them at their commit LSN, and exposes the synced and confirmed positions for the DBLog watermark logic. |
|
sourcepull
Package sourcepull adapts a push-based change channel into the pull-based source.Reader.Next surface, encoding buffered changes into wire batches.
|
Package sourcepull adapts a push-based change channel into the pull-based source.Reader.Next surface, encoding buffered changes into wire batches. |
|
supervisor
Package supervisor manages external plugin subprocesses with restart backoff, circuit breaker protection, and graceful shutdown.
|
Package supervisor manages external plugin subprocesses with restart backoff, circuit breaker protection, and graceful shutdown. |
|
transport
Flight data-plane codec: change batches travel as Arrow IPC records with a BatchMeta proto in the FlightData app_metadata.
|
Flight data-plane codec: change batches travel as Arrow IPC records with a BatchMeta proto in the FlightData app_metadata. |
|
version
Package version exposes build metadata injected via ldflags (see Makefile).
|
Package version exposes build metadata injected via ldflags (see Makefile). |
|
worker
Package worker implements the worker plane: per-table batchers that accumulate changes and flush them collapsed to a committer, with commits strictly serialized per table.
|
Package worker implements the worker plane: per-table batchers that accumulate changes and flush them collapsed to a committer, with commits strictly serialized per table. |
|
Package position defines the replication position contract.
|
Package position defines the replication position contract. |
|
Package sink defines the destination catalog contract.
|
Package sink defines the destination catalog contract. |
|
Package source defines the replication source contract.
|
Package source defines the replication source contract. |
|
Package spec defines the resolvedSpec: the contract between the planner, the admission webhook, and coordinator boot.
|
Package spec defines the resolvedSpec: the contract between the planner, the admission webhook, and coordinator boot. |
|
test
|
|
|
e2e
Package e2e holds end-to-end tests against the docker-compose stack (MySQL + Postgres + Polaris catalog + Trino).
|
Package e2e holds end-to-end tests against the docker-compose stack (MySQL + Postgres + Polaris catalog + Trino). |
|
plugin
Package plugin is a proof that the door is open: it implements a source and a sink entirely outside internal/, importing only the public contracts (source, sink, driver, core, change, position, spec) and registering them with the driver registry.
|
Package plugin is a proof that the door is open: it implements a source and a sink entirely outside internal/, importing only the public contracts (source, sink, driver, core, change, position, spec) and registering them with the driver registry. |
|
plugin/standalone
command
Package main is an example Go plugin that registers a source and sink via the driver registry.
|
Package main is an example Go plugin that registers a source and sink via the driver registry. |