outboxer

package module
v0.6.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 14, 2026 License: MIT Imports: 13 Imported by: 0

README

outboxer

CI Lint CodeQL Go Reference Go Version Release License

A transactional outbox for Postgres and Go. Write events to a table inside the same transaction as your business data; a relay claims them and hands each to your publish function, at-least-once, safely across replicas. Depends on pgx, and on gokern/panics, which has none of its own.

// Producer: built once at wiring time, then the row and the business data
// commit together because they go through the same handle.
producer, err := outboxer.NewProducer()

tx, _ := pool.Begin(ctx)
createUser(ctx, tx, user)
producer.Insert(ctx, tx, outboxer.Message{
    Topic:   "user.created",
    Payload: data,
})
tx.Commit(ctx)

// Relay, in its own binary or its own goroutine.
relay, err := outboxer.NewRelay(pool, publish)
err = relay.Run(ctx)

Wiring you can copy is in example_test.go and on pkg.go.dev. The rest of this file is the contract and the behaviour worth knowing before wiring it in.

Requirements

Requires Go 1.26+ and PostgreSQL 14 through 18.

That Postgres range is what CI runs the whole suite against on every commit, not what the SQL looks like it ought to support. Nothing here inspects the server version, so a range no job executes would be a promise nobody could discover was broken.

The two sides

Two constructors, one per direction, and each validates everything it was given before it returns. Nothing configurable is left to be discovered on the path that moves rows. A third, NewSampler, reads the table's state rather than moving anything through it; see Observability.

NewProducer builds the write side. Producer.Insert takes the Postgres handle per call, so the transaction-manager seam is the call site itself: pass a transaction and the write is atomic with your data, pass the pool and it commits on its own. Both are legitimate; the producer holds no connection and never decides for you.

NewRelay builds the read side over a pool, and Run drains it until its context is cancelled. The two sides do not have to share a process, and a producer that only writes never opens a LISTEN connection.

All three name their table with WithTable, which defaults to outbox. That name is the one fact they have to agree on, so it is also the only option every constructor accepts. Every other setting belongs to the relay, and the compiler refuses it on NewProducer and NewSampler alike.

Delivery semantics

At-least-once. A publish can succeed and the mark that follows can fail, and the row is then redelivered. Consumers must be idempotent: use Delivery.ID as the broker's dedupe key.

No ordering. Rows are claimed with FOR UPDATE SKIP LOCKED, so concurrent workers and replicas are safe but unordered.

Two different paths produce a duplicate, and it is worth knowing which is which. A mark that fails after a successful publish is fatal, so the process exits and the row falls due again at lease expiry. That gap is bounded by Lease, and it is the one a broker's dedupe window has to cover. A publish whose acknowledgement is lost after the server stored it looks like a plain failure from here, so it goes through RetryFunc instead, and that gap is bounded by your own policy. The second path is why RetryFunc carries no cap: capping every caller's policy to narrow a window idempotency already covers is the wrong trade.

Schema

Your application owns the table and its migrations; this package never runs one. migration/0001_create_outbox.up.sql is the reference DDL, meant to be copied into your own history. It is deliberately plain: no IF NOT EXISTS guards, because a guard turns a mis-ordered migration into a silent success, and no lock strategy either way, because that policy is yours.

One thing that policy has to account for here. A later migration that holds ACCESS EXCLUSIVE on this table for more than five seconds while a relay is running produces a duplicate delivery. The relay publishes a row and then marks it; the mark blocks on the lock and gives up at the internal five-second bound, and the row, already at the broker, comes back at lease expiry and goes out again. ALTER TABLE, a non-concurrent CREATE INDEX, VACUUM FULL, REINDEX and CLUSTER all take that lock. It is a duplicate and never a loss, so an idempotent consumer absorbs it; to avoid it anyway, stop the relays for the migration or use the CONCURRENTLY forms.

A statement_timeout on the relay's role is the same trap by another route. Set below that five-second bound it turns any mark the server holds up into the same duplicate, so keep it comfortably above, or leave it unset on that role. Set below what a claim over a large backlog takes, it stops the relay outright with SQLSTATE 57014, which at least says so.

Column Type Meaning
id BIGINT identity row identity; the broker dedupe key
topic TEXT routing, carried verbatim
payload BYTEA opaque bytes, carried verbatim
headers JSONB optional metadata; a flat object of strings
attempts INT incremented by the claim
created_at TIMESTAMPTZ row age, on the database clock
ready_at TIMESTAMPTZ when the row is next due
published_at TIMESTAMPTZ NULL means pending

Extra columns (tenant ids, partition keys) are fine and never touched.

Three things beyond the table itself, in order of how much you need them:

  • outbox_due_idx, a partial index on (ready_at, id) WHERE published_at IS NULL. Without it every claim is a sequential scan. The ordering leads with ready_at rather than id, which lets the claim seek straight past deferred rows instead of walking them and throwing each one away. Deferral is ordinary traffic here: a broker outage defers the whole backlog at once, and Message.Delay defers by design.

    Adding INCLUDE (created_at, attempts) to it is tempting and mostly is not worth it. At a moderate backlog beside many published rows it turns the bitmap heap scan into an index-only scan and saves a fraction of a millisecond, which is where the read was already cheap. In the case that hurts it does nothing: at a large backlog the planner takes the sequential scan anyway, and forcing the index-only scan instead measured half again slower. An index-only scan wants an all-visible heap, and a live outbox never has one, because every claim updates the pending rows it would otherwise skip reading. Weigh that against a heavier write path.

  • The AFTER INSERT trigger calling pg_notify on the channel. Without it the relay polls, which is correct everywhere and slower only where latency is measured in tens of milliseconds. The channel name is the table name.

  • outbox_published_idx, only if you turn age retention on.

  • outbox_published_order_idx, for count retention. Apply migration/0002_index_published_order.up.sql after the reference schema. This partial B-tree orders published rows by (published_at, id) and also serves age scans. The additive migration leaves the original index in place. A normal index build blocks writers; schedule its application before relays start or in an adopter-controlled maintenance window.

Keep the two clocks close

ready_at and created_at are stamped by the database, while Message.Delay and every timeout you configure are measured by your process. The relay compares the two, with a floor so that skew can never turn a wait into a spin. But a client running ahead of the server by more than the shortest Delay you use will poll harder than it needs to for as long as the skew lasts. Ordinary NTP is enough; a suspended VM or an unsynchronised container is not.

ready_at carries one meaning: when the row is next due. The claim sets it to now + lease, a failed publish sets it to now + RetryFunc(...), and a producer sets it forward through Message.Delay. A scheduled event therefore falls out of the same column that implements the lease.

attempts is incremented by the claim, in the same statement, and never by the failure path. Counting only publish failures is blind to the failure that matters most: a row that kills the process before the failure path runs is reclaimed at lease expiry with its counter unchanged, so a poison row loops forever at zero. Claiming is attempting. Delivery.Attempt is therefore honest, and PublishFunc may refuse a row whose count has gone absurd. That policy belongs to you, not here.

Headers must be a flat JSON object of strings

That is the only shape map[string]string can hold. The write side always produces one: nil becomes {} and never NULL, and a key or value that is not valid UTF-8 is refused instead of letting json.Marshal quietly replace the bytes with U+FFFD. A row written by hand that carries something else ({"n":1}, an array, a nested object) is skipped and reported through Observer.Warned, neither delivered nor fatal: the rows around it go out, and it keeps its lease, so it comes back and is reported again instead of disappearing. NULL headers arrive as nil.

Keeping the schema correct is yours, and there is no Check method

This package never inspects the table. A missing table or column is the one thing it hears about, because Postgres raises it on the first statement that touches the table. It arrives as ErrSchemaMismatch, and nothing else does.

Everything past that is silent and stays silent. Drop outbox_due_idx and every claim becomes a sequential scan that grows with the table. Drop the trigger, rename the channel, or let the LISTEN session go through a transaction-pooling pooler, and the relay stays correct but never gets a push wake-up and finds work only on its poll tick.

Those are real failures, and they belong to the migration review and to whatever already watches your database. A library cannot see them without guessing. The only answer available to it is a string match against catalogue internals: indpred containing published_at IS NULL, prosrc containing pg_notify(...). That calls any schema meeting the contract by other means broken, and a wrong answer given confidently is worse than none.

Configuration

Every option returns an error. Nothing is clamped and nothing is ignored, so what you declared and what runs are the same thing. That is also why there is no Config type to read values back from: you already hold them.

Option Default Bounds
WithTable outbox the table every constructor names; the only one NewProducer and NewSampler accept
WithMaxConcurrency 16 publishes in flight at once; a ceiling, not a steady count
WithLease 60s how long a claimed row stays undue
WithPublishTimeout 5s one PublishFunc call
WithPollInterval 10s how long the wait blocks without a notification
WithRetention 0 (off) removes published rows older than this age
WithMaxPublishedRows 0 (off) keeps at most this many newest published rows after an unobstructed sweep; pending rows are never deleted
WithPruneInterval 1h, or the retention window when shorter how often cleanup sweeps; needs either retention policy, and refused if longer than an enabled age window
WithRetry one lease when a failed row becomes due again
WithDialer none (poll-only) how to open the dedicated LISTEN session
WithObserver none everything the relay reports

Age and count retention work independently. When both are enabled, a row is eligible for deletion by either policy: count cleanup can remove rows before that age. Each sweep first removes old rows, then fixes one count cutoff and removes excess published rows in batches of 1000. Equal publication timestamps are ordered by ID. Locked eligible rows are revisited on a later sweep.

The count limit is soft: publication can exceed it between sweeps or while cleanup is blocked. It does not cap unpublished backlog, payload bytes or filesystem usage. Each count cutoff lookup scans up to maxRows + 1 visible published index entries; it is not constant-time. Choose the interval against measured publication and cleanup rates. PostgreSQL vacuum reclaims deleted space for reuse; deletion alone does not shrink database files.

No default is exported. A caller who needs to compute with an interval, a liveness window say, should declare that interval and compute from what they declared. A number inherited from this package changes silently when this package changes its mind.

NewRelay refuses a lease that does not exceed the publish timeout plus an internal 5s mark bound, because a shorter one lets a second replica reclaim a row the first is still publishing. The error names all three durations, so a legal lease can be computed from it without knowing the constant.

A PollInterval at or above Lease is reported, not refused. A longer poll delays the reclaim of a crashed replica's row, which is slower but not wrong.

Size the pool to MaxConcurrency plus your own headroom. The listener holds one extra connection that does not come from the pool.

Behind a transaction-pooling pooler

PgBouncer and friends: the claim, mark, defer, prune and insert paths all work, whether pgx prepares its statements or runs in exec mode where no parameter's column type is ever resolved. That is verified against a real PgBouncer in transaction mode and not only simulated; mise run test-pgbouncer runs it against yours. If your pooler predates PgBouncer 1.21 it cannot route prepared statements at all, and pgx has to be put in exec mode for reasons that have nothing to do with this package.

The LISTEN session must not go through one, and the failure is silent. LISTEN is accepted, so the dial succeeds and Observer.ListenerChanged never fires. The pooler then hands the server connection back and drops every notification that arrives while no client holds it. What is left is a relay that is correct, polling, and indistinguishable from one that is working. That behaviour was measured against a real pooler, and it is why WithDialer exists instead of a DSN option: derive that connection from the pool's own DSN and it is right until the day somebody puts a pooler in front of it.

Run owns the connection its DialFunc returns. It closes that connection on exit and re-dials after a break, so the function must return a fresh connection every time, never a shared handle.

What stops the relay

  • A publish failure is never fatal. The row is deferred by RetryFunc and reported through Observer.Published. The deferral is written before the observer runs, so a slow observer cannot sit between a failure and the recording of its consequence.
  • A PublishFunc that panics is a publish failure, not a crash. The panic is recovered and reaches you through both RetryFunc and Observer.Published: panics.Is(err) reports it, and panics.As(err) gives you the value it was raised with and the frames it came from. It is contained for the same reason skipping a row with undecodable headers does not stop the relay: one message must not stop every other one. Uncontained, it would abandon every delivery in flight beside it, including rows already published and about to be marked, each of which returns at lease expiry as a duplicate. The same row would then meet the same bug after every restart. Match on it if a row that panics deserves a policy of its own; it will very likely panic again. The test is a broad one: it holds for a panic contained anywhere, including one your own code recovered and reported as an ordinary error, which the relay cannot tell apart from a panic of its own making.
  • Every other function you supply is contained too. A RetryFunc, a DialFunc and every Observer field run on the relay's goroutines, and a panic in one of those is an advisory rather than the delivery's own error: it reaches Observer.Warned, and a DialFunc's reaches Observer.ListenerChanged.
  • A storage failure on the claim is fatal. Run returns it. Nothing was claimed and nothing was published, so the cost is delay and not correctness. The relay stops anyway, because how long an unreachable database is worth waiting for is your policy and not this package's. A dropped connection is the ordinary cause and is not exotic: a failover, a pooler restart and an administrator's pg_terminate_backend all produce one, and it clears by itself once the pool has cycled out its dead connections. See Supervision below, since Run is single-use and carrying on is not a matter of calling it again.
  • A storage failure on the mark is fatal. The row was published and the database does not know, so the duplicate is already guaranteed. The only thing left to control is that it is one duplicate and not a stream. The relay is stopped before Observer.Published is told, because every row published during that call would be another the database does not know about.
  • A claim that failed because the context was cancelled is shutdown, not failure. Without that carve-out every clean deploy reports itself as a crash.
  • Housekeeping is never fatal. A failed retention sweep is reported through Observer.Pruned and nothing else.
  • A wedged relay stops itself. A PublishFunc that ignores its context never returns and nothing in Go can make it, so once every slot is occupied for twice as long as one delivery may legally take, Run returns ErrPublishStalled. Restarting the process is the only recovery, and only you can do that: the goroutines holding those slots outlive Run.
  • A shutdown that gave up waiting is returned, not whispered. Cancellation waits for the in-flight publishes on a context detached from it, so a row already claimed finishes its publish and its mark, and that wait is itself bounded so a publisher ignoring its deadline cannot hang your task group. Run returns nil only when those publishes actually finished, and ErrShutdownIncomplete when the wait expired instead. They are still running detached and will mark themselves if the process lives, so exiting on that error is what turns it into duplicates.

There is no circuit breaker. How many consecutive failures mean "stop" is policy: you see every failure through Observer.Published and cancel the context you passed to Run.

Supervision

Run is single-use, and a second call returns ErrAlreadyRun. Keeping the outbox draining across a failover therefore means building another Relay, not calling Run again. ExampleRelay_Run is that loop. Two errors are the exception and should end the process instead: after ErrPublishStalled and ErrShutdownIncomplete the publisher goroutines are still running and still holding pool connections, so rebuilding in a loop leaks another set of them on every pass.

Observability

The package has no logger and no metrics of its own. Two things report, and they fail in opposite directions: Observer runs inside the relay and stops when it does, while Sampler reads the table and keeps answering when no relay is running at all.

Observer

Every field is optional.

Field Fires
Published after every publish attempt, successful or not
Woke every time the relay stops waiting, the LISTEN session coming up included; the idle heartbeat
ListenerChanged once per LISTEN connection transition; nil means recovered. A first successful dial is not a transition and is not reported, so silence is not "down"
Pruned after each retention sweep, with the row count
Warned non-fatal advisories: a poll interval at or above the lease, a clamped negative retry, a row whose headers could not be decoded, a deferral the database refused, a failed read of when the next row falls due, a callback of yours that panicked

Published is one field on purpose. Splitting success from failure would lose the caller who answers both with the same signal, a liveness probe for instance, since a relay that cannot reach the broker is failing but not wedged.

All of them run on the relay's own goroutines. Hand work off instead of blocking in one.

Sampler, and the same reading in SQL

Four numbers about the table, read on demand:

sampler, err := outboxer.NewSampler(pool, outboxer.WithTable("outbox"))
stats, err := sampler.Stats(ctx) // Pending, Due, OldestAge, PeakAttempts

The same four numbers in SQL, for the monitoring that owns the database rather than the process — a postgres_exporter custom query, a Datadog check, a psql session during an incident:

SELECT count(*) AS pending,
       count(*) FILTER (WHERE ready_at <= statement_timestamp()) AS due,
       coalesce(extract(epoch FROM statement_timestamp() - min(created_at)), 0) AS oldest_age_seconds,
       coalesce(max(attempts), 0) AS peak_attempts
  FROM outbox
 WHERE published_at IS NULL;

statement_timestamp() and not now(), which is transaction-start time. Run inside a transaction your exporter happened to open, now() misses every row that fell due since it began, and reports a negative age for a row written since — a claimable row reads as due = 0, and the age comes back as minus however long the transaction has been open.

Count pending and due separately or the numbers mislead. pending includes rows deferred by Message.Delay and rows under lease right now, so it tracks load; due is what should have gone out and did not.

No single field is the alert. They discriminate failures together:

Failure Due OldestAge PeakAttempts Pending
relay dead or wedged grows grows frozen grows
relay too small for the traffic grows grows slowly low grows
broker down ≈0, oscillating grows climbing grows
one poison row ≈0 grows climbing on one row flat

Due reads near zero during a broker outage because the relay keeps claiming: each failing row is due for an instant, then deferred by RetryFunc for a whole lease. PeakAttempts frozen against climbing is what separates a dead relay from a dead broker, because the claim counts the attempt as it takes the lease.

OldestAge has one caveat by construction: a row scheduled far ahead with Message.Delay counts from when it was written, so a service that schedules days out carries a permanently large age and should alert on Due and PeakAttempts instead. Excluding those rows is impossible — deferred, leased and retried rows are indistinguishable in the schema.

Scrape on the order of tens of seconds, never per request. While the backlog is small Stats seeks the partial index and never reads a published row, so it costs microseconds however many published rows have accumulated. Once the pending rows are a large fraction of the table the planner switches to a sequential scan, which reads the published rows too, and the read grows by two to three orders of magnitude. The cost stops tracking the backlog and starts tracking the table, precisely when the backlog is worst.

Metrics

Every metric worth having comes from Observer and Sampler, and the package ships none of them, because names and labels are your organisation's conventions. outboxprom picks one set and exports it, in a module of its own so client_golang never reaches this go.mod:

metrics, err := outboxprom.NewMetrics()
relay, err := outboxer.NewRelay(pool, publish, outboxer.WithObserver(metrics.Observer()))

sampler, err := outboxer.NewSampler(pool)
backlog, err := outboxprom.NewBacklog(sampler)

prometheus.MustRegister(metrics, backlog)

Five options cover what the defaults do not: WithNamespace for the metric prefix, WithConstLabels for a shard or tenant, WithoutTopicLabel when topics are unbounded, WithLagBuckets for your own SLO, and WithSampleTimeout to bound the scrape's query.

Wiring your own instead is a table lookup:

Metric Where it comes from
publishes by topic and outcome Observer.Published, err == nil
publish duration wrap your own PublishFunc; the package is not involved
insert-to-publish lag time.Since(delivery.CreatedAt) in Published, on success only
attempt distribution delivery.Attempt
listener transitions Observer.ListenerChanged as a counter, never a gauge; a level built from it reads "down" on a healthy relay
retention volume and failures Observer.Pruned
advisories by class Observer.Warned with errors.Is — never by message text
backlog, age, worst attempt count Sampler.Stats

Two things to get right whichever way you export them:

  • Cardinality. Topic is a label only if your topics are a bounded set. Never label by Delivery.ID, a key, or an error's text.
  • Replicas. Every replica reports the same table, so aggregate the backlog with max by, never sum, or it gets multiplied by the replica count.

What this package deliberately does not do

Each of these is withheld on purpose, not missing.

It does not Why, and whose job it is
Open a transaction Execer is one method, Exec: there is no Begin to call, so the write side cannot. The relay holds a *pgxpool.Pool and is held to it by discipline instead. Every statement it issues is a single statement, which is how the claim takes a lease and counts an attempt indivisibly. Atomicity is whatever the handle you pass to Producer.Insert provides, and rollback belongs to whoever opened the transaction.
Open a connection The pool and the LISTEN session are yours: your DSN, your TLS, your custom types, your AfterConnect. WithDialer takes a function rather than a connection because that session is re-dialled after every drop.
Run a migration, or inspect the schema The table, its indexes, the NOTIFY trigger and the channel name are yours to create and keep correct. A missing table or column comes back as ErrSchemaMismatch on the first statement; a missing index or trigger is silent, and always will be. See Schema.
Keep one table name across your two binaries WithTable defaults to outbox and both sides have to be given the same value. Declare it once in your own configuration and pass that to each. Where both tables exist a mismatch is invisible from in here, the same way a publisher and a subscriber on different subjects are invisible to a broker.
Bound its own queries Only your context deadline bounds them, and the mark and the deferral run detached from cancellation on purpose. A statement_timeout in your AfterConnect bounds every query in your service, not only these.
Interpret your payload or topic Bytes and a string, carried verbatim. Serialization, schema registry and routing conventions are yours.
Guarantee order, or exactly-once Replicas are safe but unordered, and delivery is at-least-once; see Delivery semantics.
Give up on a poison row It hands you Delivery.Attempt and never picks the threshold. Refusing a row whose count has gone absurd is your policy, inside PublishFunc.
Log, or break a circuit Every outcome reaches you through Observer; stopping is cancelling the context you passed to Run.
Export metrics from this module This go.mod keeps its one external dependency whatever you export them with. outboxprom does the exporting, as a module of its own: importing it is opt-in, and not importing it costs you nothing.
Scrape itself Sampler has no ticker and starts no goroutine. It reads when you call it, so the cadence, the timeout and the registry belong to your scraper.

Development

Needs a Postgres the test suite may own outright, since it drops and recreates its tables on every run.

mise run db               # a throwaway Postgres on port 15433
mise run test             # go test -race -shuffle=on; root needs the database,
                          # the outboxprom pass does not
mise run lint
mise run check            # lint, tidy, test, vuln

mise tasks lists everything. mise run db honours DB_IMAGE and DB_PORT from the environment, which is how one leg of the CI matrix is reproduced locally: DB_IMAGE=postgres:14-alpine DB_PORT=15444 mise run db.

outboxprom is a module of its own, so nothing at the root reaches it: go test ./..., go mod tidy and golangci-lint run all stop at the module boundary, and CI runs each of them a second time inside that directory. It shares this .golangci.yaml, which golangci-lint finds by searching upwards; the few rules that differ for it are marked by path there.

It depends on the root through a replace onto the working tree, so a change here is compiled against the collectors that use it. Go ignores a dependency's own replace, though, so what an adopter compiles against is the require beside it — and the replace is what hides a wrong version there until somebody imports the module. That require is a floor rather than a pin: it names the oldest root carrying what the module uses, and an adopter who asks for a newer one gets the newer one, so it does not move with every release here.

Nothing in this repository ever compiles the pair an adopter gets, and CI cannot: while a change to both sides is unreleased, that pair is expected not to build. mise run release-check compiles it in a copy with the replace dropped. Run it before tagging the module.

Module versions are tags of the form outboxprom/vX.Y.Z. The directory prefix is not decoration — without it Go does not find the tag — and v*.*.* in release.yml does not match it, so module versions get no GitHub release. The Releases page tracks the library.

The tests run shuffled and in parallel, here and in CI. Every test creates its own table, named after itself, and the reference DDL names the NOTIFY channel after the table, so one test's inserts can neither be seen by nor wake another's relay. They do still share one database, which is why the order is shuffled: a test that leaves a trigger or a function behind changes what the next one measures, and a fixed order hides that until somebody adds a test.

The tests fail rather than skip without a database. That is deliberate: a suite that can go green without one reports coverage it does not have, and the first thing to go unexecuted is the replica safety that distinguishes this package from the alternatives, the one property no unit test can stand in for.

The suite has two harnesses, because one shape cannot produce both properties it needs. The temp-table one copies the migrated table LIKE ... INCLUDING ALL on a single pinned connection: perfect isolation, and structurally incapable of row contention. The real-table one creates a uniquely named table on a multi-connection pool, where contention between two relays is the measurement.

License

MIT.

Documentation

Overview

Package outboxer is a transactional outbox for Postgres.

A producer appends messages to an outbox table inside the transaction that produced them, so the fact and the intent to publish it commit together or not at all. A relay claims those rows, hands each to a publish function the caller supplies, and marks it delivered. The two sides are independent: a producer that only writes never runs a relay, and neither has to know what broker the other end talks to.

The two sides

NewProducer builds the write side. Producer.Insert takes the Postgres handle per call: pass a transaction-bound handle and the write is atomic with the business data, pass the pool and the row commits on its own. Both are legitimate, and the call site decides.

NewRelay builds the read side over a pool, and Relay.Run drains it until its context is cancelled. Delivery is at-least-once and there is no ordering guarantee: rows are claimed with FOR UPDATE SKIP LOCKED, so concurrent workers and replicas are safe but unordered.

NewSampler builds a third thing that is neither: a reading of the table itself. Sampler.Stats reports how much is undelivered, how much is overdue, how old the oldest unpublished fact is and the worst attempt count among them. It is separate from the relay because that reading is wanted precisely when no relay is running: a crashed process reports nothing.

All three name their table through WithTable, which defaults to "outbox". That name is the one fact they have to agree on, so it is also the one option every constructor accepts. Every other setting belongs to the relay, and the compiler refuses it on NewProducer and NewSampler alike.

Panics in caller code

Every function the caller supplies runs on a relay goroutine: PublishFunc, RetryFunc, DialFunc and every field of Observer. A panic in one of them is contained and never fatal, because a panic that reached the runtime would abandon every delivery in flight beside it, rows already published and about to be marked included, each of which comes back at lease expiry as a duplicate.

Where the contained panic arrives says which function raised it. A PublishFunc panic is the outcome of the delivery, so it becomes that publish's error and travels to RetryFunc and Observer.Published, and the row is deferred like any other failed publish. Every other panic is an advisory alongside the work and reaches Observer.Warned, except a DialFunc's, which reaches Observer.ListenerChanged, since from the listener's side it is a dial that produced no connection. A panic in Warned itself arrives nowhere: the reporting channel cannot report its own failure. A panicking RetryFunc also forfeits its say, and the row is deferred by one lease, which is what it would have waited had the process died.

Recovery goes through github.com/gokern/panics: panics.Is(err) reports that an error carries a panic, and panics.As(err) reaches the value it was raised with and the frames it came from. The test is a broad one: it holds for a panic contained anywhere, including one a PublishFunc recovered itself and reported as an ordinary error. The relay cannot tell those apart, and does not try.

A row that panics will very likely panic again, so a policy of its own belongs in RetryFunc.

What this package does not own

It never opens a transaction. Every statement it issues is a single statement, so atomicity is whatever the handle passed to Producer.Insert provides, and rollback belongs to whoever opened the transaction.

On the write side the type system enforces that: Execer has one method, and no Begin to call. On the read side it is discipline, not type. The relay holds a *pgxpool.Pool, which has Begin and everything else. The discipline holds because a relay operation needing two statements would need a transaction, and the shape of the package would change with it.

It never opens a connection. LISTEN/NOTIFY is opt-in through WithDialer, which takes a function called when the relay needs its dedicated session. The application keeps ownership of DSN, TLS, custom types and any AfterConnect hook, which is also where a statement_timeout belongs. Without a dialer the relay polls, and polling is correct everywhere.

It never runs a migration and never inspects the schema. The table, its indexes and its NOTIFY trigger are the application's to create and to keep correct; migration/0001_create_outbox.up.sql is the reference DDL and the whole of the contract. Postgres reports a missing table or column on the first statement that touches it, immediately and fatally, and this package only translates those SQLSTATEs into ErrSchemaMismatch.

What Postgres never reports, it never reports here either. A dropped partial index turns every claim into a sequential scan. A missing or misdirected NOTIFY trigger, or a LISTEN session opened through a transaction-pooling pooler, leaves the relay correct but poll-only. Both failures are silent. Catching them belongs to the migration review and to the monitoring that already owns the database, not to a library that would have to guess at the catalogue to do it.

It has no circuit breaker, no logger and no metrics. Every outcome the relay produces reaches the caller through Observer, and the state that outlives the relay through Sampler; stopping is the caller cancelling the context it passed to Relay.Run. Turning either into metrics is the caller's, since metric names and label sets are an organisation's conventions and not a library's. The outboxprom module beside this one picks a set of them and exports it, so adopting them is an import rather than a transcription.

Index

Examples

Constants

This section is empty.

Variables

View Source
var (
	// ErrInvalidConfig reports something the caller declared that this package
	// refuses rather than clamping or ignoring, so a declared value and the
	// effective one can never silently differ.
	//
	// Every option returns it, and so do NewProducer and NewRelay. It also
	// reaches the caller on three paths that are not settings but are the same
	// mistake, a value nobody configured: Producer.Insert on a zero-value
	// producer or a nil handle, Run on a zero-value relay, and Run again when a
	// DialFunc returns a nil connection and a nil error. Check for it on the
	// insert path too. Read as "storage failure" and retried, a nil handle
	// retries forever.
	ErrInvalidConfig = errors.New("invalid config")

	// ErrSchemaMismatch reports a table that does not match the contract: a
	// missing table or a missing column, translated from the SQLSTATE Postgres
	// raised on the first statement that touched it. This package never
	// inspects the schema itself, so this is the only way it can be reported.
	ErrSchemaMismatch = errors.New("schema mismatch")

	// ErrInvalidMessage reports a message this package refuses to write: an
	// empty topic, a nil payload, a negative delay, or a topic or header that
	// is not valid UTF-8 or carries a NUL byte. The reason is spelled out in
	// the message text.
	//
	// It is the distinction the write path turns on, and the reason it is
	// exported. A refused message is refused permanently: the same call will
	// fail the same way forever. A storage failure is transient, and there the
	// caller's transaction should be retried. Without a sentinel the two look
	// alike, so a caller with a retry loop around its transaction either
	// retries a poisoned message forever or gives up on a blip.
	//
	// Nothing was written when this is returned: a batch is rejected whole.
	ErrInvalidMessage = errors.New("invalid message")

	// ErrAlreadyRun reports a second call to Run on the same Relay. A Relay is
	// single-use, so a supervisor that restarts Run on error would otherwise
	// spin here forever without being able to tell why.
	ErrAlreadyRun = errors.New("relay has already been run")

	// ErrPublishStalled reports every publish slot occupied for twice as long
	// as one delivery may legally take, and is returned by Run.
	//
	// A PublishFunc that respects its context returns by the publish timeout,
	// and the slot frees once the mark that follows is done. Twice that bound
	// with nothing freed means at least one publish is not coming back. Go
	// cannot make it return, so the relay publishes nothing for the rest of the
	// process. Stopping lets a supervisor restart it, and only a restart
	// recovers.
	//
	// The goroutines it was waiting on are still running when Run returns.
	// There is no way to reclaim them, and the doc for Run says so.
	ErrPublishStalled = errors.New("no publish slot freed; a PublishFunc is not returning")

	// ErrShutdownIncomplete reports a Run that stopped waiting for publishes
	// still in flight, and is returned by Run.
	//
	// Run otherwise returns only after every in-flight publish has finished, so
	// a caller may close its pool and exit. On this path they have not: they
	// keep running detached, and a process that exits now leaves rows published
	// but unmarked, each of which returns at lease expiry as a duplicate.
	ErrShutdownIncomplete = errors.New("gave up waiting for in-flight publishes")

	// ErrRetryNegative reports a RetryFunc that returned a negative delay,
	// clamped to zero. It arrives through Observer.Warned. Unclamped it would
	// leave the row due in the past forever, so the clamp is not optional. But
	// it is the caller's own policy misbehaving, and they should hear about it.
	ErrRetryNegative = errors.New("retry policy returned a negative delay; clamped to 0")

	// ErrHeadersNotStrings reports a headers column this package cannot
	// represent, and arrives through Observer.Warned. Only a hand-written row
	// produces one: the write side always stores a flat object of strings.
	//
	// Skipping the row is not fatal to the relay. It keeps its lease and comes
	// back, so the advisory repeats until somebody fixes the row.
	ErrHeadersNotStrings = errors.New("headers are not a JSON object of strings")
)

None of the sentinel texts names the package: every path out of it wraps them with context that carries the one "outboxer: " prefix, so a prefix in the sentinel would render twice.

Functions

This section is empty.

Types

type Delivery

type Delivery struct {
	// ID is the row's identity. Use it as the broker's dedupe key. Delivery is
	// at-least-once, so a consumer that is not idempotent will eventually see
	// the same message twice.
	ID int64

	// Attempt is which attempt this delivery is: a freshly claimed row arrives
	// with Attempt == 1. The number is written by the claim, so a row that
	// killed the process mid-delivery still counted and cannot loop forever at
	// zero.
	//
	// It is here so PublishFunc can refuse a row whose count has gone absurd.
	// That policy belongs to the caller; this package never gives up on a row.
	Attempt int

	// Topic is Message.Topic as written.
	Topic string

	// Headers is Message.Headers as written. It is nil when the row's headers
	// column is NULL, which only a hand-written insert produces.
	Headers map[string]string

	// Payload is Message.Payload as written.
	Payload []byte

	// CreatedAt is when the row was written, on the database clock. It is the
	// honest age of the fact: a producer-stamped header can be absent, skewed,
	// or written by a process whose clock disagrees with Postgres.
	CreatedAt time.Time
}

Delivery is a row claimed for publication.

type DialFunc

type DialFunc func(ctx context.Context) (*pgx.Conn, error)

DialFunc opens the dedicated session the relay LISTENs on. It is called when the relay needs that connection and again after the connection breaks, so it must return a fresh one every time — never a shared handle. Run owns what it returns and closes it.

It must honour ctx. The context is how the relay ends a dial at shutdown, and a DialFunc that ignores it holds up Run's return the same way a PublishFunc that ignores its deadline would.

It exists instead of a DSN option because the correct connection is the one the application knows how to open: its TLS, its custom types, its AfterConnect hook, and above all a session that does not run through a transaction-pooling pooler. Such a pooler registers LISTEN on a server connection it hands straight back to the pool, and then delivers nothing for the life of the process.

type Execer added in v0.3.0

type Execer interface {
	Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
}

Execer is a Postgres handle this package writes through: a *pgxpool.Pool, a *pgxpool.Conn or a pgx.Tx all satisfy it.

The single method is the guarantee: with no Begin to call, this package cannot open a transaction, and every statement it issues is a single statement. It is also the whole burden on anyone wrapping a handle (a query logger, a tenant router, a fake), who would otherwise have to produce a pgx.Rows for a method that is never called.

Atomicity is therefore whatever the handle provides. Pass the transaction that produced the business data and the outbox row commits with it; pass the pool and the row commits on its own. Both are legitimate, the call site decides, and rollback belongs to whoever opened the transaction.

type Message

type Message struct {
	// Topic routes the message. It is carried verbatim and handed back to
	// PublishFunc; this package never parses it. It must be non-empty.
	Topic string

	// Headers is optional string metadata carried verbatim alongside Payload
	// and handed back to PublishFunc at relay time. Its main use is propagating
	// trace context across the async hop, since the relay publishes in a
	// different goroutine long after the producer's span has closed: inject
	// your propagator's carrier here at insert time and read it back when
	// publishing. Idempotency keys and routing hints fit too. Nil when unused.
	Headers map[string]string

	// Payload is stored and relayed verbatim as raw bytes, so it can carry any
	// encoding the broker expects. It must be non-nil; an empty slice is legal.
	Payload []byte

	// Delay defers the message: it becomes due Delay after the insert rather
	// than as soon as the transaction commits. Zero is the ordinary case.
	// Negative is an error rather than a silent zero.
	Delay time.Duration
}

Message is an event to append to the outbox.

type Observer

type Observer struct {
	// Published is called after every publish attempt, successful or not. On
	// failure the row has already been deferred by the time this runs, so a
	// slow observer cannot sit between a failure and the recording of its
	// consequence.
	//
	// It is one field on purpose. Splitting success from failure would lose the
	// caller who wants to answer both with the same signal, a liveness probe
	// for instance, since a relay that cannot reach the broker is failing but
	// not wedged.
	//
	// ctx is the context Run was given, detached from cancellation: values
	// travel, so a tracer's baggage survives the async hop, but shutdown does
	// not cancel it — an attempt that is finishing during shutdown still gets
	// its report.
	Published func(ctx context.Context, delivery Delivery, err error)

	// Woke is called every time the relay stops waiting, whatever woke it: a
	// notification, the poll tick, a deferral coming due, the LISTEN session
	// coming up, or the run ending. It is the relay's idle heartbeat: it says
	// the wait loop is turning, and nothing about whether there was work.
	//
	// A LISTEN transition produces a beat of its own, on a pass that did not
	// wait at all: the relay claims straight away rather than sleep on a
	// conclusion its subscription was not covering. One beat per transition and
	// sometimes two, so a rate alert on this reads a little higher from a relay
	// whose session flaps.
	Woke func()

	// ListenerChanged is called on each LISTEN/NOTIFY connection transition,
	// once per transition rather than once per attempt: a non-nil err reports
	// the connection was lost or could not be opened, nil reports it came back.
	//
	// It exists because the degrade is otherwise silent. The relay keeps
	// draining on its poll tick with no push wake-ups, which looks like nothing
	// at all until latency is measured.
	//
	// A first successful dial is not a transition and is not reported. So
	// silence here means the connection has never changed state, not that it is
	// down, and a relay that dials once and keeps that session for a month
	// calls this exactly zero times. Count the changes if you want a signal;
	// deriving a level from the silence gets a healthy relay backwards.
	//
	// It also cannot see a push path that dies with the connection intact — a
	// session opened through a transaction-pooling pooler, or a table with no
	// NOTIFY trigger. Both leave this quiet and the relay polling. Delivery
	// latency is what catches those, because it measures the outcome instead of
	// the mechanism.
	ListenerChanged func(err error)

	// Pruned is called after each retention sweep with how many rows it deleted
	// and whether it failed. Housekeeping is never fatal, so this is the only
	// place a failing sweep is visible.
	Pruned func(deleted int64, err error)

	// Warned carries the non-fatal advisories the relay cannot refuse and will
	// not swallow: a poll interval at or above the lease, a RetryFunc whose
	// negative return had to be clamped, a row whose headers could not be
	// decoded, a deferral the database would not accept, a failed read of when
	// the next row falls due, and a callback of the caller's own that panicked.
	//
	// None of them stops delivery. All of them mean something is configured or
	// behaving in a way somebody should know about.
	//
	// The two conditions that do stop it, every publish slot wedged and a
	// shutdown that gave up waiting, are returned by Run instead. A caller with
	// no observer wired, which is every caller by default, must not be the one
	// who misses those.
	Warned func(err error)
}

Observer is how everything the relay does becomes visible. The package has no logger and no metrics of its own: it reports, and the caller decides what a report is worth. Every field is optional; a nil one is simply not called.

All of them run on the relay's own goroutines, so a slow observer slows the relay down. Hand work off rather than blocking in one. Published in particular runs inside the delivery and holds its publish slot until it returns, so an observer that blocks there does not merely slow the relay: with every slot held it is indistinguishable from a wedged PublishFunc and will trip ErrPublishStalled.

They also run concurrently. Published in particular is called from every publisher goroutine, so up to WithMaxConcurrency of them at once, and Warned is reached from the dispatcher, from publishers and from Run itself. A callback that touches state of its own needs its own synchronisation. The obvious counter incremented in Published is a data race.

A callback that panics is contained, not fatal. It runs beside deliveries in flight, and letting the process die would abandon all of them; the package doc has the full cost. The panic becomes an advisory and is reported through Warned.

type Producer added in v0.3.0

type Producer struct {
	// contains filtered or unexported fields
}

Producer appends messages to one outbox table. It is the whole write side: build one per table when the process starts, keep it for the life of the process, and hand it whichever handle is active at each call.

It holds no connection on purpose. A write repository that stored its pool could not commit with the caller's transaction, which is the one thing the outbox pattern is for.

The zero value is not usable. NewProducer renders the statement, so a Producer nobody built carries none. Insert reports ErrInvalidConfig instead of handing an empty statement to the driver, which panics.

func NewProducer added in v0.3.0

func NewProducer(opts ...ProducerOption) (*Producer, error)

NewProducer builds the write side for a table, "outbox" unless WithTable says otherwise.

The name is validated here and never again: a Producer that exists is one whose statement is safe to run, so Insert can only fail for reasons that have something to do with inserting. Build it where the error can be returned, in a provider or a constructor or main, and not on the path that writes rows.

Example

A differently-named table, and a handle resolved per call: the shape a transaction manager produces, where the active transaction lives on the context and the pool is the fallback outside one.

package main

import (
	"context"

	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"

	"github.com/gokern/outboxer"
)

func main() {
	var pool *pgxpool.Pool

	producer, err := outboxer.NewProducer(outboxer.WithTable("events"))
	if err != nil {
		return
	}

	// The relay draining this table has to be given the same name.
	//
	// A key type of this package's own, because a bare string is what
	// context.WithValue's own doc warns against: the key space is global to the
	// process, so any other package storing "tx" would silently collide here.
	type txKey struct{}

	handle := func(ctx context.Context) outboxer.Execer {
		if tx, ok := ctx.Value(txKey{}).(pgx.Tx); ok {
			return tx
		}

		return pool
	}

	ctx := context.Background()

	_ = producer.Insert(ctx, handle(ctx), outboxer.Message{
		Topic:   "invoice.expired",
		Payload: []byte(`{"id":42}`),
	})
}

func (*Producer) Insert added in v0.3.0

func (p *Producer) Insert(ctx context.Context, db Execer, msgs ...Message) error

Insert appends messages to the outbox through db, in one statement.

Whether the write is atomic with the caller's business data is decided by the handle, not here: a transaction-bound one gives atomicity, a bare pool writes on its own. Inserting nothing is not an error and touches no connection.

Example

The write side: the outbox row and the business data commit together, because they go through the same handle.

package main

import (
	"context"

	"github.com/jackc/pgx/v5/pgxpool"

	"github.com/gokern/outboxer"
)

func main() {
	var (
		ctx  context.Context
		pool *pgxpool.Pool
	)

	// Built once, where an error can still be returned. After this the name is
	// known good, so Insert can only fail for reasons to do with inserting.
	producer, err := outboxer.NewProducer()
	if err != nil {
		return
	}

	tx, err := pool.Begin(ctx)
	if err != nil {
		return
	}
	defer func() { _ = tx.Rollback(ctx) }()

	_, err = tx.Exec(ctx, `INSERT INTO users (email) VALUES ($1)`, "a@example.com")
	if err != nil {
		return
	}

	// The transaction, not the pool: that is what makes the row and the user
	// commit together or not at all.
	err = producer.Insert(ctx, tx, outboxer.Message{
		Topic:   "user.created",
		Payload: []byte(`{"email":"a@example.com"}`),
	})
	if err != nil {
		return
	}

	_ = tx.Commit(ctx)
}

type ProducerOption added in v0.3.0

type ProducerOption interface {
	// contains filtered or unexported methods
}

ProducerOption configures a Producer. The interface is sealed, its method unexported, so the only options that exist are the ones in this package and a setting belonging to the relay cannot be handed to NewProducer at all.

That separation is why the two sides do not share one option type. A mistake the compiler can refuse beats one a constructor has to report, and this package will not silently ignore an option that does not apply.

type PublishFunc

type PublishFunc func(ctx context.Context, delivery Delivery) error

PublishFunc hands one message to the broker. The caller supplies it, which is what keeps this package broker-agnostic.

It runs on a context detached from the relay's and bounded by the publish timeout, so a shutdown cannot abandon a row mid-publish. Returning an error is not fatal: RetryFunc defers the row and it is tried again.

type Querier added in v0.2.0

type Querier interface {
	QueryRow(ctx context.Context, sql string, args ...any) pgx.Row
}

Querier is a Postgres handle this package reads through: a *pgxpool.Pool, a *pgxpool.Conn or a pgx.Tx all satisfy it.

The single method is a guarantee, for the same reason Execer's is: with no Exec and no Begin, a Sampler cannot write and cannot open a transaction. It is also the whole burden on anyone wrapping a handle (a query logger, a tenant router, a fake), who would otherwise have to produce a pgx.Rows for a method that is never called.

Pass a pool. A *pgx.Conn satisfies this too and is not safe for concurrent use — the hazard NewRelay avoids by taking no interface at all — and here a Sampler keeps its handle for life, where a Producer is handed one per call.

type Relay

type Relay struct {
	// contains filtered or unexported fields
}

Relay drains one outbox table to a broker. Build it with NewRelay and run it with Run; there is nothing else to call, and nothing to close.

The zero value is not usable. NewRelay supplies the pool and the statements, so a Relay nobody built would claim nothing and report a clean stop when its context ended, which is indistinguishable from an idle one. Run reports ErrInvalidConfig instead.

func NewRelay

func NewRelay(pool *pgxpool.Pool, publish PublishFunc, opts ...RelayOption) (*Relay, error)

NewRelay builds a relay over pool, publishing through publish and draining the table WithTable names — "outbox" unless it says otherwise.

The pool is a concrete *pgxpool.Pool and not an interface. That is the one asymmetry with the write side, and it is deliberate. Producer.Insert takes an Execer because that write has to be able to go through the caller's transaction. Nothing the relay does belongs in the caller's transaction, and the relay reads and writes from every publisher goroutine at once. An interface narrow enough to be useful here would also be satisfied by *pgx.Conn, which is not safe for concurrent use, and that data race would compile.

It validates the configuration as a whole, which is why the settings are one value rather than a set of closures. The invariant worth naming here is lease > publish timeout + the internal mark bound. A shorter lease lets a second replica reclaim a row the first is still publishing, and a broker's dedupe window cannot be relied on to absorb the duplicate that follows. The error spells out all three durations, the internal one included, so a legal lease can be computed without knowing this package's constants.

A poll interval at or above the lease is reported through Observer.Warned instead of refused. It delays the reclaim of a crashed replica's row, which is slower but not wrong.

Example

The read side, wired the way a relay binary would: a dialer for the LISTEN session, a retry policy, and an observer for everything else.

package main

import (
	"context"
	"log/slog"
	"time"

	"github.com/jackc/pgx/v5"
	"github.com/jackc/pgx/v5/pgxpool"

	"github.com/gokern/outboxer"
)

func main() {
	var (
		ctx     context.Context
		pool    *pgxpool.Pool
		dsn     string
		publish func(ctx context.Context, delivery outboxer.Delivery) error
	)

	relay, err := outboxer.NewRelay(pool, publish,
		// "outbox" is the default, so this line is only needed once the table
		// has been renamed, and then the producer needs the same one.
		outboxer.WithTable("outbox"),
		// The relay never opens a connection itself: this is the one place the
		// application says how. It must not run through a transaction-pooling
		// pooler, which would drop the LISTEN between statements.
		outboxer.WithDialer(func(ctx context.Context) (*pgx.Conn, error) {
			return pgx.Connect(ctx, dsn)
		}),
		outboxer.WithMaxConcurrency(16),
		outboxer.WithLease(time.Minute),
		outboxer.WithPublishTimeout(5*time.Second),
		outboxer.WithPollInterval(10*time.Second),
		outboxer.WithRetention(7*24*time.Hour),
		outboxer.WithRetry(func(delivery outboxer.Delivery, _ error) time.Duration {
			return min(time.Duration(delivery.Attempt)*time.Second, time.Minute)
		}),
		outboxer.WithObserver(outboxer.Observer{
			Published: func(_ context.Context, delivery outboxer.Delivery, err error) {
				if err != nil {
					slog.Warn("outbox publish failed, row stays deferred",
						"topic", delivery.Topic, "id", delivery.ID, "attempt", delivery.Attempt, "err", err)
				}
			},
			ListenerChanged: func(err error) {
				if err != nil {
					slog.Warn("outbox listener lost, falling back to polling", "err", err)

					return
				}

				slog.Info("outbox listener recovered")
			},
			Pruned: func(deleted int64, err error) {
				slog.Info("outbox pruned", "deleted", deleted, "err", err)
			},
			Warned: func(err error) { slog.Warn("outbox", "err", err) },
		}),
	)
	if err != nil {
		return
	}

	// Run returns nil when ctx is cancelled, after in-flight publishes finish.
	err = relay.Run(ctx)
	if err != nil {
		slog.Error("outbox relay stopped", "err", err)
	}
}

func (*Relay) Run

func (r *Relay) Run(ctx context.Context) error

Run drains the outbox until ctx is cancelled.

It returns nil only on a clean stop: ctx cancelled, and every in-flight publish finished and recorded. A nil return is therefore the caller's licence to close the pool and exit.

Everything else it returns is a condition the relay could not carry on through, and each has a sentinel to match:

  • ErrShutdownIncomplete: the bounded wait for in-flight publishes expired. They are still running, detached, and will finish and mark themselves if the process lives long enough. Exiting now leaves rows published but unmarked, and each returns at lease expiry as a duplicate. The bound is not optional, or a PublishFunc that ignores its own deadline would hang the caller's whole task group.
  • ErrPublishStalled: every publish slot occupied for twice as long as one delivery may legally take, so nothing is being published at all. The goroutines holding those slots outlive this call. Nothing can reclaim them, and only a restart of the process recovers.
  • ErrSchemaMismatch or a storage error from the claim. A dropped connection is the ordinary cause and is not exotic: a failover, a pooler restart and an administrator's pg_terminate_backend all produce one, and it clears by itself once the pool has cycled out its dead connections. Nothing was claimed and nothing was published, so the cost is delay and not correctness. The relay stops anyway, because how long an unreachable database is worth waiting for is the caller's policy and not this package's. Build another Relay and run that; ExampleRelay_Run is the supervisor this implies.
  • A storage error from the mark. The row was published and the database does not know, so that duplicate is already guaranteed; stopping is what keeps it from becoming a stream of them.
  • ErrAlreadyRun: a second call. A Relay is single-use and says so instead of panicking.

A publish failure is never fatal, housekeeping is never fatal, and a claim that failed because ctx was cancelled is shutdown, not failure. Without that carve-out every clean deploy would report itself as a crash.

It owns whatever DialFunc returns: it closes that connection on exit and re-dials after a break.

On the two error paths above that leave publishers running, those goroutines still hold pool connections. Closing the pool immediately blocks until they release it instead of failing them, so a caller that wants a prompt exit should treat a non-nil error as a reason to wait or to abandon the process, not as a reason to close and carry on.

Example

Supervising the relay, which is what keeps an outbox draining across a failover.

Run is single-use, and a second call returns ErrAlreadyRun. It returns on any storage failure, of which a connection dropped by a failover or a pooler restart is the ordinary one. Carrying on therefore means building another relay, not calling Run again.

package main

import (
	"context"
	"errors"
	"log/slog"
	"time"

	"github.com/jackc/pgx/v5/pgxpool"

	"github.com/gokern/outboxer"
)

func main() {
	var (
		pool    *pgxpool.Pool
		publish func(ctx context.Context, delivery outboxer.Delivery) error
	)

	// The context your process shuts down on: cancelling it is what makes this
	// loop return instead of building yet another relay.
	ctx := context.Background()

	for {
		relay, err := outboxer.NewRelay(pool, publish)
		if err != nil {
			// Configuration, not conditions: the next attempt fails identically,
			// so looping here would spin forever.
			slog.Error("outbox relay misconfigured", "err", err)

			return
		}

		err = relay.Run(ctx)

		switch {
		case ctx.Err() != nil:
			return

		case errors.Is(err, outboxer.ErrPublishStalled),
			errors.Is(err, outboxer.ErrShutdownIncomplete):
			// Both leave publisher goroutines running, and still holding pool
			// connections. Nothing in Go can reclaim them, so rebuilding here
			// would leak another set on every pass. End the process instead
			// and let whatever supervises it start a clean one.
			slog.Error("outbox relay needs a fresh process", "err", err)

			return
		}

		slog.Error("outbox relay stopped, rebuilding", "err", err)

		select {
		case <-ctx.Done():
			return
		case <-time.After(time.Second):
		}
	}
}

type RelayOption

type RelayOption interface {
	// contains filtered or unexported methods
}

RelayOption configures a Relay. Every one returns an error instead of clamping or ignoring a bad value, so what the caller declared and what the relay runs are the same thing. That is also why there is no Config type to read the values back from: the caller already holds them.

func WithDialer

func WithDialer(dial DialFunc) RelayOption

WithDialer turns on LISTEN/NOTIFY, opening the dedicated session it needs through dial. Without it the relay polls, which is correct everywhere and slower only where latency is measured in the tens of milliseconds.

It takes a dialer rather than a DSN, and is opt-in rather than defaulted, for one and the same reason: a helper that dialled the pool's own connection string would succeed behind a transaction-pooling pooler and then deliver no notifications at all. That failure looks like working software.

func WithLease

func WithLease(lease time.Duration) RelayOption

WithLease sets how long a claimed row stays undue. It protects against a crashed worker and nothing else: a process that died ran no callback and could report nothing, so the only thing that can release its rows is time.

NewRelay refuses a lease that does not exceed the publish timeout plus the internal mark bound, because a shorter one lets a second replica reclaim a row the first is still publishing. The refused inequality covers those two bounds and nothing else — the claim's round trip, goroutine scheduling and the mark's travel all sit outside it — so leave slack above the floor rather than tuning the lease down to it.

func WithMaxConcurrency

func WithMaxConcurrency(n int) RelayOption

WithMaxConcurrency bounds how many publishes may be in flight at once. It is a ceiling, not a steady count: the relay claims exactly as many rows as it has free slots, so a quiet outbox runs no goroutines at all.

The default is 16 and deliberately not NumCPU. Publishing is I/O-bound, so the real limits are broker latency and pool size, and both are the caller's to declare. Size the pool to this number plus your own headroom.

func WithMaxPublishedRows added in v0.6.0

func WithMaxPublishedRows(maxRows int64) RelayOption

WithMaxPublishedRows limits retained published rows, keeping the newest by (published_at, id). Zero disables the limit, which is the default. Pending rows are never removed. Cleanup is periodic, so this is a soft row limit, not a bound on backlog, payload bytes or disk usage. WithRetention can remove additional rows by age; count retention can remove rows before that age.

Each sweep scans up to maxRows+1 visible published index entries to find its cutoff. Apply migration/0002_index_published_order.up.sql for ordered access.

func WithObserver

func WithObserver(observer Observer) RelayOption

WithObserver registers the callbacks the relay reports through. Fields left nil are not called, so wiring one of them costs nothing for the rest.

func WithPollInterval

func WithPollInterval(interval time.Duration) RelayOption

WithPollInterval sets how long the relay waits without a notification before looking anyway. Lease expiry produces no NOTIFY, so a poll is the only thing that finds a crashed replica's row.

Keep it below the lease. One at or above it is not refused, since it delays that reclaim without breaking it, but it is reported through Observer.Warned.

func WithPruneInterval

func WithPruneInterval(every time.Duration) RelayOption

WithPruneInterval sets how often retention sweeps, with a leading sweep at startup so a process that restarts more often than the interval still prunes.

It requires WithRetention or WithMaxPublishedRows. Without either policy the loop never starts, so NewRelay refuses an interval rather than accepting a setting that would do nothing.

Left alone it is an hour, or the retention window when that is shorter. An hourly sweep under a five-minute retention would keep rows twelve times longer than asked, and nobody who wrote five minutes meant an hour.

Set explicitly, NewRelay refuses an interval longer than the window: there both numbers came from the caller and they do not agree, and this package refuses rather than picking one.

func WithPublishTimeout

func WithPublishTimeout(timeout time.Duration) RelayOption

WithPublishTimeout bounds one PublishFunc call. The call runs on a context detached from cancellation so a shutdown cannot abandon a row mid-publish, which means the caller's own deadline cannot reach it and this is the only bound there is.

func WithRetention

func WithRetention(maxAge time.Duration) RelayOption

WithRetention deletes published rows older than maxAge. Zero, the default, keeps every row forever.

A row is deleted on the first sweep after it becomes eligible, so the age one actually reaches is maxAge plus up to one sweep interval: never less than maxAge, never more than maxAge plus WithPruneInterval. Saying only the lower bound would leave a five-minute retention behaving like an hourly one with nothing to explain it.

Keeping them costs disk and nothing else: the claim reads a partial index over unpublished rows, so the published ones it never visits do not slow it down however many of them there are. An outbox kept forever is a legitimate audit log. If you do enable this, add the index on published_at.

func WithRetry

func WithRetry(retry RetryFunc) RelayOption

WithRetry sets the backoff policy for a failed publish. The default defers by one lease, which is what the row would have waited for anyway had the process died instead of returning an error.

type RetryFunc

type RetryFunc func(delivery Delivery, err error) time.Duration

RetryFunc decides when a row that failed to publish becomes due again. That is policy, and therefore the caller's; the default defers by one lease.

Zero is legal and means the next dispatch pass. A negative return is clamped to zero and reported through Observer.Warned, because the alternative is a row whose due time is in the past forever.

It carries no cap on purpose. A lost acknowledgement for a message the broker did store looks exactly like a failure from here, so a duplicate is possible whatever this returns; capping every caller's policy to narrow a window that idempotency already covers is the wrong trade.

type Sampler added in v0.2.0

type Sampler struct {
	// contains filtered or unexported fields
}

Sampler reads the state of one outbox table. Build one per table when the process starts and keep it for the life of the process.

It is deliberately not a method on Relay. Run is single-use, and this reading is wanted precisely when no relay is running: a crashed process reports nothing, and the table is the only thing left that can.

It holds its handle, unlike Producer, which cannot: a producer that stored a pool could not commit with the caller's transaction, and that is the whole point of the outbox. A read is under no such constraint, and nothing here mutates after NewSampler returns, so a Sampler is safe for concurrent use exactly as far as its handle is: a pool yes, a single connection no.

The zero value is not usable. NewSampler renders the statement, so a Sampler nobody built carries none, and Stats reports ErrInvalidConfig rather than letting the mistake surface as a nil dereference on its own handle.

func NewSampler added in v0.2.0

func NewSampler(db Querier, opts ...SamplerOption) (*Sampler, error)

NewSampler builds the reading side for a table, "outbox" unless WithTable says otherwise.

The name is validated here and never again, so a Sampler that exists is one whose statement is safe to run. Build it where the error can be returned and not on the path that scrapes.

Example

The reading that outlives the relay: built once beside the producer, called on a timer by whatever exports your metrics. Nothing here starts a goroutine, so the cadence and the timeout belong to the caller.

package main

import (
	"context"
	"log/slog"
	"time"

	"github.com/jackc/pgx/v5/pgxpool"

	"github.com/gokern/outboxer"
)

func main() {
	var pool *pgxpool.Pool

	sampler, err := outboxer.NewSampler(pool, outboxer.WithTable("events"))
	if err != nil {
		return
	}

	ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
	defer cancel()

	stats, err := sampler.Stats(ctx)
	if err != nil {
		slog.Error("outbox unreadable", "err", err)

		// Report nothing, never a zero backlog: a gauge that drops to zero
		// because the database is unreachable is indistinguishable from an
		// outbox that has just been drained.
		return
	}

	// Due says nothing is claiming. A broker outage hides from it instead: the
	// relay keeps claiming and deferring, so Due stays near zero while
	// PeakAttempts climbs.
	slog.Info("outbox",
		"pending", stats.Pending,
		"due", stats.Due,
		"oldest_age_seconds", stats.OldestAge.Seconds(),
		"peak_attempts", stats.PeakAttempts)
}

func (*Sampler) Stats added in v0.3.0

func (s *Sampler) Stats(ctx context.Context) (Stats, error)

Stats reads the table's current state in one statement.

While the backlog is small this is a seek into the claim's partial index and the published rows are never touched, however many have accumulated. That stops holding once the pending rows are a large enough fraction that the planner prefers a sequential scan, which reads the published rows too, so the cost crosses over to tracking the whole table at the moment the backlog is worst: two to three orders of magnitude, measured on a table carrying as many pending rows as published ones.

Scrape it on an interval of tens of seconds, never per request.

type SamplerOption added in v0.3.0

type SamplerOption interface {
	// contains filtered or unexported methods
}

SamplerOption configures a Sampler. Sealed like the other two, and for the same reason: the reading side takes the table name and nothing else, so a lease handed to NewSampler is a compile error and not a setting ignored.

type Stats added in v0.2.0

type Stats struct {
	// Pending is every undelivered row: overdue, leased and deferred alike. It
	// tracks load, and on its own it says nothing.
	Pending int64

	// Due is the rows claimable at this instant. It detects exactly one thing,
	// that nothing is claiming.
	//
	// A broker outage is not that: there the relay keeps claiming, and each
	// failing row is due for an instant, then deferred by RetryFunc for a whole
	// lease, so this oscillates near zero while the backlog grows.
	Due int64

	// OldestAge is how long the oldest undelivered row has been waiting,
	// measured on the database clock. It is the SLO signal.
	//
	// A row scheduled far ahead with Message.Delay counts from when it was
	// written and not from when it becomes due, so a producer that schedules
	// days out carries a permanently large age here. Excluding those is
	// impossible — a deferred, a leased and a retried row all look the same —
	// so such a service alerts on Due and PeakAttempts instead.
	OldestAge time.Duration

	// PeakAttempts is the highest attempt count among undelivered rows. The
	// claim counts an attempt as it takes the lease, so frozen means nothing is
	// claiming and climbing means rows are claimed and failing.
	PeakAttempts int
}

Stats is one reading of the outbox table: the signal that outlives the process. Everything Observer reports dies with the relay, which is a poor place to keep the evidence that the relay is gone.

No single field is the alerting signal. They discriminate together, and the combination is what says which failure you have:

  • relay dead or wedged: Due grows, PeakAttempts frozen.
  • relay underprovisioned: Due grows, PeakAttempts low.
  • broker down: Due near zero, PeakAttempts climbing.
  • one poison row: Due near zero, PeakAttempts climbing on that row alone, Pending flat.

Due and OldestAge are lock-blind: a row a replica holds mid-claim has not committed its new ready_at yet, so it still counts. The window is one claim round trip, which under heavy contention reads as a handful of rows that look unclaimed and are not.

type TableOption

type TableOption string

TableOption carries the table name to whichever side is being built. It is the one option every constructor accepts.

It is a concrete type rather than an interface satisfying all three, because an exported interface would read as "the option type of this package" and invite `func myOptions() []outboxer.Option` — which compiles, and then does not: interface slices are invariant, so []Option is assignable to none of []ProducerOption, []RelayOption or []SamplerOption. A value built by hand instead of through WithTable is still validated where it is applied.

func WithTable

func WithTable(name string) TableOption

WithTable names the outbox table, on either side. It defaults to "outbox", which is what the reference DDL creates.

Unlike every other default here, this one has to agree with something outside the process: the migration that created the table, and the other binary. If you renamed the table, both the producer and the relay have to be given this option — where both tables exist a mismatch is invisible from in here, rows piling up in one table while the relay drains the other, with no error anywhere. Declare the name once in your own configuration and pass that value to both.

The name must be a plain unquoted identifier: lowercase, starting with a letter or underscore, at most 63 bytes, which is also PostgreSQL's limit on the channel name it becomes. That restriction is what makes it safe to interpolate into a statement. Schema qualification is out of scope; set search_path on the connection instead.

Directories

Path Synopsis
outboxprom module

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL