Documentation
¶
Overview ¶
Package outboxer is a transactional outbox for Postgres.
A producer appends messages to an outbox table inside the transaction that produced them, so the fact and the intent to publish it commit together or not at all. A relay claims those rows, hands each to a publish function the caller supplies, and marks it delivered. The two sides are independent: a producer that only writes never runs a relay, and neither has to know what broker the other end talks to.
The two sides ¶
NewProducer builds the write side. Producer.Insert takes the Postgres handle per call: pass a transaction-bound handle and the write is atomic with the business data, pass the pool and the row commits on its own. Both are legitimate, and the call site decides.
NewRelay builds the read side over a pool, and Relay.Run drains it until its context is cancelled. Delivery is at-least-once and there is no ordering guarantee: rows are claimed with FOR UPDATE SKIP LOCKED, so concurrent workers and replicas are safe but unordered.
NewSampler builds a third thing that is neither: a reading of the table itself. Sampler.Stats reports how much is undelivered, how much is overdue, how old the oldest unpublished fact is and the worst attempt count among them. It is separate from the relay because that reading is wanted precisely when no relay is running: a crashed process reports nothing.
All three name their table through WithTable, which defaults to "outbox". That name is the one fact they have to agree on, so it is also the one option every constructor accepts. Every other setting belongs to the relay, and the compiler refuses it on NewProducer and NewSampler alike.
Panics in caller code ¶
Every function the caller supplies runs on a relay goroutine: PublishFunc, RetryFunc, DialFunc and every field of Observer. A panic in one of them is contained and never fatal, because a panic that reached the runtime would abandon every delivery in flight beside it, rows already published and about to be marked included, each of which comes back at lease expiry as a duplicate.
Where the contained panic arrives says which function raised it. A PublishFunc panic is the outcome of the delivery, so it becomes that publish's error and travels to RetryFunc and Observer.Published, and the row is deferred like any other failed publish. Every other panic is an advisory alongside the work and reaches Observer.Warned, except a DialFunc's, which reaches Observer.ListenerChanged, since from the listener's side it is a dial that produced no connection. A panic in Warned itself arrives nowhere: the reporting channel cannot report its own failure. A panicking RetryFunc also forfeits its say, and the row is deferred by one lease, which is what it would have waited had the process died.
Recovery goes through github.com/gokern/panics: panics.Is(err) reports that an error carries a panic, and panics.As(err) reaches the value it was raised with and the frames it came from. The test is a broad one: it holds for a panic contained anywhere, including one a PublishFunc recovered itself and reported as an ordinary error. The relay cannot tell those apart, and does not try.
A row that panics will very likely panic again, so a policy of its own belongs in RetryFunc.
What this package does not own ¶
It never opens a transaction. Every statement it issues is a single statement, so atomicity is whatever the handle passed to Producer.Insert provides, and rollback belongs to whoever opened the transaction.
On the write side the type system enforces that: Execer has one method, and no Begin to call. On the read side it is discipline, not type. The relay holds a *pgxpool.Pool, which has Begin and everything else. The discipline holds because a relay operation needing two statements would need a transaction, and the shape of the package would change with it.
It never opens a connection. LISTEN/NOTIFY is opt-in through WithDialer, which takes a function called when the relay needs its dedicated session. The application keeps ownership of DSN, TLS, custom types and any AfterConnect hook, which is also where a statement_timeout belongs. Without a dialer the relay polls, and polling is correct everywhere.
It never runs a migration and never inspects the schema. The table, its indexes and its NOTIFY trigger are the application's to create and to keep correct; migration/0001_create_outbox.up.sql is the reference DDL and the whole of the contract. Postgres reports a missing table or column on the first statement that touches it, immediately and fatally, and this package only translates those SQLSTATEs into ErrSchemaMismatch.
What Postgres never reports, it never reports here either. A dropped partial index turns every claim into a sequential scan. A missing or misdirected NOTIFY trigger, or a LISTEN session opened through a transaction-pooling pooler, leaves the relay correct but poll-only. Both failures are silent. Catching them belongs to the migration review and to the monitoring that already owns the database, not to a library that would have to guess at the catalogue to do it.
It has no circuit breaker, no logger and no metrics. Every outcome the relay produces reaches the caller through Observer, and the state that outlives the relay through Sampler; stopping is the caller cancelling the context it passed to Relay.Run. Turning either into metrics is the caller's, since metric names and label sets are an organisation's conventions and not a library's. The outboxprom module beside this one picks a set of them and exports it, so adopting them is an import rather than a transcription.
Index ¶
- Variables
- type Delivery
- type DialFunc
- type Execer
- type Message
- type Observer
- type Producer
- type ProducerOption
- type PublishFunc
- type Querier
- type Relay
- type RelayOption
- func WithDialer(dial DialFunc) RelayOption
- func WithLease(lease time.Duration) RelayOption
- func WithMaxConcurrency(n int) RelayOption
- func WithObserver(observer Observer) RelayOption
- func WithPollInterval(interval time.Duration) RelayOption
- func WithPruneInterval(every time.Duration) RelayOption
- func WithPublishTimeout(timeout time.Duration) RelayOption
- func WithRetention(maxAge time.Duration) RelayOption
- func WithRetry(retry RetryFunc) RelayOption
- type RetryFunc
- type Sampler
- type SamplerOption
- type Stats
- type TableOption
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrInvalidConfig reports something the caller declared that this package // refuses rather than clamping or ignoring, so a declared value and the // effective one can never silently differ. // // Every option returns it, and so do NewProducer and NewRelay. It also // reaches the caller on three paths that are not settings but are the same // mistake, a value nobody configured: Producer.Insert on a zero-value // producer or a nil handle, Run on a zero-value relay, and Run again when a // DialFunc returns a nil connection and a nil error. Check for it on the // insert path too. Read as "storage failure" and retried, a nil handle // retries forever. ErrInvalidConfig = errors.New("invalid config") // ErrSchemaMismatch reports a table that does not match the contract: a // missing table or a missing column, translated from the SQLSTATE Postgres // raised on the first statement that touched it. This package never // inspects the schema itself, so this is the only way it can be reported. ErrSchemaMismatch = errors.New("schema mismatch") // ErrInvalidMessage reports a message this package refuses to write: an // empty topic, a nil payload, a negative delay, or a topic or header that // is not valid UTF-8 or carries a NUL byte. The reason is spelled out in // the message text. // // It is the distinction the write path turns on, and the reason it is // exported. A refused message is refused permanently: the same call will // fail the same way forever. A storage failure is transient, and there the // caller's transaction should be retried. Without a sentinel the two look // alike, so a caller with a retry loop around its transaction either // retries a poisoned message forever or gives up on a blip. // // Nothing was written when this is returned: a batch is rejected whole. ErrInvalidMessage = errors.New("invalid message") // ErrAlreadyRun reports a second call to Run on the same Relay. A Relay is // single-use, so a supervisor that restarts Run on error would otherwise // spin here forever without being able to tell why. ErrAlreadyRun = errors.New("relay has already been run") // ErrPublishStalled reports every publish slot occupied for twice as long // as one delivery may legally take, and is returned by Run. // // A PublishFunc that respects its context returns by the publish timeout, // and the slot frees once the mark that follows is done. Twice that bound // with nothing freed means at least one publish is not coming back. Go // cannot make it return, so the relay publishes nothing for the rest of the // process. Stopping lets a supervisor restart it, and only a restart // recovers. // // The goroutines it was waiting on are still running when Run returns. // There is no way to reclaim them, and the doc for Run says so. ErrPublishStalled = errors.New("no publish slot freed; a PublishFunc is not returning") // ErrShutdownIncomplete reports a Run that stopped waiting for publishes // still in flight, and is returned by Run. // // Run otherwise returns only after every in-flight publish has finished, so // a caller may close its pool and exit. On this path they have not: they // keep running detached, and a process that exits now leaves rows published // but unmarked, each of which returns at lease expiry as a duplicate. ErrShutdownIncomplete = errors.New("gave up waiting for in-flight publishes") // ErrRetryNegative reports a RetryFunc that returned a negative delay, // clamped to zero. It arrives through Observer.Warned. Unclamped it would // leave the row due in the past forever, so the clamp is not optional. But // it is the caller's own policy misbehaving, and they should hear about it. ErrRetryNegative = errors.New("retry policy returned a negative delay; clamped to 0") // ErrHeadersNotStrings reports a headers column this package cannot // represent, and arrives through Observer.Warned. Only a hand-written row // produces one: the write side always stores a flat object of strings. // // Skipping the row is not fatal to the relay. It keeps its lease and comes // back, so the advisory repeats until somebody fixes the row. ErrHeadersNotStrings = errors.New("headers are not a JSON object of strings") )
None of the sentinel texts names the package: every path out of it wraps them with context that carries the one "outboxer: " prefix, so a prefix in the sentinel would render twice.
Functions ¶
This section is empty.
Types ¶
type Delivery ¶
type Delivery struct {
// ID is the row's identity. Use it as the broker's dedupe key. Delivery is
// at-least-once, so a consumer that is not idempotent will eventually see
// the same message twice.
ID int64
// Attempt is which attempt this delivery is: a freshly claimed row arrives
// with Attempt == 1. The number is written by the claim, so a row that
// killed the process mid-delivery still counted and cannot loop forever at
// zero.
//
// It is here so PublishFunc can refuse a row whose count has gone absurd.
// That policy belongs to the caller; this package never gives up on a row.
Attempt int
// Topic is Message.Topic as written.
Topic string
// Headers is Message.Headers as written. It is nil when the row's headers
// column is NULL, which only a hand-written insert produces.
Headers map[string]string
// Payload is Message.Payload as written.
Payload []byte
// CreatedAt is when the row was written, on the database clock. It is the
// honest age of the fact: a producer-stamped header can be absent, skewed,
// or written by a process whose clock disagrees with Postgres.
CreatedAt time.Time
}
Delivery is a row claimed for publication.
type DialFunc ¶
DialFunc opens the dedicated session the relay LISTENs on. It is called when the relay needs that connection and again after the connection breaks, so it must return a fresh one every time — never a shared handle. Run owns what it returns and closes it.
It must honour ctx. The context is how the relay ends a dial at shutdown, and a DialFunc that ignores it holds up Run's return the same way a PublishFunc that ignores its deadline would.
It exists instead of a DSN option because the correct connection is the one the application knows how to open: its TLS, its custom types, its AfterConnect hook, and above all a session that does not run through a transaction-pooling pooler. Such a pooler registers LISTEN on a server connection it hands straight back to the pool, and then delivers nothing for the life of the process.
type Execer ¶ added in v0.3.0
type Execer interface {
Exec(ctx context.Context, sql string, args ...any) (pgconn.CommandTag, error)
}
Execer is a Postgres handle this package writes through: a *pgxpool.Pool, a *pgxpool.Conn or a pgx.Tx all satisfy it.
The single method is the guarantee: with no Begin to call, this package cannot open a transaction, and every statement it issues is a single statement. It is also the whole burden on anyone wrapping a handle (a query logger, a tenant router, a fake), who would otherwise have to produce a pgx.Rows for a method that is never called.
Atomicity is therefore whatever the handle provides. Pass the transaction that produced the business data and the outbox row commits with it; pass the pool and the row commits on its own. Both are legitimate, the call site decides, and rollback belongs to whoever opened the transaction.
type Message ¶
type Message struct {
// Topic routes the message. It is carried verbatim and handed back to
// PublishFunc; this package never parses it. It must be non-empty.
Topic string
// Headers is optional string metadata carried verbatim alongside Payload
// and handed back to PublishFunc at relay time. Its main use is propagating
// trace context across the async hop, since the relay publishes in a
// different goroutine long after the producer's span has closed: inject
// your propagator's carrier here at insert time and read it back when
// publishing. Idempotency keys and routing hints fit too. Nil when unused.
Headers map[string]string
// Payload is stored and relayed verbatim as raw bytes, so it can carry any
// encoding the broker expects. It must be non-nil; an empty slice is legal.
Payload []byte
// Delay defers the message: it becomes due Delay after the insert rather
// than as soon as the transaction commits. Zero is the ordinary case.
// Negative is an error rather than a silent zero.
Delay time.Duration
}
Message is an event to append to the outbox.
type Observer ¶
type Observer struct {
// Published is called after every publish attempt, successful or not. On
// failure the row has already been deferred by the time this runs, so a
// slow observer cannot sit between a failure and the recording of its
// consequence.
//
// It is one field on purpose. Splitting success from failure would lose the
// caller who wants to answer both with the same signal, a liveness probe
// for instance, since a relay that cannot reach the broker is failing but
// not wedged.
//
// ctx is the context Run was given, detached from cancellation: values
// travel, so a tracer's baggage survives the async hop, but shutdown does
// not cancel it — an attempt that is finishing during shutdown still gets
// its report.
Published func(ctx context.Context, delivery Delivery, err error)
// Woke is called every time the relay stops waiting, whatever woke it: a
// notification, the poll tick, a deferral coming due, or the run ending. It
// is the relay's idle heartbeat: it says the wait loop is turning, and
// nothing about whether there was work.
Woke func()
// ListenerChanged is called on each LISTEN/NOTIFY connection transition,
// once per transition rather than once per attempt: a non-nil err reports
// the connection was lost or could not be opened, nil reports it came back.
//
// It exists because the degrade is otherwise silent. The relay keeps
// draining on its poll tick with no push wake-ups, which looks like nothing
// at all until latency is measured.
ListenerChanged func(err error)
// Pruned is called after each retention sweep with how many rows it deleted
// and whether it failed. Housekeeping is never fatal, so this is the only
// place a failing sweep is visible.
Pruned func(deleted int64, err error)
// Warned carries the non-fatal advisories the relay cannot refuse and will
// not swallow: a poll interval at or above the lease, a RetryFunc whose
// negative return had to be clamped, a row whose headers could not be
// decoded, a deferral the database would not accept, a failed read of when
// the next row falls due, and a callback of the caller's own that panicked.
//
// None of them stops delivery. All of them mean something is configured or
// behaving in a way somebody should know about.
//
// The two conditions that do stop it, every publish slot wedged and a
// shutdown that gave up waiting, are returned by Run instead. A caller with
// no observer wired, which is every caller by default, must not be the one
// who misses those.
Warned func(err error)
}
Observer is how everything the relay does becomes visible. The package has no logger and no metrics of its own: it reports, and the caller decides what a report is worth. Every field is optional; a nil one is simply not called.
All of them run on the relay's own goroutines, so a slow observer slows the relay down. Hand work off rather than blocking in one. Published in particular runs inside the delivery and holds its publish slot until it returns, so an observer that blocks there does not merely slow the relay: with every slot held it is indistinguishable from a wedged PublishFunc and will trip ErrPublishStalled.
They also run concurrently. Published in particular is called from every publisher goroutine, so up to WithMaxConcurrency of them at once, and Warned is reached from the dispatcher, from publishers and from Run itself. A callback that touches state of its own needs its own synchronisation. The obvious counter incremented in Published is a data race.
A callback that panics is contained, not fatal. It runs beside deliveries in flight, and letting the process die would abandon all of them; the package doc has the full cost. The panic becomes an advisory and is reported through Warned.
type Producer ¶ added in v0.3.0
type Producer struct {
// contains filtered or unexported fields
}
Producer appends messages to one outbox table. It is the whole write side: build one per table when the process starts, keep it for the life of the process, and hand it whichever handle is active at each call.
It holds no connection on purpose. A write repository that stored its pool could not commit with the caller's transaction, which is the one thing the outbox pattern is for.
The zero value is not usable. NewProducer renders the statement, so a Producer nobody built carries none. Insert reports ErrInvalidConfig instead of handing an empty statement to the driver, which panics.
func NewProducer ¶ added in v0.3.0
func NewProducer(opts ...ProducerOption) (*Producer, error)
NewProducer builds the write side for a table, "outbox" unless WithTable says otherwise.
The name is validated here and never again: a Producer that exists is one whose statement is safe to run, so Insert can only fail for reasons that have something to do with inserting. Build it where the error can be returned, in a provider or a constructor or main, and not on the path that writes rows.
Example ¶
A differently-named table, and a handle resolved per call: the shape a transaction manager produces, where the active transaction lives on the context and the pool is the fallback outside one.
package main
import (
"context"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/gokern/outboxer"
)
func main() {
var pool *pgxpool.Pool
producer, err := outboxer.NewProducer(outboxer.WithTable("events"))
if err != nil {
return
}
// The relay draining this table has to be given the same name.
//
// A key type of this package's own, because a bare string is what
// context.WithValue's own doc warns against: the key space is global to the
// process, so any other package storing "tx" would silently collide here.
type txKey struct{}
handle := func(ctx context.Context) outboxer.Execer {
if tx, ok := ctx.Value(txKey{}).(pgx.Tx); ok {
return tx
}
return pool
}
ctx := context.Background()
_ = producer.Insert(ctx, handle(ctx), outboxer.Message{
Topic: "invoice.expired",
Payload: []byte(`{"id":42}`),
})
}
Output:
func (*Producer) Insert ¶ added in v0.3.0
Insert appends messages to the outbox through db, in one statement.
Whether the write is atomic with the caller's business data is decided by the handle, not here: a transaction-bound one gives atomicity, a bare pool writes on its own. Inserting nothing is not an error and touches no connection.
Example ¶
The write side: the outbox row and the business data commit together, because they go through the same handle.
package main
import (
"context"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/gokern/outboxer"
)
func main() {
var (
ctx context.Context
pool *pgxpool.Pool
)
// Built once, where an error can still be returned. After this the name is
// known good, so Insert can only fail for reasons to do with inserting.
producer, err := outboxer.NewProducer()
if err != nil {
return
}
tx, err := pool.Begin(ctx)
if err != nil {
return
}
defer func() { _ = tx.Rollback(ctx) }()
_, err = tx.Exec(ctx, `INSERT INTO users (email) VALUES ($1)`, "a@example.com")
if err != nil {
return
}
// The transaction, not the pool: that is what makes the row and the user
// commit together or not at all.
err = producer.Insert(ctx, tx, outboxer.Message{
Topic: "user.created",
Payload: []byte(`{"email":"a@example.com"}`),
})
if err != nil {
return
}
_ = tx.Commit(ctx)
}
Output:
type ProducerOption ¶ added in v0.3.0
type ProducerOption interface {
// contains filtered or unexported methods
}
ProducerOption configures a Producer. The interface is sealed, its method unexported, so the only options that exist are the ones in this package and a setting belonging to the relay cannot be handed to NewProducer at all.
That separation is why the two sides do not share one option type. A mistake the compiler can refuse beats one a constructor has to report, and this package will not silently ignore an option that does not apply.
type PublishFunc ¶
PublishFunc hands one message to the broker. The caller supplies it, which is what keeps this package broker-agnostic.
It runs on a context detached from the relay's and bounded by the publish timeout, so a shutdown cannot abandon a row mid-publish. Returning an error is not fatal: RetryFunc defers the row and it is tried again.
type Querier ¶ added in v0.2.0
Querier is a Postgres handle this package reads through: a *pgxpool.Pool, a *pgxpool.Conn or a pgx.Tx all satisfy it.
The single method is a guarantee, for the same reason Execer's is: with no Exec and no Begin, a Sampler cannot write and cannot open a transaction. It is also the whole burden on anyone wrapping a handle (a query logger, a tenant router, a fake), who would otherwise have to produce a pgx.Rows for a method that is never called.
Pass a pool. A *pgx.Conn satisfies this too and is not safe for concurrent use — the hazard NewRelay avoids by taking no interface at all — and here a Sampler keeps its handle for life, where a Producer is handed one per call.
type Relay ¶
type Relay struct {
// contains filtered or unexported fields
}
Relay drains one outbox table to a broker. Build it with NewRelay and run it with Run; there is nothing else to call, and nothing to close.
The zero value is not usable. NewRelay supplies the pool and the statements, so a Relay nobody built would claim nothing and report a clean stop when its context ended, which is indistinguishable from an idle one. Run reports ErrInvalidConfig instead.
func NewRelay ¶
func NewRelay(pool *pgxpool.Pool, publish PublishFunc, opts ...RelayOption) (*Relay, error)
NewRelay builds a relay over pool, publishing through publish and draining the table WithTable names — "outbox" unless it says otherwise.
The pool is a concrete *pgxpool.Pool and not an interface. That is the one asymmetry with the write side, and it is deliberate. Producer.Insert takes an Execer because that write has to be able to go through the caller's transaction. Nothing the relay does belongs in the caller's transaction, and the relay reads and writes from every publisher goroutine at once. An interface narrow enough to be useful here would also be satisfied by *pgx.Conn, which is not safe for concurrent use, and that data race would compile.
It validates the configuration as a whole, which is why the settings are one value rather than a set of closures. The invariant worth naming here is lease > publish timeout + the internal mark bound. A shorter lease lets a second replica reclaim a row the first is still publishing, and a broker's dedupe window cannot be relied on to absorb the duplicate that follows. The error spells out all three durations, the internal one included, so a legal lease can be computed without knowing this package's constants.
A poll interval at or above the lease is reported through Observer.Warned instead of refused. It delays the reclaim of a crashed replica's row, which is slower but not wrong.
Example ¶
The read side, wired the way a relay binary would: a dialer for the LISTEN session, a retry policy, and an observer for everything else.
package main
import (
"context"
"log/slog"
"time"
"github.com/jackc/pgx/v5"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/gokern/outboxer"
)
func main() {
var (
ctx context.Context
pool *pgxpool.Pool
dsn string
publish func(ctx context.Context, delivery outboxer.Delivery) error
)
relay, err := outboxer.NewRelay(pool, publish,
// "outbox" is the default, so this line is only needed once the table
// has been renamed, and then the producer needs the same one.
outboxer.WithTable("outbox"),
// The relay never opens a connection itself: this is the one place the
// application says how. It must not run through a transaction-pooling
// pooler, which would drop the LISTEN between statements.
outboxer.WithDialer(func(ctx context.Context) (*pgx.Conn, error) {
return pgx.Connect(ctx, dsn)
}),
outboxer.WithMaxConcurrency(16),
outboxer.WithLease(time.Minute),
outboxer.WithPublishTimeout(5*time.Second),
outboxer.WithPollInterval(10*time.Second),
outboxer.WithRetention(7*24*time.Hour),
outboxer.WithRetry(func(delivery outboxer.Delivery, _ error) time.Duration {
return min(time.Duration(delivery.Attempt)*time.Second, time.Minute)
}),
outboxer.WithObserver(outboxer.Observer{
Published: func(_ context.Context, delivery outboxer.Delivery, err error) {
if err != nil {
slog.Warn("outbox publish failed, row stays deferred",
"topic", delivery.Topic, "id", delivery.ID, "attempt", delivery.Attempt, "err", err)
}
},
ListenerChanged: func(err error) {
if err != nil {
slog.Warn("outbox listener lost, falling back to polling", "err", err)
return
}
slog.Info("outbox listener recovered")
},
Pruned: func(deleted int64, err error) {
slog.Info("outbox pruned", "deleted", deleted, "err", err)
},
Warned: func(err error) { slog.Warn("outbox", "err", err) },
}),
)
if err != nil {
return
}
// Run returns nil when ctx is cancelled, after in-flight publishes finish.
err = relay.Run(ctx)
if err != nil {
slog.Error("outbox relay stopped", "err", err)
}
}
Output:
func (*Relay) Run ¶
Run drains the outbox until ctx is cancelled.
It returns nil only on a clean stop: ctx cancelled, and every in-flight publish finished and recorded. A nil return is therefore the caller's licence to close the pool and exit.
Everything else it returns is a condition the relay could not carry on through, and each has a sentinel to match:
- ErrShutdownIncomplete: the bounded wait for in-flight publishes expired. They are still running, detached, and will finish and mark themselves if the process lives long enough. Exiting now leaves rows published but unmarked, and each returns at lease expiry as a duplicate. The bound is not optional, or a PublishFunc that ignores its own deadline would hang the caller's whole task group.
- ErrPublishStalled: every publish slot occupied for twice as long as one delivery may legally take, so nothing is being published at all. The goroutines holding those slots outlive this call. Nothing can reclaim them, and only a restart of the process recovers.
- ErrSchemaMismatch or a storage error from the claim. A dropped connection is the ordinary cause and is not exotic: a failover, a pooler restart and an administrator's pg_terminate_backend all produce one, and it clears by itself once the pool has cycled out its dead connections. Nothing was claimed and nothing was published, so the cost is delay and not correctness. The relay stops anyway, because how long an unreachable database is worth waiting for is the caller's policy and not this package's. Build another Relay and run that; ExampleRelay_Run is the supervisor this implies.
- A storage error from the mark. The row was published and the database does not know, so that duplicate is already guaranteed; stopping is what keeps it from becoming a stream of them.
- ErrAlreadyRun: a second call. A Relay is single-use and says so instead of panicking.
A publish failure is never fatal, housekeeping is never fatal, and a claim that failed because ctx was cancelled is shutdown, not failure. Without that carve-out every clean deploy would report itself as a crash.
It owns whatever DialFunc returns: it closes that connection on exit and re-dials after a break.
On the two error paths above that leave publishers running, those goroutines still hold pool connections. Closing the pool immediately blocks until they release it instead of failing them, so a caller that wants a prompt exit should treat a non-nil error as a reason to wait or to abandon the process, not as a reason to close and carry on.
Example ¶
Supervising the relay, which is what keeps an outbox draining across a failover.
Run is single-use, and a second call returns ErrAlreadyRun. It returns on any storage failure, of which a connection dropped by a failover or a pooler restart is the ordinary one. Carrying on therefore means building another relay, not calling Run again.
package main
import (
"context"
"errors"
"log/slog"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/gokern/outboxer"
)
func main() {
var (
pool *pgxpool.Pool
publish func(ctx context.Context, delivery outboxer.Delivery) error
)
// The context your process shuts down on: cancelling it is what makes this
// loop return instead of building yet another relay.
ctx := context.Background()
for {
relay, err := outboxer.NewRelay(pool, publish)
if err != nil {
// Configuration, not conditions: the next attempt fails identically,
// so looping here would spin forever.
slog.Error("outbox relay misconfigured", "err", err)
return
}
err = relay.Run(ctx)
switch {
case ctx.Err() != nil:
return
case errors.Is(err, outboxer.ErrPublishStalled),
errors.Is(err, outboxer.ErrShutdownIncomplete):
// Both leave publisher goroutines running, and still holding pool
// connections. Nothing in Go can reclaim them, so rebuilding here
// would leak another set on every pass. End the process instead
// and let whatever supervises it start a clean one.
slog.Error("outbox relay needs a fresh process", "err", err)
return
}
slog.Error("outbox relay stopped, rebuilding", "err", err)
select {
case <-ctx.Done():
return
case <-time.After(time.Second):
}
}
}
Output:
type RelayOption ¶
type RelayOption interface {
// contains filtered or unexported methods
}
RelayOption configures a Relay. Every one returns an error instead of clamping or ignoring a bad value, so what the caller declared and what the relay runs are the same thing. That is also why there is no Config type to read the values back from: the caller already holds them.
func WithDialer ¶
func WithDialer(dial DialFunc) RelayOption
WithDialer turns on LISTEN/NOTIFY, opening the dedicated session it needs through dial. Without it the relay polls, which is correct everywhere and slower only where latency is measured in the tens of milliseconds.
It takes a dialer rather than a DSN, and is opt-in rather than defaulted, for one and the same reason: a helper that dialled the pool's own connection string would succeed behind a transaction-pooling pooler and then deliver no notifications at all. That failure looks like working software.
func WithLease ¶
func WithLease(lease time.Duration) RelayOption
WithLease sets how long a claimed row stays undue. It protects against a crashed worker and nothing else: a process that died ran no callback and could report nothing, so the only thing that can release its rows is time.
NewRelay refuses a lease that does not exceed the publish timeout plus the internal mark bound, because a shorter one lets a second replica reclaim a row the first is still publishing. The refused inequality covers those two bounds and nothing else — the claim's round trip, goroutine scheduling and the mark's travel all sit outside it — so leave slack above the floor rather than tuning the lease down to it.
func WithMaxConcurrency ¶
func WithMaxConcurrency(n int) RelayOption
WithMaxConcurrency bounds how many publishes may be in flight at once. It is a ceiling, not a steady count: the relay claims exactly as many rows as it has free slots, so a quiet outbox runs no goroutines at all.
The default is 16 and deliberately not NumCPU. Publishing is I/O-bound, so the real limits are broker latency and pool size, and both are the caller's to declare. Size the pool to this number plus your own headroom.
func WithObserver ¶
func WithObserver(observer Observer) RelayOption
WithObserver registers the callbacks the relay reports through. Fields left nil are not called, so wiring one of them costs nothing for the rest.
func WithPollInterval ¶
func WithPollInterval(interval time.Duration) RelayOption
WithPollInterval sets how long the relay waits without a notification before looking anyway. Lease expiry produces no NOTIFY, so a poll is the only thing that finds a crashed replica's row.
Keep it below the lease. One at or above it is not refused, since it delays that reclaim without breaking it, but it is reported through Observer.Warned.
func WithPruneInterval ¶
func WithPruneInterval(every time.Duration) RelayOption
WithPruneInterval sets how often retention sweeps, with a leading sweep at startup so a process that restarts more often than the interval still prunes.
It requires WithRetention. Without a window there is nothing to sweep and the loop never starts, so NewRelay refuses the pair rather than accepting a setting that would do nothing.
Left alone it is an hour, or the retention window when that is shorter. An hourly sweep under a five-minute retention would keep rows twelve times longer than asked, and nobody who wrote five minutes meant an hour.
Set explicitly, NewRelay refuses an interval longer than the window: there both numbers came from the caller and they do not agree, and this package refuses rather than picking one.
func WithPublishTimeout ¶
func WithPublishTimeout(timeout time.Duration) RelayOption
WithPublishTimeout bounds one PublishFunc call. The call runs on a context detached from cancellation so a shutdown cannot abandon a row mid-publish, which means the caller's own deadline cannot reach it and this is the only bound there is.
func WithRetention ¶
func WithRetention(maxAge time.Duration) RelayOption
WithRetention deletes published rows older than maxAge. Zero, the default, keeps every row forever.
A row is deleted on the first sweep after it becomes eligible, so the age one actually reaches is maxAge plus up to one sweep interval: never less than maxAge, never more than maxAge plus WithPruneInterval. Saying only the lower bound would leave a five-minute retention behaving like an hourly one with nothing to explain it.
Keeping them costs disk and nothing else: the claim reads a partial index over unpublished rows, so the published ones it never visits do not slow it down however many of them there are. An outbox kept forever is a legitimate audit log. If you do enable this, add the index on published_at.
func WithRetry ¶
func WithRetry(retry RetryFunc) RelayOption
WithRetry sets the backoff policy for a failed publish. The default defers by one lease, which is what the row would have waited for anyway had the process died instead of returning an error.
type RetryFunc ¶
RetryFunc decides when a row that failed to publish becomes due again. That is policy, and therefore the caller's; the default defers by one lease.
Zero is legal and means the next dispatch pass. A negative return is clamped to zero and reported through Observer.Warned, because the alternative is a row whose due time is in the past forever.
It carries no cap on purpose. A lost acknowledgement for a message the broker did store looks exactly like a failure from here, so a duplicate is possible whatever this returns; capping every caller's policy to narrow a window that idempotency already covers is the wrong trade.
type Sampler ¶ added in v0.2.0
type Sampler struct {
// contains filtered or unexported fields
}
Sampler reads the state of one outbox table. Build one per table when the process starts and keep it for the life of the process.
It is deliberately not a method on Relay. Run is single-use, and this reading is wanted precisely when no relay is running: a crashed process reports nothing, and the table is the only thing left that can.
It holds its handle, unlike Producer, which cannot: a producer that stored a pool could not commit with the caller's transaction, and that is the whole point of the outbox. A read is under no such constraint, and nothing here mutates after NewSampler returns, so a Sampler is safe for concurrent use exactly as far as its handle is: a pool yes, a single connection no.
The zero value is not usable. NewSampler renders the statement, so a Sampler nobody built carries none, and Stats reports ErrInvalidConfig rather than letting the mistake surface as a nil dereference on its own handle.
func NewSampler ¶ added in v0.2.0
func NewSampler(db Querier, opts ...SamplerOption) (*Sampler, error)
NewSampler builds the reading side for a table, "outbox" unless WithTable says otherwise.
The name is validated here and never again, so a Sampler that exists is one whose statement is safe to run. Build it where the error can be returned and not on the path that scrapes.
Example ¶
The reading that outlives the relay: built once beside the producer, called on a timer by whatever exports your metrics. Nothing here starts a goroutine, so the cadence and the timeout belong to the caller.
package main
import (
"context"
"log/slog"
"time"
"github.com/jackc/pgx/v5/pgxpool"
"github.com/gokern/outboxer"
)
func main() {
var pool *pgxpool.Pool
sampler, err := outboxer.NewSampler(pool, outboxer.WithTable("events"))
if err != nil {
return
}
ctx, cancel := context.WithTimeout(context.Background(), 5*time.Second)
defer cancel()
stats, err := sampler.Stats(ctx)
if err != nil {
slog.Error("outbox unreadable", "err", err)
// Report nothing, never a zero backlog: a gauge that drops to zero
// because the database is unreachable is indistinguishable from an
// outbox that has just been drained.
return
}
// Due says nothing is claiming. A broker outage hides from it instead: the
// relay keeps claiming and deferring, so Due stays near zero while
// PeakAttempts climbs.
slog.Info("outbox",
"pending", stats.Pending,
"due", stats.Due,
"oldest_age_seconds", stats.OldestAge.Seconds(),
"peak_attempts", stats.PeakAttempts)
}
Output:
func (*Sampler) Stats ¶ added in v0.3.0
Stats reads the table's current state in one statement.
While the backlog is small this is a seek into the claim's partial index and the published rows are never touched, however many have accumulated. That stops holding once the pending rows are a large enough fraction that the planner prefers a sequential scan, which reads the published rows too, so the cost crosses over to tracking the whole table at the moment the backlog is worst: two to three orders of magnitude, measured on a table carrying as many pending rows as published ones.
Scrape it on an interval of tens of seconds, never per request.
type SamplerOption ¶ added in v0.3.0
type SamplerOption interface {
// contains filtered or unexported methods
}
SamplerOption configures a Sampler. Sealed like the other two, and for the same reason: the reading side takes the table name and nothing else, so a lease handed to NewSampler is a compile error and not a setting ignored.
type Stats ¶ added in v0.2.0
type Stats struct {
// Pending is every undelivered row: overdue, leased and deferred alike. It
// tracks load, and on its own it says nothing.
Pending int64
// Due is the rows claimable at this instant. It detects exactly one thing,
// that nothing is claiming.
//
// A broker outage is not that: there the relay keeps claiming, and each
// failing row is due for an instant, then deferred by RetryFunc for a whole
// lease, so this oscillates near zero while the backlog grows.
Due int64
// OldestAge is how long the oldest undelivered row has been waiting,
// measured on the database clock. It is the SLO signal.
//
// A row scheduled far ahead with Message.Delay counts from when it was
// written and not from when it becomes due, so a producer that schedules
// days out carries a permanently large age here. Excluding those is
// impossible — a deferred, a leased and a retried row all look the same —
// so such a service alerts on Due and PeakAttempts instead.
OldestAge time.Duration
// PeakAttempts is the highest attempt count among undelivered rows. The
// claim counts an attempt as it takes the lease, so frozen means nothing is
// claiming and climbing means rows are claimed and failing.
PeakAttempts int
}
Stats is one reading of the outbox table: the signal that outlives the process. Everything Observer reports dies with the relay, which is a poor place to keep the evidence that the relay is gone.
No single field is the alerting signal. They discriminate together, and the combination is what says which failure you have:
- relay dead or wedged: Due grows, PeakAttempts frozen.
- relay underprovisioned: Due grows, PeakAttempts low.
- broker down: Due near zero, PeakAttempts climbing.
- one poison row: Due near zero, PeakAttempts climbing on that row alone, Pending flat.
Due and OldestAge are lock-blind: a row a replica holds mid-claim has not committed its new ready_at yet, so it still counts. The window is one claim round trip, which under heavy contention reads as a handful of rows that look unclaimed and are not.
type TableOption ¶
type TableOption string
TableOption carries the table name to whichever side is being built. It is the one option every constructor accepts.
It is a concrete type rather than an interface satisfying all three, because an exported interface would read as "the option type of this package" and invite `func myOptions() []outboxer.Option` — which compiles, and then does not: interface slices are invariant, so []Option is assignable to none of []ProducerOption, []RelayOption or []SamplerOption. A value built by hand instead of through WithTable is still validated where it is applied.
func WithTable ¶
func WithTable(name string) TableOption
WithTable names the outbox table, on either side. It defaults to "outbox", which is what the reference DDL creates.
Unlike every other default here, this one has to agree with something outside the process: the migration that created the table, and the other binary. If you renamed the table, both the producer and the relay have to be given this option — where both tables exist a mismatch is invisible from in here, rows piling up in one table while the relay drains the other, with no error anywhere. Declare the name once in your own configuration and pass that value to both.
The name must be a plain unquoted identifier: lowercase, starting with a letter or underscore, at most 63 bytes, which is also PostgreSQL's limit on the channel name it becomes. That restriction is what makes it safe to interpolate into a statement. Schema qualification is out of scope; set search_path on the connection instead.