leaderelection

README

leaderelection

Leader election for services that share nothing but a database.

elector, _ := leaderelection.New(leaderelection.Config{
    Store: store,
    Name:  "invoice-reconciler",
    Callbacks: leaderelection.Callbacks{
        OnStartedLeading: func(ctx context.Context, fence int64) {
            reconcile(ctx, fence) // stop when ctx is cancelled
        },
    },
})
elector.Run(ctx)

---

Why this exists

Kubernetes ships a perfectly good leader election, and you should use it — right up until your replicas stop sharing a cluster.

Run the same service in two regions, in two clusters with no network path between them, and coordination.k8s.io/Lease cannot help: neither cluster's API server knows the other exists. The replicas still have exactly one thing in common — the database they were already talking to. So that is where the election happens.

This library elects one leader among processes that share a PostgreSQL (or, soon, MySQL) instance and nothing else. No etcd, no ZooKeeper, no Consul, no extra thing to operate. A table with one row per election.

It is built for the awkward version of that deployment: participants on the far side of a WAN, clocks that disagree, a PgBouncer in the path, and a database that occasionally stops answering.

Guarantees

At most one participant considers itself the leader at any instant. That holds through network partitions, database outages, connection pooler restarts, and processes killed without warning.

Three choices carry that weight.

Every timestamp comes from the database. Participants in different clusters have unrelated clocks, so a lease expiry expressed as an absolute time would mean different things to different participants. Each statement returns both the expiry and the server's current time, and only their difference — a duration — is ever used. Your clock can be three days off; nothing changes.

That duration is measured against the local monotonic clock, from the moment the request was sent. Not from when the reply arrived: the database started counting somewhere in between, and anchoring at the earlier bound keeps the local deadline strictly earlier than the real one.

A watchdog enforces that deadline independently of the renewal loop. This is what makes an unreachable database safe rather than merely inconvenient. A renewal that hangs forever cannot delay the step-down, because nothing about stepping down waits on the database. If a process cannot prove it still holds the lease, it stops acting as though it does — and it does so before the lease could possibly be handed to anyone else.

Uncertainty always resolves to not leader.

What it does not guarantee

It is not a consensus protocol. The database is the single source of truth and the single point of failure. If it is down, nobody leads. That is the correct outcome, and it is the trade you are making by not running etcd.

Liveness is bounded by the lease. A leader that dies without releasing takes up to LeaseDuration to be replaced. Measured on the test suite: ~3.09s with a 3s lease after an unannounced crash, ~0.43s after a clean shutdown, because a graceful exit releases the lease instead of letting it expire.

Stepping down is not the same as having stopped. The library cancels your context before anyone else can lead, but it cannot unsend a write already in flight. For that you need fencing tokens — see below.

The database's own clock has to move forwards. Participants' clocks may be days out and may run at different rates; the database's may not jump ahead. A server clock that steps forward retires a live lease early, and no participant reading that clock can tell it happened — the incumbent's deadline still says it holds what the database has already given away. Steps backwards are safe: they make every lease look longer to everyone at once, and the elector refuses to believe a lease longer than the one it asked for.

Install

go get github.com/brualan/leaderelection
go get github.com/brualan/leaderelection/postgres

The core module has no dependencies — standard library only, tests included. Drivers live in separate modules, so importing the PostgreSQL store never pulls in a MySQL driver and vice versa.

Quick start

package main

import (
	"context"
	"log/slog"
	"os/signal"
	"syscall"

	"github.com/brualan/leaderelection"
	"github.com/brualan/leaderelection/postgres"
)

func main() {
	ctx, stop := signal.NotifyContext(context.Background(), syscall.SIGINT, syscall.SIGTERM)
	defer stop()

	store, err := postgres.Connect(ctx, "postgres://user:pass@db:5432/app?sslmode=require")
	if err != nil {
		panic(err)
	}
	defer store.Close()

	if err := store.EnsureSchema(ctx); err != nil {
		panic(err)
	}

	elector, err := leaderelection.New(leaderelection.Config{
		Store:  store,
		Name:   "invoice-reconciler",
		Logger: slog.Default(),
		Callbacks: leaderelection.Callbacks{
			OnStartedLeading: func(ctx context.Context, fence int64) {
				// Runs for the whole term, in its own goroutine. The context is
				// cancelled the instant leadership becomes uncertain -- which
				// includes the database merely becoming unreachable. Respect it
				// promptly; nobody else can lead until this returns.
				reconcileUntil(ctx, fence)
			},
			OnStoppedLeading: func() {
				slog.Info("no longer the leader")
			},
		},
	})
	if err != nil {
		panic(err)
	}

	// Blocks until ctx is cancelled. Returns nil on a clean shutdown, so it
	// drops straight into an errgroup.
	if err := elector.Run(ctx); err != nil {
		panic(err)
	}
}

Polling instead of callbacks works too:

go elector.Run(ctx)

if elector.IsLeader() {
    // IsLeader re-checks the deadline on every call rather than trusting a
    // cached flag, so a descheduled watchdog cannot widen the window in which
    // a stale leader believes itself current.
}

Behind PgBouncer

Nothing changes:

store, err := postgres.Connect(ctx, "postgres://user:pass@pgbouncer:6432/app")

Connect configures the pool to survive transaction pooling by default. See PgBouncer for what that means and why.

Schema

CREATE TABLE IF NOT EXISTS leader_election (
    name        text        PRIMARY KEY,
    holder      text        NOT NULL,
    fence       bigint      NOT NULL,
    acquired_at timestamptz NOT NULL,
    renewed_at  timestamptz NOT NULL,
    expires_at  timestamptz NOT NULL,
    payload     bytea
) WITH (fillfactor = 70);

store.EnsureSchema(ctx) creates it — idempotent, safe on every replica on every start. If your deployment withholds DDL rights from the application, get the statement for your migration tool instead:

ddl, _ := postgres.Schema("", postgres.DefaultTable)

One row per election; unrelated elections share the table. Use postgres.WithTable(schema, table) for a different name.

fillfactor is not decoration. The row is rewritten on every renewal — several times a minute, forever — and leaving room on the page keeps those updates HOT, so the primary key index is untouched and autovacuum can reclaim old versions without an index scan.

Configuration

Field	Default	What it controls
LeaseDuration	15s	How long a lease stays valid. Sets worst-case failover after a crash.
RenewInterval	5s	How often the leader extends its lease.
RetryPeriod	3s	How often a follower polls. Jittered.
OperationTimeout	3s	Bounds a single database call.
SafetyMargin	LeaseDuration/10, min 1s	How early the leader steps down.
Identity	generated	Distinguishes this participant.
Payload	none	Data published with the lease, readable by followers.
DisableRelease	false	Skips the release on shutdown.

One invariant is enforced at construction, and New refuses a config that breaks it:

RenewInterval + 2*OperationTimeout + retryDelay <= LeaseDuration - SafetyMargin

A leader must be able to lose one renewal outright — request sent, timeout burned, nothing back — and still get a second attempt to complete before it has to step down. Without the slack, a single slow round trip costs leadership, and the election flaps under exactly the conditions it exists to survive.

OperationTimeout is charged twice on purpose: once for the attempt that is lost and once for the attempt that has to succeed. retryDelay is max(RenewInterval/4, 50ms).

For participants on the far side of a WAN, raise OperationTimeout and LeaseDuration together rather than shortening the renew interval.

Identity

Generated as hostname-pid-<random>. The random part matters. If a restarted process reused its predecessor's identity, it could renew the lease that predecessor still believes it holds — and the predecessor may well be alive, blocked in a syscall or a stop-the-world pause. Setting Identity to something stable across restarts, such as a StatefulSet pod name, defeats the guarantee.

Fencing tokens

Stepping down promptly bounds how long a stalled leader can act. It cannot unsend a write already in flight. If a leader is descheduled mid-write, steps down while frozen, and the write lands after its successor started — the election was correct and your data is still wrong.

The fix is standard and the library hands you the tool. Every term carries a strictly increasing fence:

OnStartedLeading: func(ctx context.Context, fence int64) {
    for job := range jobs {
        // Stamp the write, and have the receiving system reject anything
        // carrying a token older than the highest it has seen.
        if err := sink.Write(ctx, job, fence); err != nil {
            return
        }
    }
}

-- The receiving side, in whatever system it lives in.
UPDATE state SET value = $1, fence = $2 WHERE id = $3 AND fence <= $2;

No two terms ever share a token, anywhere in the election.

At the database level the token advances on one condition: reclaiming an expired lease. That covers a new holder taking over, and equally the previous holder returning to a lease it let lapse — in between, anyone could have led. Extending a lease that never lapsed is the same term and keeps its token.

There is one case the database cannot see. A leader whose renewals stop landing steps down locally, while its lease is still valid on the server — the local deadline is deliberately the earlier of the two. If the network heals, that participant could re-acquire its own untouched lease and the database would have no reason to renumber it. The library gives the lease up instead and takes a fresh token, so a write left in flight by the abandoned term can still be told apart from the new one's.

Cross-cluster leader discovery

Followers can read what the leader published, which is often the only channel between clusters that exists:

Config{Payload: []byte("https://leader.eu-west.internal:8443")}

// From any participant, leader or not:
status := elector.Status()
proxyTo(status.Leader, status.LeaderPayload)

Failure modes

What happens	What the library does	How long
Leader loses its network path	Steps down, cancels your context	< LeaseDuration - SafetyMargin
Database hangs without closing connections	Steps down; the watchdog does not wait on the network	< LeaseDuration - SafetyMargin
Database goes down entirely	Nobody leads	within one lease
PgBouncer restarts	Pools rebuild; leadership usually survives untouched	—
PgBouncer PAUSE	Nobody leads until RESUME	within one lease
Leader is killed with SIGKILL	Successor waits out the lease	≈ LeaseDuration
Leader shuts down cleanly	Releases; successor takes over at once	≈ RetryPeriod
Lease taken by someone else	Steps down on the failed renewal, not at the deadline	≈ RenewInterval
Table missing, no privileges	Nobody leads; postgres.Fatal(err) reports it	until fixed

Errors never come back from Run. They are reported through Status().LastError and handled by giving up leadership rather than by giving up. postgres.Retryable and postgres.Fatal classify them for alerting: both stop the election, but only one will fix itself.

PgBouncer

Transaction pooling hands back a different server connection after every transaction. Anything session-scoped breaks, so this library uses none of it:

• No pg_advisory_lock. Session-scoped, so it would break outright. Lease state lives in a row instead.
• No LISTEN/NOTIFY, no temporary tables, no SET.
• No server-side prepared statements. Connect defaults to pgx.QueryExecModeExec and disables the statement and description caches. Named prepared statements are per-session, so under transaction pooling they land on whichever backend the pooler hands out next and fail at random.
• One statement per operation. Nothing spans two transactions, so nothing spans two backends.

Bounded connection lifetimes and health checks are set too, so the pool sheds connections a pooler restart left unusable instead of discovering them one failed election at a time.

What you set in the DSN wins, with one exception: pgx cannot tell a DSN that asked for cache_statement from one that said nothing, so that mode is still moved to exec. Any other mode is left alone, and a deployment with no pooler can build its own pool and pass it to postgres.New.

The acceptance suite runs unchanged against both paths, with max_prepared_statements=0 on the pooler. No exemptions: "works directly but not through the pooler" is this library's likeliest bug, and the tests are arranged so it cannot hide.

Testing

Every production package is at 100% statement coverage, enforced in CI. The only exclusions are test-support packages, whose uncovered statements are the assertion failures inside them — reachable only by making a test fail. The exclusion is per package, never per line.

make test        # unit, simulation and fuzz corpora, race detector, no Docker
make test-int    # against real PostgreSQL, direct and via PgBouncer
make test-chaos  # partitions, hangs, restarts, locks, pooler failures
make test-soak   # long randomised runs, rolling restarts, pool exhaustion
make fuzz        # explore each fuzz target (FUZZTIME=5m for a longer look)
make cover       # merged profiles, 100% floor

Five layers:

Unit tests drive the election loop on a fake clock, so timing assertions do not depend on timing. The database's clock is deliberately offset from the local one by three days: any code that compared them would produce nonsense and the tests would notice.

A simulation runs a whole fleet on virtual time — several participants, one shared record, and a fault schedule taken from fuzz input. It buys two things no other layer can. Minutes of election happen in milliseconds, so a fuzzer can explore thousands of schedules; and each participant's clock can be made to run at a different rate, which is precisely what SafetyMargin exists to absorb. The oracle is sampled at every step of virtual time: no two participants leading at once, no fence token used twice, and — once the faults stop — somebody leading again, because an election that stays safe by never electing anyone has not survived anything.

Fuzz targets cover the arithmetic the safety argument rests on: the timing invariant that decides which configurations are accepted, the lease duration maths, the backoff delays, the quoting of a table name, and an operation-level model of the store contract that runs against both the in-memory reference and real PostgreSQL. Their seed corpora run as ordinary tests, so every go test replays them.

The acceptance suite (storetest) pins down store semantics — atomic takeover, fence tokens that never repeat, guarded renewals that fail rather than silently succeeding. Every implementation runs it, including the in-memory reference, which is what keeps the guarantee from becoming dialect-specific.

Chaos scenarios run a real fleet against real infrastructure and break it underneath them. Participants share one process so their leadership intervals can be compared directly; the safety oracle asserts that no two ever overlap and that fence tokens never go backwards.

Scenario	What breaks
TestSteadyState	nothing — the baseline
TestLeaderIsPartitioned	the leader's path is severed, others keep theirs
TestLeaderIsBlackholed	packets stop moving; connections stay up
TestDatabaseIsPaused	SIGSTOP on PostgreSQL, connections intact
TestDatabaseRestarts	PostgreSQL stopped and started
TestPgBouncerRestarts	the pooler bounced under live clients
TestPgBouncerPauses	PAUSE on the admin console
TestLeaderCrashes	the leader dies without releasing
TestGracefulHandoverIsFast	a clean shutdown, measuring failover
TestHighLatencyIsSurvivable	WAN latency and jitter on every path
TestRepliesAreLostButRequestsArrive	renewals land; the answers never come back
TestConnectionsAreResetMidQuery	every connection RSTs on first use
TestBandwidthCollapse	the path is up and too slow to renew over
TestFlappingPath	the leader's path cut and healed, repeatedly
TestEveryoneIsPartitioned	nobody can reach the database at all
TestParticipantsSeeDifferentDatabaseClocks	each participant is days out, differently
TestTableIsLockedByAnotherSession	ACCESS EXCLUSIVE, as a migration takes it
TestElectionRowIsLocked	reads work, the row cannot be written
TestBackendsAreTerminated	pg_terminate_backend, as a failover does
TestDatabaseGoesReadOnly	writes fail with 25006, reads do not
TestPostgresIsKilled	SIGKILL and crash recovery
TestRandomisedSoak	a minute of randomly injected faults

Most run against both the direct and the pooled path. Longer scenarios live behind make test-soak: a randomised soak that also replaces participants mid-flight, a rolling restart of the whole fleet, more participants than the pooler has server connections, and two elections sharing one table while one of them is broken.

Testing your own application

An election backed by memstore exercises the real elector — promotion, renewal, watchdog, fencing — without a database:

import "github.com/brualan/leaderelection/memstore"

elector, _ := leaderelection.New(leaderelection.Config{
    Store: memstore.New(),
    Name:  "test",
})

It elects among goroutines in one process, so it is not a substitute for the real stores.

Operating it

-- Who leads what, and for how long.
SELECT name, holder, fence,
       expires_at - now() AS remaining,
       now() - acquired_at AS term_age
FROM leader_election;

Worth alerting on: remaining persistently negative (nobody is leading), fence climbing quickly (leadership is flapping — usually the timing invariant is too tight for your latency), and any error where postgres.Fatal is true.

The table is small and hot. fillfactor = 70 is set by EnsureSchema; if you run the DDL yourself, keep it.

Compared with

	Trade
Kubernetes Lease	Better inside one cluster. Cannot span clusters at all.
etcd / ZooKeeper / Consul	Real consensus, no database SPOF — and another cluster to operate.
Redis Redlock	Contested safety argument on a store without durable, linearizable writes.
pg_advisory_lock	Simpler, and unusable behind a transaction pooler; a dropped session releases the lock silently, with no fencing token.

Pick this one when the database is already there and already the thing you cannot lose.

Contributing

See CONTRIBUTING.md. The short version: make check must pass, coverage stays at 100%, and a change to the election loop needs a chaos scenario that would have caught the bug it fixes.

License

MIT.

Documentation

Overview

Package leaderelection elects a single leader among processes that share nothing but a database.

It exists for a deployment the built-in Kubernetes lease cannot serve: replicas of one service spread across separate clusters with no network path between them. The only thing they have in common is a Postgres or MySQL instance, so that is where the election happens.

Safety

The guarantee is that at most one participant considers itself leader at any instant, and it survives network partitions, database outages, connection pooler restarts and abrupt process death. It rests on three choices:

Every timestamp comes from the database. Participants in different clusters have unrelated clocks, so a lease expiry expressed as an absolute time would mean different things to different participants. Each statement returns both the expiry and the database's current time, and only their difference -- a duration -- is ever used.

That duration is then measured against the local monotonic clock, starting from the moment the request was sent rather than when the reply arrived. The database began counting somewhere in between, so the resulting local deadline is always earlier than the true one.

A watchdog enforces that deadline independently of the renewal loop. This is what makes an unreachable database safe rather than merely inconvenient: a renewal that hangs forever cannot delay the step-down, because nothing about stepping down waits on the database. Uncertainty always resolves to "not leader" -- if this process cannot prove it still holds the lease, it stops acting as though it does, and it does so before the lease could possibly be handed to anyone else.

Fencing

Stepping down promptly bounds how long a stalled leader can act, but it cannot unsend a write already in flight. Systems that must be protected from that -- an object store, a queue, another database -- need a fencing token.

Every term is handed a Lease.Fence, through Callbacks.OnStartedLeading and Elector.Fence. Across the whole election those tokens are strictly increasing: no two terms anywhere ever share one, whether they belong to the same participant or to different ones. Attach the token to external writes and have the receiving system reject anything stamped with one older than the highest it has seen.

Example

The common shape: a callback that runs for the whole term and stops when its context is cancelled.

package main

import (
	"context"
	"fmt"

	"github.com/brualan/leaderelection"
	"github.com/brualan/leaderelection/memstore"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	done := make(chan struct{})

	elector, err := leaderelection.New(leaderelection.Config{
		Store: memstore.New(),
		Name:  "invoice-reconciler",
		Callbacks: leaderelection.Callbacks{
			OnStartedLeading: func(ctx context.Context, fence int64) {
				fmt.Println("leading with fence", fence)
				close(done)
				<-ctx.Done()
			},
		},
	})
	if err != nil {
		panic(err)
	}

	go elector.Run(ctx)
	<-done

}

Output:
leading with fence 1

Index

• Constants
• Variables
• func GenerateIdentity() (string, error)
• type Callbacks
• type Config
  • func (c Config) Validate() error
• type ConfigError
  • func (e *ConfigError) Error() string
• type Elector
  • func New(cfg Config) (*Elector, error)
  • func (e *Elector) Fence() (int64, bool)
  • func (e *Elector) IsLeader() bool
  • func (e *Elector) Run(ctx context.Context) error
  • func (e *Elector) Status() Status
• type Lease
  • func (l Lease) Expired() bool
  • func (l Lease) HeldBy(identity string) bool
  • func (l Lease) TTL() time.Duration
• type Status
• type Store

Examples

• Package
• Config.Validate
• Elector.Fence
• Elector.Status

Constants

const (
	DefaultLeaseDuration    = 15 * time.Second
	DefaultRenewInterval    = 5 * time.Second
	DefaultRetryPeriod      = 3 * time.Second
	DefaultOperationTimeout = 3 * time.Second

	// DefaultSafetyMarginFraction is the fraction of LeaseDuration reserved as
	// the safety margin when none is configured, subject to
	// MinSafetyMargin.
	DefaultSafetyMarginFraction = 10
	// MinSafetyMargin is the floor for a defaulted safety margin.
	MinSafetyMargin = time.Second
)

Defaults for Config. They are tuned for participants that reach a shared database across cluster boundaries, where round trips are tens of milliseconds rather than sub-millisecond.

Variables

var (
	// ErrNoLease is returned by Store.Inspect when no record exists for the
	// requested election. It means "nobody has ever been leader here", not
	// "the lease is free" -- although both are claimable.
	ErrNoLease = errors.New("leaderelection: no lease record")

	// ErrLeaseLost is returned by Store.Renew when the guarded update matched
	// no row. The caller is not the leader any more, and must stop acting as
	// one before doing anything else.
	ErrLeaseLost = errors.New("leaderelection: lease lost")

	// ErrLeaseExpiredLocally is recorded in [Status.LastError] when a leader
	// stepped down because it could not confirm its lease in time.
	//
	// It does not mean the lease was taken away -- nobody may have taken it,
	// and the database may be perfectly healthy on the other side of a broken
	// network path. It means this process ran out of evidence that it was
	// still the leader, which is the only situation in which it is allowed to
	// keep acting as one.
	ErrLeaseExpiredLocally = errors.New("leaderelection: lease expired before it could be renewed")

	// ErrAlreadyRunning is returned by Elector.Run when the elector is already
	// running. An Elector drives a single participant in a single election and
	// cannot be shared between goroutines by running it twice.
	ErrAlreadyRunning = errors.New("leaderelection: elector is already running")
)

Functions

func GenerateIdentity

func GenerateIdentity() (string, error)

GenerateIdentity builds an identity for this process: hostname, PID and a random suffix, joined by hyphens.

The random suffix is not decoration. Leader election is only safe if a restarted process cannot renew the lease that its own predecessor held: the predecessor may still be alive, blocked in a syscall or a stop-the-world pause, and still convinced it is the leader. Distinguishing incarnations by identity is what forces the successor to wait out the lease instead.

Identity generation fails only if the system's randomness source does, in which case the caller must not fall back to a predictable value.

Types

type Callbacks

type Callbacks struct {
	// OnStartedLeading runs for the duration of a leadership term, in its own
	// goroutine. Its context is cancelled the instant the Elector stops
	// believing it is the leader -- including when the database has merely
	// become unreachable -- so leader-only work must respect it promptly.
	//
	// fence is the token for this term. Use it to guard writes to systems that
	// cannot participate in the election themselves.
	OnStartedLeading func(ctx context.Context, fence int64)

	// OnStoppedLeading runs after a term ends, once the OnStartedLeading
	// goroutine has returned. Run does not proceed until it returns.
	OnStoppedLeading func()

	// OnNewLeader runs when a new term is observed: a different identity
	// holding the lease, or the same identity holding it under a new fence.
	// The identity may be this participant's own.
	//
	// It is advisory: use it for logging and for routing requests towards the
	// leader, never to decide whether this process is the leader.
	OnNewLeader func(identity string, fence int64)
}

Callbacks are invoked as leadership changes. All of them are optional, and all of them are called from the goroutine running Elector.Run except OnStartedLeading, which gets its own goroutine.

Callbacks must not call back into the Elector that invoked them.

type Config

type Config struct {
	// Store is the database-backed lease store. Required.
	Store Store

	// Name identifies the election. Participants contending for the same
	// leadership must use the same Name; unrelated elections use different
	// ones and may share a table. Required.
	Name string

	// Identity distinguishes this participant from every other one. When
	// empty, a value combining hostname, PID and random bytes is generated.
	//
	// Identity must be unique across restarts. Reusing a stable identity -- a
	// StatefulSet pod name, for instance -- lets a restarted process renew a
	// lease that its own stalled predecessor may still believe it holds.
	Identity string

	// Payload is opaque data published alongside the lease, readable by
	// followers through [Elector.Status]. A leader typically advertises the
	// address at which other clusters can reach it. Optional.
	Payload []byte

	// LeaseDuration is how long a lease stays valid after a successful
	// acquire or renew. It sets the worst-case failover time when a leader
	// disappears without releasing. Defaults to DefaultLeaseDuration.
	LeaseDuration time.Duration

	// RenewInterval is how often the leader extends its lease. Defaults to
	// DefaultRenewInterval.
	RenewInterval time.Duration

	// RetryPeriod is how often a follower polls for a claimable lease.
	// Defaults to DefaultRetryPeriod. Actual delays are jittered.
	RetryPeriod time.Duration

	// OperationTimeout bounds a single database call. It must be small
	// relative to LeaseDuration: a call that hangs for longer than the lease
	// would otherwise be indistinguishable from a healthy one.
	// Defaults to DefaultOperationTimeout.
	OperationTimeout time.Duration

	// SafetyMargin is how long before the lease's local expiry the Elector
	// gives up leadership. It absorbs the difference in rate between this
	// process's clock and the database's, plus the time the application needs
	// to notice its context was cancelled.
	//
	// Defaults to LeaseDuration/DefaultSafetyMarginFraction, floored at
	// MinSafetyMargin.
	SafetyMargin time.Duration

	// DisableRelease keeps the Elector from releasing its lease when Run
	// returns. Releasing lets a successor take over immediately instead of
	// waiting out LeaseDuration, so leaving this false is almost always right.
	DisableRelease bool

	// Logger receives structured logs about elections and failures. Defaults
	// to a logger that discards everything.
	Logger *slog.Logger

	// Callbacks are invoked as leadership changes. Optional.
	Callbacks Callbacks
	// contains filtered or unexported fields
}

Config describes one participant in one election.

func (Config) Validate

func (c Config) Validate() error

Validate reports whether the configuration is usable, applying no defaults. New calls it after defaulting, so callers rarely need it directly.

Example

New refuses a configuration whose timings leave a leader no room to lose a single renewal, because such a configuration flaps under exactly the conditions the election exists to survive.

package main

import (
	"fmt"
	"time"

	"github.com/brualan/leaderelection"
	"github.com/brualan/leaderelection/memstore"
)

func main() {
	err := leaderelection.Config{
		Store:            memstore.New(),
		Name:             "too-tight",
		Identity:         "a",
		LeaseDuration:    10 * time.Second,
		RenewInterval:    5 * time.Second,
		RetryPeriod:      time.Second,
		OperationTimeout: 3 * time.Second,
		SafetyMargin:     time.Second,
	}.Validate()

	fmt.Println(err)

}

Output:
leaderelection: invalid config: RenewInterval: a lost renewal needs RenewInterval+2*OperationTimeout+retry (12.25s) but only LeaseDuration-SafetyMargin (9s) is available

type ConfigError

type ConfigError struct {
	// Field names the offending configuration field.
	Field string
	// Reason explains what is wrong with it.
	Reason string
}

ConfigError reports an invalid Config. It is returned by New and by Config.Validate.

func (*ConfigError) Error

func (e *ConfigError) Error() string

type Elector

type Elector struct {
	// contains filtered or unexported fields
}

Elector runs one participant in one election. Use New to build one and Elector.Run to drive it.

func New

func New(cfg Config) (*Elector, error)

New validates cfg, fills in defaults, and returns an Elector.

func (*Elector) Fence

func (e *Elector) Fence() (int64, bool)

Fence returns the fence token of the current term, and whether there is one. Use it to stamp writes to systems outside the election.

Example

A fence token is what protects a system that cannot take part in the election. Stamp every write with it, and have the receiving side refuse anything older than the highest token it has seen.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/brualan/leaderelection"
	"github.com/brualan/leaderelection/memstore"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	elector, err := leaderelection.New(leaderelection.Config{
		Store: memstore.New(),
		Name:  "shipping",
	})
	if err != nil {
		panic(err)
	}
	go elector.Run(ctx)

	for {
		if fence, ok := elector.Fence(); ok {
			fmt.Println("would write with fence", fence)
			break
		}
		time.Sleep(time.Millisecond)
	}

}

Output:
would write with fence 1

func (*Elector) IsLeader

func (e *Elector) IsLeader() bool

IsLeader reports whether this participant may act as leader right now.

It is safe to call from any goroutine and is the authoritative answer: it re-checks the term deadline on every call rather than trusting a flag, so a descheduled watchdog goroutine cannot widen the window during which a stale leader believes itself current. When in any doubt it returns false.

func (*Elector) Run

func (e *Elector) Run(ctx context.Context) error

Run participates in the election until ctx is cancelled.

It blocks. It returns nil on a clean shutdown, which makes it well behaved inside an errgroup, and ErrAlreadyRunning if this Elector is already running. Database failures are never returned: they are transient by assumption, reported through Status.LastError, and handled by giving up leadership rather than by giving up.

On return, leadership has been fully relinquished: the OnStartedLeading context is cancelled, that callback has returned, OnStoppedLeading has run, and the lease has been released unless Config.DisableRelease is set.

func (*Elector) Status

func (e *Elector) Status() Status

Status returns a snapshot of the elector's current state.

Example

Followers can read what the leader published, which across clusters is often the only channel that exists.

package main

import (
	"context"
	"fmt"
	"time"

	"github.com/brualan/leaderelection"
	"github.com/brualan/leaderelection/memstore"
)

func main() {
	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	store := memstore.New()
	leader, err := leaderelection.New(leaderelection.Config{
		Store:    store,
		Name:     "gateway",
		Identity: "eu-west",
		Payload:  []byte("https://leader.eu-west.internal:8443"),
	})
	if err != nil {
		panic(err)
	}
	go leader.Run(ctx)

	// Wait for the term to start before the follower joins, so that the
	// follower's first read already sees a leader rather than an empty
	// election it would have to poll again for.
	for !leader.IsLeader() {
		time.Sleep(time.Millisecond)
	}

	follower, err := leaderelection.New(leaderelection.Config{
		Store:    store,
		Name:     "gateway",
		Identity: "us-east",
	})
	if err != nil {
		panic(err)
	}
	go follower.Run(ctx)

	for {
		if status := follower.Status(); status.Leader == "eu-west" {
			fmt.Printf("leader %s reachable at %s\n", status.Leader, status.LeaderPayload)
			break
		}
		time.Sleep(time.Millisecond)
	}

}

Output:
leader eu-west reachable at https://leader.eu-west.internal:8443

type Lease

type Lease struct {
	// Name identifies the election this record belongs to.
	Name string

	// Holder is the identity of the participant currently holding the lease.
	Holder string

	// Fence is a monotonically increasing token. It is incremented every time
	// a new leadership term begins -- either because the holder changed or
	// because an expired lease was reclaimed, even by its previous holder.
	//
	// Fence tokens are the only safe way to guard side effects in external
	// systems against a stalled former leader. See the package documentation.
	Fence int64

	// Payload is opaque data published by the holder, for example the address
	// at which the leader can be reached from other clusters. It may be nil.
	Payload []byte

	// AcquiredAt is when the current term began, on the database clock.
	AcquiredAt time.Time

	// RenewedAt is when the holder last extended the lease, on the database
	// clock.
	RenewedAt time.Time

	// ExpiresAt is when the lease becomes claimable by others, on the database
	// clock.
	ExpiresAt time.Time

	// Now is the database clock at the instant this record was read. It is
	// captured by the same statement that produced the other fields, so
	// comparisons against it are free of clock skew.
	Now time.Time
}

Lease is the state of a single leader-election record as reported by the database.

Every timestamp in a Lease is generated by the database server, never by the client. Participants may run in different Kubernetes clusters whose clocks are not synchronised with each other, so absolute client timestamps are meaningless for comparison. Only durations derived from a single Lease -- primarily Lease.TTL -- are safe to reason about, because both endpoints of the subtraction come from the same clock read on the same server.

func (Lease) Expired

func (l Lease) Expired() bool

Expired reports whether the lease was already claimable at the instant it was read.

func (Lease) HeldBy

func (l Lease) HeldBy(identity string) bool

HeldBy reports whether identity holds this lease and the lease is still valid.

func (Lease) TTL

func (l Lease) TTL() time.Duration

TTL reports how much longer the lease is valid according to the database clock. It returns zero for a lease that has already expired.

TTL is a duration, not a point in time, which is what makes it usable by a client whose own clock disagrees with the database's.

type Status

type Status struct {
	// IsLeader reports whether this participant may act as leader right now.
	IsLeader bool

	// Fence is the token for the current term, valid only when IsLeader.
	Fence int64

	// Identity is this participant's identity.
	Identity string

	// Leader is the identity last observed holding the lease. It is empty
	// before the first successful read, and may name a participant whose
	// lease has since expired.
	Leader string

	// LeaderFence is the fence token observed alongside Leader.
	LeaderFence int64

	// LeaderPayload is the payload published by the observed leader.
	LeaderPayload []byte

	// ObservedAt is the local time of the last successful database read.
	ObservedAt time.Time

	// ValidUntil is the local instant at which this participant must stop
	// acting as leader unless it renews first. It is zero when not leading.
	//
	// It is derived from the database's own view of the lease, translated into
	// local monotonic time, and is deliberately earlier than the database's
	// expiry.
	ValidUntil time.Time

	// LastError is the most recent failure, cleared by the next success. A
	// non-nil LastError while IsLeader is true means renewal is struggling
	// and leadership is at risk.
	LastError error
}

Status is a snapshot of what an Elector currently believes.

Everything in a Status except IsLeader is advisory and may already be stale when it is read. IsLeader is the only field with a safety guarantee behind it, and it is evaluated at the moment Elector.Status is called.

type Store

type Store interface {
	// Acquire attempts to start or continue a leadership term for identity,
	// setting the lease to expire ttl after the database's current time.
	//
	// It succeeds -- takes over -- when the record does not exist, when the
	// existing lease has expired, or when identity already holds it. Otherwise
	// the existing lease is left untouched.
	//
	// Acquire returns the resulting record in either case, so that the caller
	// can learn who the leader is without a second round trip. Losing the race
	// is not an error; callers check the result with [Lease.HeldBy]. A non-nil
	// error means the outcome is unknown.
	Acquire(ctx context.Context, name, identity string, ttl time.Duration, payload []byte) (Lease, error)

	// Renew extends a lease the caller believes it holds. The update is
	// guarded by name, identity and fence together, and applies only while the
	// lease has not yet expired on the database clock.
	//
	// Renew returns ErrLeaseLost if the guard does not match, which means the
	// caller is no longer the leader and must stop acting as one immediately.
	Renew(ctx context.Context, name, identity string, fence int64, ttl time.Duration, payload []byte) (Lease, error)

	// Release voluntarily ends a leadership term so that the next election can
	// happen without waiting for expiry. It is guarded by identity and fence.
	//
	// Releasing a lease the caller no longer holds is not an error: the
	// intended end state -- this caller is not the leader -- already holds.
	Release(ctx context.Context, name, identity string, fence int64) error

	// Inspect reads the current record without modifying it. It returns
	// ErrNoLease if no record exists for name.
	//
	// Followers use Inspect rather than Acquire so that a large fleet of
	// non-leaders does not generate a write per poll.
	Inspect(ctx context.Context, name string) (Lease, error)
}

Store is the persistence contract an Elector needs. Implementations are provided by the github.com/brualan/leaderelection/postgres and github.com/brualan/leaderelection/mysql modules.

Every method must be safe for concurrent use. Every method must derive all timestamps -- including the returned Lease.Now -- from the database server within a single statement or transaction, so that the caller can compute a skew-free TTL.

Implementations must not rely on session-scoped database state (session advisory locks, LISTEN/NOTIFY, temporary tables, SET). Doing so breaks under connection poolers such as PgBouncer in transaction pooling mode, which is a supported deployment for this library.

The storetest package provides an acceptance suite that every implementation is expected to pass.