Documentation
¶
Overview ¶
Package redlease implements lease-based leader election on Redis with fencing tokens.
One instance holds a Redis lock with a TTL and runs the caller's work while it is leader; if it cannot renew, it steps down so another instance takes over. Unlike a plain Redis lock, every leadership term is assigned a strictly increasing fencing token. Writes routed through the Elector's Fence* helpers are rejected when they carry a token below the newest elected term or applied write — electing a term fences out its predecessors immediately — so a paused or clock-skewed leader that still believes it holds the lock cannot overwrite newer state.
When you need this ¶
Fencing matters only when a stale leader's write to shared state would be harmful — for example a value that must not regress, a sequence number, or a counter. If the leader does no writes, or writes self-healing last-writer-wins state, a plain lock is enough and you do not need this package.
Correctness boundary ¶
The fence token is generated and stored in Redis. On a single Redis instance this gives a strict, monotonic guarantee — provided the fence keys survive: crash-restart durability requires AOF appendfsync always (RDB and AOF everysec lose recent writes), and eviction must not touch the keys (run noeviction or a volatile-* policy; the fence keys carry no TTL). On a replicated deployment (Sentinel or Cluster), Redis replication is asynchronous: a primary can acknowledge the acquire — and the token increment — before it reaches a replica, and a failover to that replica can lose them. In that window the monotonicity the fence relies on can be violated. For strict correctness across failover, source the fencing token from a linearizable store (etcd, ZooKeeper) instead of Redis. This package is the right tool when you run Redis single-instance, or when a brief, rare token regression on failover is acceptable.
On Redis Cluster, also wrap Config.Name in a hash tag (e.g. "{name}") so the lock, fence, and applied keys hash to the same slot; otherwise the acquire script fails with CROSSSLOT. The same constraint extends to fenced writes: each Fence* script touches the applied key and your target keys in one script, so every key written through the fence must carry the same hash tag as Name — on Cluster, fenced application state must live in the elector's slot. If that does not fit your data layout, enforce the fence at the resource yourself using the raw token (see Elector.Token).
Example ¶
This example runs a periodic job on exactly one instance at a time. Several instances call Run with the same Config.Name; only the elected leader executes the job. Each write the leader makes carries its leadership term's fencing token, so a stale leader that has not yet noticed it lost the lock cannot overwrite a newer leader's state.
package main
import (
"context"
"log"
"time"
goredis "github.com/redis/go-redis/v9"
"github.com/nijatsdev/redlease"
)
func main() {
rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})
e, err := redlease.New(rc, redlease.Config{
Name: "report-builder",
TTL: 5 * time.Second,
})
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Run blocks until ctx is cancelled. The callback runs only while this
// instance is the leader; its context is cancelled the moment leadership
// is lost. The Fencer carries this term's fencing token — use it for every
// write to shared state.
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-leaderCtx.Done():
return
case <-ticker.C:
// Record progress under the fencing token. If a newer leader
// has taken over, applied is false: this term is stale and
// must stop working.
applied, err := f.HSet(leaderCtx, "jobs:report", "status", "running")
if err != nil {
log.Printf("write failed: %v", err)
continue
}
if !applied {
log.Printf("fenced out by a newer leader; stopping")
return
}
}
}
})
}
Output:
Example (DatabaseFencing) ¶
This example fences a write to a resource other than Redis — here a SQL database. redlease enforces the fence for you only on Redis writes; for any other store it gives you the term's fencing token and you enforce it at the resource, atomically with your write. In SQL that means a conditional UPDATE that only applies when the row's stored fence is not newer than your token.
The table is assumed to hold a fence column alongside the value:
CREATE TABLE state (id text PRIMARY KEY, value text, fence bigint NOT NULL DEFAULT 0);
sql.Open needs a registered driver; a real program imports one, e.g.
import _ "github.com/jackc/pgx/v5/stdlib" // database/sql driver "pgx"
The LeaderFunc here performs one write and returns, which ends the term (see LeaderFunc); a leader with ongoing work loops until leaderCtx is done, as in the package example above.
package main
import (
"context"
"database/sql"
"log"
"time"
goredis "github.com/redis/go-redis/v9"
"github.com/nijatsdev/redlease"
)
func main() {
rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})
db, err := sql.Open("pgx", "postgres://localhost/app")
if err != nil {
log.Fatal(err)
}
defer func() { _ = db.Close() }()
e, err := redlease.New(rc, redlease.Config{Name: "report-builder", TTL: 5 * time.Second})
if err != nil {
log.Print(err)
return
}
ctx := context.Background()
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
// Take the term's token from the Fencer and enforce it yourself in SQL.
// The UPDATE applies only when our token is at least the stored fence,
// then advances it; a stale leader's lower token matches no row.
const q = `UPDATE state
SET value = $1, fence = $2
WHERE id = $3 AND fence <= $2`
res, err := db.ExecContext(leaderCtx, q, "running", f.Token(), "report")
if err != nil {
log.Printf("write failed: %v", err)
return
}
n, err := res.RowsAffected()
if err != nil {
log.Printf("rows affected: %v", err)
return
}
if n == 0 {
// No row updated: a newer leader has advanced the fence past our
// token — we are stale and must stop working. (Zero rows is also
// what a missing row yields; this example assumes the row is
// seeded. An INSERT ... ON CONFLICT upsert with the same fence
// condition covers the first write too.)
log.Printf("fenced out by a newer leader; stopping")
return
}
})
}
Output:
Index ¶
- Constants
- Variables
- type Config
- type Elector
- func (e *Elector) FenceEval(ctx context.Context, token int64, body string, writeKeys []string, args ...any) (applied bool, err error)
- func (e *Elector) FenceHSet(ctx context.Context, token int64, hashKey, field, value string) (applied bool, err error)
- func (e *Elector) FenceSet(ctx context.Context, token int64, key, value string) (applied bool, err error)
- func (e *Elector) Fencer() (f Fencer, ok bool)
- func (e *Elector) InstanceID() string
- func (e *Elector) IsLeader() bool
- func (e *Elector) Resign()
- func (e *Elector) Run(ctx context.Context, fn LeaderFunc)
- func (e *Elector) Token() (token int64, ok bool)
- type Fencer
- func (f Fencer) Eval(ctx context.Context, body string, writeKeys []string, args ...any) (applied bool, err error)
- func (f Fencer) HSet(ctx context.Context, hashKey, field, value string) (applied bool, err error)
- func (f Fencer) Set(ctx context.Context, key, value string) (applied bool, err error)
- func (f Fencer) Token() int64
- type LeaderFunc
- type Observer
- type Redis
Examples ¶
Constants ¶
const DefaultTTL = 5 * time.Second
DefaultTTL is the lock TTL used when Config.TTL is unset. Override via Config.
Variables ¶
var ErrTooManyEvalBodies = errors.New("redlease: too many distinct FenceEval bodies; keep bodies constant and pass variable data through writeKeys and args")
ErrTooManyEvalBodies is returned (wrapped) by Elector.FenceEval once more than maxEvalScripts distinct Lua bodies have been used. It signals a caller bug — variable data interpolated into the body — not a transient Redis error: check for it with errors.Is, fix the body to be constant (pass variable data through writeKeys and args), and do not retry.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Name identifies the lock. All instances contending for the same leadership
// must use the same Name; different Names are independent locks.
//
// The lock, fence, and applied keys all derive from Name, and a single Lua
// script touches more than one of them. On Redis Cluster they must therefore
// hash to the same slot: wrap Name in a hash tag, e.g. "{report-builder}",
// so all derived keys share it. Without one they scatter across slots and the
// acquire script fails with CROSSSLOT. Keys written through the Fence*
// helpers must carry the same hash tag too; see the package documentation.
Name string
// TTL is how long the lock lives without renewal. A leader that cannot renew
// within TTL loses leadership. Shorter TTL means faster failover but less
// tolerance for pauses. Must be at least 100ms so a renewal can complete
// before expiry. Defaults to DefaultTTL.
//
// Redis applies the TTL at millisecond granularity (PX), so a fractional
// remainder is truncated.
TTL time.Duration
// RenewInterval is how often the leader renews the lock. Must be at most
// TTL/2, so at least two renewal attempts fit before expiry, and at least
// 1ms (it doubles as each renewal's timeout). Note that at exactly TTL/2
// the retry after a dropped renewal lands right at expiry; keep it strictly
// below TTL/2 (the TTL/3 default) for real retry headroom.
RenewInterval time.Duration
// AcquireInterval is how often a follower retries acquiring the lock. Must be
// less than TTL, so a follower polls at least once per lock lifetime, and at
// least 1ms, like RenewInterval. Defaults to TTL/2.
//
// The interval governs the ordinary case of losing the race to another
// instance. When the acquire round trip errors instead (Redis unreachable),
// the retry delay doubles per consecutive error, capped at TTL, and resets
// once Redis answers again.
AcquireInterval time.Duration
// InstanceID uniquely identifies this instance; it is the lock's value and
// guards renewal and release so an instance never affects another's lock. If
// empty, a hostname + random suffix is generated.
//
// The ID must be unique among live processes, not merely stable: a lock
// found holding this instance's own ID is treated as its leftover and taken
// over, so two live replicas sharing an ID seize the lock from each other
// and both run as leader — permanently and silently. Derive a fixed ID from
// something unique per replica (pod or host name), never a value shared
// across replicas like the service name.
InstanceID string
// Observer receives lifecycle events. All fields are optional; the library
// emits no output of its own, so callers wire these to their own logger,
// metrics, or tracing as they see fit.
Observer Observer
}
Config configures an Elector. Only Name is required (the client is passed separately to New); zero-valued timing fields fall back to the Default* constants.
type Elector ¶
type Elector struct {
// contains filtered or unexported fields
}
Elector runs leader election for a single lock and mints fencing tokens.
func New ¶
New returns an Elector from cfg. It returns an error if required fields are missing or timing parameters are inconsistent: TTL below 100ms, RenewInterval greater than TTL/2, AcquireInterval not less than TTL, or either interval below 1ms.
func (*Elector) FenceEval ¶
func (e *Elector) FenceEval(ctx context.Context, token int64, body string, writeKeys []string, args ...any) (applied bool, err error)
FenceEval fences an arbitrary Redis write supplied as a Lua body, for writes the typed helpers do not cover (ZADD, XADD, multi-key updates, and so on). The body runs atomically only when token is current; the fence returns 1 when applied and 0 when fenced out.
The body must not contain its own return statement: FenceEval appends the fence's epilogue, which advances the token high-water mark and returns 1. A return inside body skips that epilogue, so the result reports applied == false and the mark does not advance even on a write that ran. Just perform the write and let FenceEval supply the return.
Within body, the protected token check is already done. Address your own keys and arguments starting at index 2 — KEYS[1] and ARGV[1] are reserved for the fence (the high-water key and the token). Pass your keys in writeKeys and your arguments in args; they appear as KEYS[2..] and ARGV[2..] in that order. On Redis Cluster, every key in writeKeys must share the elector's hash slot; see Elector.FenceHSet.
Each distinct body compiles once and is cached on the Elector for EVALSHA reuse, capped at 256 bodies; past the cap FenceEval returns an error wrapping ErrTooManyEvalBodies. Keep bodies constant and pass variable data through writeKeys and args — interpolating values into the body creates a new cache entry (and a new script for Redis) per call.
Example — fence a ZADD:
applied, err := e.FenceEval(ctx, token,
"redis.call('zadd', KEYS[2], ARGV[2], ARGV[3])",
[]string{"myset"}, "1.0", "member")
func (*Elector) FenceHSet ¶
func (e *Elector) FenceHSet(ctx context.Context, token int64, hashKey, field, value string) (applied bool, err error)
FenceHSet performs a fenced HSET of field=value into hashKey: the write is applied only if token is at least the fence high-water mark — the highest token minted by an election or carried by an applied fenced write. It returns true when applied and false when token is stale — a newer term has been elected or has written — so the caller can stop emitting derived events. token must be the value passed to the LeaderFunc for the current term. A non-nil error means the write could not be attempted.
On Redis Cluster, hashKey must hash to the same slot as the elector's keys: the check and the write run in one script against the applied key and hashKey together, so give hashKey the same hash tag as Config.Name or the call fails with CROSSSLOT. The same holds for every Fence* helper's keys.
func (*Elector) FenceSet ¶
func (e *Elector) FenceSet(ctx context.Context, token int64, key, value string) (applied bool, err error)
FenceSet performs a fenced SET of key=value, applied only if token is current. Semantics match Elector.FenceHSet.
func (*Elector) Fencer ¶ added in v0.3.0
Fencer returns a Fencer bound to the current leadership term and true while this instance is the leader, or an inert Fencer and false otherwise — inert meaning it carries token 0, the "not leader" sentinel every fenced write rejects, so even a caller that ignores ok cannot slip a write through. It is the token-driven-style counterpart to the Fencer a LeaderFunc receives: code outside the callback can take a Fencer and pass it down to its writers.
Like Elector.Token, the returned Fencer is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fencer's methods.
func (*Elector) InstanceID ¶
InstanceID returns this elector's instance identity.
func (*Elector) IsLeader ¶
IsLeader reports whether this instance currently holds leadership. It is advisory only: leadership can be lost the instant after it returns, so never gate a correctness-sensitive write on it. For writes, carry the token from Elector.Token through a Fence* helper, which rejects a stale token at write time.
func (*Elector) Resign ¶ added in v1.0.0
func (e *Elector) Resign()
Resign voluntarily ends the current leadership term, if any: the LeaderFunc's context is cancelled, the lock is released, and Run re-contends after AcquireInterval — so this instance may well win again unless another takes the lock first. It is a no-op when this instance is not leading, and safe to call from any goroutine. It is the step-down lever for the token-driven style (Run with a nil LeaderFunc), which otherwise could stop leading only by cancelling Run's context entirely.
func (*Elector) Run ¶
func (e *Elector) Run(ctx context.Context, fn LeaderFunc)
Run contends for leadership until ctx is cancelled. Each time this instance wins, it invokes fn (a LeaderFunc) with a Fencer for the term, then steps down and re-contends when the term ends — leadership lost, fn returned, Elector.Resign called, or ctx cancelled. Run blocks until ctx is cancelled and fn (if running) has returned; on the final shutdown step-down the OnSteppedDown observer still fires.
fn may be nil. Then Run just keeps this instance elected — acquiring, renewing, and releasing the lock — while the caller does its leader work from another goroutine, gated on Elector.Token. This is the token-driven style; the callback style and this one are interchangeable, pick whichever fits.
Run must not be called more than once concurrently on the same Elector; the two calls would corrupt shared leadership state. A concurrent call panics. Use one Elector per Run. Sequential calls (a new Run after a previous one returned) are fine.
func (*Elector) Token ¶
Token returns the current leadership term's fencing token and true while this instance is the leader, or 0 and false otherwise. It is safe to call from any goroutine, so code outside the LeaderFunc can perform fenced writes:
if token, ok := e.Token(); ok {
e.FenceHSet(ctx, token, "state", "key", "value")
}
The returned token is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fence* helpers — so unlike Elector.IsLeader there is no time-of-check-to-time-of-use hazard in acting on it.
type Fencer ¶ added in v0.3.0
type Fencer struct {
// contains filtered or unexported fields
}
Fencer binds a leadership term's fencing token to the Elector that minted it, so a single value carries everything a fenced write needs. It is what a LeaderFunc receives, and what you pass down into the code that writes shared state.
A Fencer is a small value; copy it freely. Its methods are the bound equivalents of the Elector's Fence* helpers: each applies its write only when the bound token is at least the fence high-water mark, and returns applied == false when the token is stale (a newer term has been elected or has written), so the caller can stop. A Fencer is valid only for its term: its writes are fenced out once a successor is elected. Between a voluntary step-down and the next election, writes under the old token may still apply (there is no newer state to protect); treat the term context's cancellation, not a failing write, as the signal to stop working.
func NewFencer ¶ added in v0.3.0
NewFencer binds an explicit token to e, for callers that already hold a token — for example one persisted from a prior leadership term, or supplied in a test. Most code should instead take the Fencer a LeaderFunc receives or call Elector.Fencer. The token is not validated against the elector's current term (the fence is enforced at write time), but it must originate from this package's electors or stay within their magnitude: the fence comparison is exact only up to 2^53, and stamping an arbitrarily large token would permanently fence out every legitimate writer.
e must be non-nil — the Fencer's methods call through it. Likewise a hand-declared zero Fencer{} panics on use; obtain Fencers from the elector.
func (Fencer) Eval ¶ added in v0.3.0
func (f Fencer) Eval(ctx context.Context, body string, writeKeys []string, args ...any) (applied bool, err error)
Eval fences an arbitrary Lua write. See Elector.FenceEval.
func (Fencer) HSet ¶ added in v0.3.0
HSet performs a fenced HSET of field=value into hashKey. See Elector.FenceHSet for the full semantics.
func (Fencer) Set ¶ added in v0.3.0
Set performs a fenced SET of key=value. See Elector.FenceSet.
type LeaderFunc ¶
LeaderFunc is the work run while an instance is leader. It receives a context cancelled when leadership is lost or Run's context is cancelled, and a Fencer bound to this leadership term — use it for every fenced write, or pass it down to the code that writes shared state. LeaderFunc must return promptly once its context is cancelled; Run does not release the lock or re-contend until it does.
Returning also ends the term: holding the lock with no work running would only block other instances, so Run releases it and re-contends. A LeaderFunc that wants to stay leader must block until its context is cancelled.
A panic in a LeaderFunc is not recovered: it crashes the process like any other goroutine panic, and the lock is left to expire at its TTL rather than being released, so failover waits out the TTL. Recover inside your LeaderFunc if you want a panicking term to step down gracefully instead.
type Observer ¶
type Observer struct {
// OnElected fires when this instance wins a leadership term, with that
// term's fencing token. It fires before the LeaderFunc is invoked.
OnElected func(token int64)
// OnSteppedDown fires when this instance loses or relinquishes leadership,
// after the LeaderFunc has returned.
OnSteppedDown func()
// OnFollower fires when this instance is, for now, a follower: an acquire
// attempt found the lock held by another instance. It fires once per
// transition into the follower role — on the first lost attempt, and again
// only after an intervening leadership term — not on every retry. A consumer
// can use it to learn its initial role at startup without waiting to win.
OnFollower func()
// OnError fires when a Redis round trip fails during acquire, renewal, or
// release. The elector has already handled the failure — retried, backed
// off, or stepped down — so the callback exists purely for visibility: wire
// it to your logger or metrics to see trouble before it costs leadership.
// It is not called for the context cancellation that ends Run.
//
// Deterministic failures (a missing Cluster hash tag surfacing as
// CROSSSLOT, auth errors) are retried forever at the backoff cadence, and
// OnError is their only signal — wire it up before first deploying.
OnError func(err error)
}
Observer is a set of optional callbacks invoked on leadership transitions. A nil field is simply not called. Callbacks run on the Run goroutine and must not block — a slow OnElected eats TTL budget before the first renewal and can cost the term. A panicking callback is not recovered (see LeaderFunc); OnElected in particular fires while the lock is held, so its panic leaves the lock to expire at the TTL.
Role transitions are the primary events, mirroring the design of k8s.io/client-go/tools/leaderelection: transient Redis errors during acquire, renewal, and release are handled internally — retried, backed off, or stepped down from — and the consequence the caller cares about, losing leadership, surfaces through OnSteppedDown. OnError additionally exposes those handled errors for logging and metrics; it never requires action.
type Redis ¶
Redis is the subset of the go-redis client this package needs: the script runner used by all lock and fence operations. *goredis.Client and *goredis.ClusterClient both satisfy it, as does any compatible wrapper.
Configure the client with sane I/O timeouts: the elector bounds every round trip itself, but a call abandoned past its bound stays blocked inside a timeout-less client, pinning a pooled connection — during a long server hang those accumulate and can starve a shared client.