README
¶
redlease
Lease-based leader election on Redis, with fencing tokens.
go get github.com/nijatsdev/redlease
Pre-1.0: the API may change between
v0.xreleases.
redlease elects one long-lived leader among instances and keeps it elected — for running a singleton background job, a scheduler, a cron-like task, or the single writer in a one-writer-many-readers system. It manages the whole leadership lifecycle: acquire, renew, step down, release, and fail over.
It is not a general-purpose mutex. For short-lived mutual exclusion around a critical section, use redsync or redislock instead. The litmus test:
Is the lock held for the duration of a role or the duration of an operation? Role (be the leader, own the schedule, be the one writer) → redlease. Operation (guard this critical section, update this counter safely) → a mutex.
One instance holds a Redis lock with a TTL and runs your work while it is leader. If it cannot renew the lock, it steps down so another instance takes over. That much is what every Redis-lock library does. What redlease adds is the part most of them omit:
Every leadership term is assigned a strictly increasing fencing token, and writes routed through the Fence* helpers reject any token older than the latest applied. A paused, GC-stalled, or clock-skewed leader that still believes it holds the lock cannot overwrite newer state — its writes are refused at Redis.
Why fencing
Lock-based leader election has an unavoidable window. The lock has a TTL; if the leader pauses (GC, CPU starvation) or is partitioned, the lock can expire and a second instance can be elected while the first still thinks it is leader. For a few seconds, two leaders exist. This is inherent to any lease-based lock — shortening the TTL only shrinks the window, it never closes it.
Fencing makes that window safe. Each term gets a token; the protected resource only accepts writes whose token is at least the highest it has already seen. The stale leader's writes carry an old token and are rejected. This is the mitigation Martin Kleppmann describes in How to do distributed locking — and the reason a plain Redis lock is not safe for correctness-sensitive writes.
Do you actually need it?
Fencing matters only when a stale leader's write to shared state would be harmful:
| Your leader… | Need fencing? |
|---|---|
| writes a value that must not regress (sequence number, counter, monotonic state) | Yes |
| does no writes (runs a cron, sends notifications) | No — at most you want idempotency/dedup |
| writes self-healing last-writer-wins state that the next correct write repairs | No — a plain lock is enough |
If you are in the "No" rows, you do not need this library; a simpler lock will do. redlease is for the first row.
Usage
rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})
e, err := redlease.New(rc, redlease.Config{Name: "report-builder", TTL: 5 * time.Second})
if err != nil {
log.Fatal(err)
}
// Run blocks until ctx is cancelled. The callback runs only while leader; its
// context is cancelled the instant leadership is lost. The Fencer carries this
// term's fencing token — use it for every fenced write, or pass it down to the
// code that writes shared state.
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
applied, err := f.HSet(leaderCtx, "jobs:report", "status", "running")
if err != nil {
// Redis error.
}
if !applied {
// A newer leader has taken over; this term is stale. Stop working.
return
}
})
All instances that should contend for the same leadership must share the same Config.Name. The Fencer your work function receives is bound to the term's fencing token — pass it wherever shared state is written, and the token travels with it.
Fenced writes
A Fencer binds the term's token to the elector that minted it, so you carry one value instead of threading a token and a client separately. Each fenced write checks the token and performs the write atomically in a single Lua script, so there is no window in which the token could go stale between the check and the write. All methods share one high-water mark, so a token advanced by any of them fences every later, lower-token write:
f.HSet(ctx, hashKey, field, value) // fenced HSET
f.Set(ctx, key, value) // fenced SET
// Escape hatch for any other Redis write (ZADD, XADD, multi-key, ...).
// KEYS[1]/ARGV[1] are reserved for the fence; address yours from index 2.
f.Eval(ctx,
"redis.call('zadd', KEYS[2], ARGV[2], ARGV[3])",
[]string{"board"}, "100", "alice")
All three return (applied bool, err error): applied == false means your token is stale — a newer leader has taken over — and you should stop writing. f.Token() returns the raw token for fencing a resource that isn't Redis (see below).
The same writes are available on the elector itself — e.FenceHSet(ctx, token, ...), e.FenceSet, e.FenceEval — taking the token explicitly. Use those in the token-driven style, where you hold a token from e.Token() rather than a Fencer from the callback.
Observability
redlease emits no logs of its own — a library should not impose a log format, level, or destination on its caller. Instead it exposes leadership transitions through an optional Observer; wire them to your own logger, metrics, or tracing. Both fields are optional and a nil one is simply not called.
e, _ := redlease.New(rc, redlease.Config{
Name: "report-builder",
Observer: redlease.Observer{
OnElected: func(token int64) { slog.Info("leader elected", "fence", token) },
OnSteppedDown: func() { slog.Info("leader stepped down") },
OnFollower: func() { slog.Info("running as follower") },
},
})
OnFollower fires when an acquire attempt finds the lock held by another instance — once per transition into the follower role, not on every retry. It lets a follower learn its initial role at startup without waiting to win.
Only role transitions are reported, following the design of client-go's leaderelection. Transient Redis errors during acquire or renewal are handled internally; the consequence the caller cares about — losing leadership — surfaces through OnSteppedDown. Monitor Redis health through your Redis client, not through the Observer. Callbacks run on the Run goroutine and must not block.
Checking leadership outside the callback
The Fencer reaches your LeaderFunc directly, but sometimes another goroutine — an HTTP handler, say — needs to act as the leader too. Fencer(), Token(), and IsLeader() expose the current state from any goroutine:
if f, ok := e.Fencer(); ok {
// We are the leader. The fenced write is safe even if leadership changes
// right now: a stale token is rejected at write time.
f.HSet(ctx, "state", "key", "value")
}
Prefer Token() for anything that writes. IsLeader() exists for display/metrics, but it is advisory — leadership can be lost the instant after it returns, so never gate a correctness-sensitive write on if e.IsLeader() { write() }. That is the split-brain race fencing exists to prevent; carry the token through a Fence* helper instead.
If all your leader work happens this way — outside the callback — pass nil for the LeaderFunc. Run then just keeps the instance elected (acquire, renew, release) while your other goroutines act via Token():
go e.Run(ctx, nil) // hold leadership; do the work elsewhere via e.Token()
The callback style and this token-driven style are interchangeable — pick whichever fits your app.
How it works
- Acquire — a single Lua script does
SET name:leader <id> NX PX <ttl-ms>and, on success,INCR name:fenceto mint the token. Doing both atomically guarantees every term's token is strictly greater than any prior term's. The TTL is set in milliseconds so sub-second values are honored exactly. - Hold — the leader renews the lock on
RenewIntervalvia an ownership-checked script (it only extends a lock whose value is still its own id). A renewal that returns 0 means the lock was lost; a transient Redis error is tolerated untilTTLwould have lapsed, then the leader steps down. - Release — on graceful step-down the leader deletes the lock (ownership-checked, so it never deletes a successor's lock), letting the next instance take over without waiting for the TTL.
- Fence — each fenced write runs a Lua script that stores the highest applied token in
name:fence:appliedand rejects any write carrying a lower token, performing the write only when the token is current.
Fencing writes that don't go to Redis
Fencing is not a Redis concept — it applies to any shared resource a stale leader could corrupt (a Postgres row, an S3 object, a Kafka topic). The catch is that the fence must be enforced at the resource itself, atomically with the write, because that is the only place the check and the write can happen together.
This package enforces the fence for Redis writes, because Redis is the resource it can reach into (via Lua). If your leader writes elsewhere, redlease still gives you the universal half — the monotonic token — but you must enforce it at your resource. For example, in Postgres:
UPDATE state SET value = $1, fence = $2 WHERE key = $3 AND fence <= $2;
-- rows affected == 0 -> your token was stale; you were fenced out
So: use the Fence* helpers when you write to Redis; use the token with a conditional write (a WHERE fence <= token, a compare-and-swap, an If-Match precondition) when you write anywhere else.
Correctness boundary
Read this before using it for anything that matters.
The fencing token is generated and stored in Redis. On a single Redis instance, this gives a strict, monotonic guarantee: tokens never go backward, and the fence is sound.
On a replicated Redis deployment (Sentinel or Cluster), Redis replication is asynchronous. A primary can acknowledge the acquire — and the token INCR — before it has propagated to a replica, and a failover to that replica can lose it. In that window the monotonicity the fence depends on can be violated, and two leaders could in principle obtain non-ordered tokens. This is the same limitation that affects every Redis-based lock, fencing or not.
So redlease is the right tool when:
- you run Redis single-instance, or
- a brief, rare token regression on Redis failover is acceptable for your workload.
If you need a fencing guarantee that survives failover, source the token from a linearizable store — etcd or ZooKeeper — and apply it against your resource. redlease deliberately does not pretend Redis can provide that.
Election, not a mutex
redlease and a distributed mutex (redsync, redislock) solve different problems with different shapes. Reaching for the wrong one is the classic split-brain footgun — holding a mutex across long-running work with no renewal lets the lock expire mid-work.
| Mutex (redsync, redislock) | Election (redlease) | |
|---|---|---|
| Mental model | grab lock → short critical section → release | be leader → run long-lived work → step down |
| You call | Lock() … Unlock() manually |
Run(ctx, fn) — lifecycle managed for you |
| Lock held for | a brief operation | the whole leadership term (often hours) |
| Renewal | you add it yourself, or the lock expires | built in |
| Loser of the race | blocks / retries to acquire now | becomes a follower, re-contends in the background |
| Fencing | none | core feature |
If your lock guards an operation, use a mutex. If it confers a role, use redlease.
When to use something else
| You want… | Use |
|---|---|
| election + fencing, Redis you already run, single-instance or tolerant of rare failover regressions | redlease |
| a plain distributed mutex for short critical sections | go-redsync/redsync |
| consensus-grade election and you run on Kubernetes | k8s.io/client-go/tools/leaderelection (etcd-backed Lease) |
| consensus-grade election off Kubernetes | etcd concurrency.Election |
| leadership as part of your own replicated state machine | a Raft library (hashicorp/raft, etcd/raft) |
redlease fills one specific gap: the Redis-lock libraries that don't fence, for users whose leader writes shared state where a stale write would do harm.
License
MIT
Documentation
¶
Overview ¶
Package redlease implements lease-based leader election on Redis with fencing tokens.
One instance holds a Redis lock with a TTL and runs the caller's work while it is leader; if it cannot renew, it steps down so another instance takes over. Unlike a plain Redis lock, every leadership term is assigned a strictly increasing fencing token. Writes routed through the Elector's Fence* helpers are rejected when they carry a token older than the latest applied, so a paused or clock-skewed leader that still believes it holds the lock cannot overwrite newer state.
When you need this ¶
Fencing matters only when a stale leader's write to shared state would be harmful — for example a value that must not regress, a sequence number, or a counter. If the leader does no writes, or writes self-healing last-writer-wins state, a plain lock is enough and you do not need this package.
Correctness boundary ¶
The fence token is generated and stored in Redis. On a single Redis instance this gives a strict, monotonic guarantee. On a replicated deployment (Sentinel or Cluster), Redis replication is asynchronous: a primary can acknowledge the acquire — and the token increment — before it reaches a replica, and a failover to that replica can lose them. In that window the monotonicity the fence relies on can be violated. For strict correctness across failover, source the fencing token from a linearizable store (etcd, ZooKeeper) instead of Redis. This package is the right tool when you run Redis single-instance, or when a brief, rare token regression on failover is acceptable.
Example ¶
This example runs a periodic job on exactly one instance at a time. Several instances call Run with the same Config.Name; only the elected leader executes the job. Each write the leader makes carries its leadership term's fencing token, so a stale leader that has not yet noticed it lost the lock cannot overwrite a newer leader's state.
package main
import (
"context"
"log"
"time"
goredis "github.com/redis/go-redis/v9"
"github.com/nijatsdev/redlease"
)
func main() {
rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})
e, err := redlease.New(rc, redlease.Config{
Name: "report-builder",
TTL: 5 * time.Second,
})
if err != nil {
log.Fatal(err)
}
ctx, cancel := context.WithCancel(context.Background())
defer cancel()
// Run blocks until ctx is cancelled. The callback runs only while this
// instance is the leader; its context is cancelled the moment leadership
// is lost. The Fencer carries this term's fencing token — use it for every
// write to shared state.
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
ticker := time.NewTicker(time.Second)
defer ticker.Stop()
for {
select {
case <-leaderCtx.Done():
return
case <-ticker.C:
// Record progress under the fencing token. If a newer leader
// has taken over, applied is false: this term is stale and
// must stop working.
applied, err := f.HSet(leaderCtx, "jobs:report", "status", "running")
if err != nil {
log.Printf("write failed: %v", err)
continue
}
if !applied {
log.Printf("fenced out by a newer leader; stopping")
return
}
}
}
})
}
Output:
Example (DatabaseFencing) ¶
This example fences a write to a resource other than Redis — here a SQL database. redlease enforces the fence for you only on Redis writes; for any other store it gives you the term's fencing token and you enforce it at the resource, atomically with your write. In SQL that means a conditional UPDATE that only applies when the row's stored fence is not newer than your token.
The table is assumed to hold a fence column alongside the value:
CREATE TABLE state (id text PRIMARY KEY, value text, fence bigint NOT NULL DEFAULT 0);
package main
import (
"context"
"database/sql"
"log"
"time"
goredis "github.com/redis/go-redis/v9"
"github.com/nijatsdev/redlease"
)
func main() {
rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})
db, err := sql.Open("postgres", "postgres://localhost/app")
if err != nil {
log.Fatal(err)
}
defer func() { _ = db.Close() }()
e, err := redlease.New(rc, redlease.Config{Name: "report-builder", TTL: 5 * time.Second})
if err != nil {
log.Print(err)
return
}
ctx := context.Background()
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
// Take the term's token from the Fencer and enforce it yourself in SQL.
// The UPDATE applies only when our token is at least the stored fence,
// then advances it; a stale leader's lower token matches no row.
const q = `UPDATE state
SET value = $1, fence = $2
WHERE id = $3 AND fence <= $2`
res, err := db.ExecContext(leaderCtx, q, "running", f.Token(), "report")
if err != nil {
log.Printf("write failed: %v", err)
return
}
if n, _ := res.RowsAffected(); n == 0 {
// No row updated: a newer leader has advanced the fence past our
// token. We are stale and must stop working.
log.Printf("fenced out by a newer leader; stopping")
return
}
})
}
Output:
Index ¶
- Constants
- type Config
- type Elector
- func (e *Elector) FenceEval(ctx context.Context, token int64, body string, writeKeys []string, args ...any) (applied bool, err error)
- func (e *Elector) FenceHSet(ctx context.Context, token int64, hashKey, field, value string) (applied bool, err error)
- func (e *Elector) FenceSet(ctx context.Context, token int64, key, value string) (applied bool, err error)
- func (e *Elector) Fencer() (f Fencer, ok bool)
- func (e *Elector) InstanceID() string
- func (e *Elector) IsLeader() bool
- func (e *Elector) Run(ctx context.Context, fn LeaderFunc)
- func (e *Elector) Token() (token int64, ok bool)
- type Fencer
- func (f Fencer) Eval(ctx context.Context, body string, writeKeys []string, args ...any) (applied bool, err error)
- func (f Fencer) HSet(ctx context.Context, hashKey, field, value string) (applied bool, err error)
- func (f Fencer) Set(ctx context.Context, key, value string) (applied bool, err error)
- func (f Fencer) Token() int64
- type LeaderFunc
- type Observer
- type Redis
Examples ¶
Constants ¶
const ( DefaultTTL = 5 * time.Second DefaultRenewInterval = 2 * time.Second DefaultAcquireInterval = 2 * time.Second )
Default timing parameters. Override via Config.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Config ¶
type Config struct {
// Name identifies the lock. All instances contending for the same leadership
// must use the same Name; different Names are independent locks.
Name string
// TTL is how long the lock lives without renewal. A leader that cannot renew
// within TTL loses leadership. Shorter TTL means faster failover but less
// tolerance for pauses. Defaults to DefaultTTL.
TTL time.Duration
// RenewInterval is how often the leader renews the lock. Must be well below
// TTL. Defaults to DefaultRenewInterval.
RenewInterval time.Duration
// AcquireInterval is how often a follower retries acquiring the lock.
// Defaults to DefaultAcquireInterval.
AcquireInterval time.Duration
// InstanceID uniquely identifies this instance; it is the lock's value and
// guards renewal and release so an instance never affects another's lock. If
// empty, a hostname + random suffix is generated.
InstanceID string
// Observer receives lifecycle events. All fields are optional; the library
// emits no output of its own, so callers wire these to their own logger,
// metrics, or tracing as they see fit.
Observer Observer
}
Config configures an Elector. Only Name is required (the client is passed separately to New); zero-valued timing fields fall back to the Default* constants.
type Elector ¶
type Elector struct {
// contains filtered or unexported fields
}
Elector runs leader election for a single lock and mints fencing tokens.
func New ¶
New returns an Elector from cfg. It returns an error if required fields are missing or timing parameters are inconsistent (e.g. RenewInterval >= TTL).
func (*Elector) FenceEval ¶
func (e *Elector) FenceEval(ctx context.Context, token int64, body string, writeKeys []string, args ...any) (applied bool, err error)
FenceEval fences an arbitrary Redis write supplied as a Lua body, for writes the typed helpers do not cover (ZADD, XADD, multi-key updates, and so on). The body runs atomically only when token is current; the fence returns 1 when applied and 0 when fenced out.
The body must not contain its own return statement: FenceEval appends the fence's "return 1", so a return inside body shadows it and makes the result report applied == false even on a write that ran. Just perform the write and let FenceEval supply the return.
Within body, the protected token check is already done. Address your own keys and arguments starting at index 2 — KEYS[1] and ARGV[1] are reserved for the fence (the high-water key and the token). Pass your keys in writeKeys and your arguments in args; they appear as KEYS[2..] and ARGV[2..] in that order.
Example — fence a ZADD:
applied, err := e.FenceEval(ctx, token,
"redis.call('zadd', KEYS[2], ARGV[2], ARGV[3])",
[]string{"myset"}, "1.0", "member")
func (*Elector) FenceHSet ¶
func (e *Elector) FenceHSet(ctx context.Context, token int64, hashKey, field, value string) (applied bool, err error)
FenceHSet performs a fenced HSET of field=value into hashKey: the write is applied only if token is at least the highest token already applied through this elector. It returns true when applied and false when token is stale — a newer leadership term has since written — so the caller can stop emitting derived events. token must be the value passed to the LeaderFunc for the current term. A non-nil error means the write could not be attempted.
func (*Elector) FenceSet ¶
func (e *Elector) FenceSet(ctx context.Context, token int64, key, value string) (applied bool, err error)
FenceSet performs a fenced SET of key=value, applied only if token is current. Semantics match Elector.FenceHSet.
func (*Elector) Fencer ¶ added in v0.3.0
Fencer returns a Fencer bound to the current leadership term and true while this instance is the leader, or a zero Fencer and false otherwise. It is the token-driven-style counterpart to the Fencer a LeaderFunc receives: code outside the callback can take a Fencer and pass it down to its writers.
Like Elector.Token, the returned Fencer is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fencer's methods.
func (*Elector) InstanceID ¶
InstanceID returns this elector's instance identity.
func (*Elector) IsLeader ¶
IsLeader reports whether this instance currently holds leadership. It is advisory only: leadership can be lost the instant after it returns, so never gate a correctness-sensitive write on it. For writes, carry the token from Elector.Token through a Fence* helper, which rejects a stale token at write time.
func (*Elector) Run ¶
func (e *Elector) Run(ctx context.Context, fn LeaderFunc)
Run contends for leadership until ctx is cancelled. Each time this instance wins, it invokes fn (a LeaderFunc) with a Fencer for the term, then steps down and re-contends when leadership is lost. Run blocks until ctx is cancelled and fn (if running) has returned; on the final shutdown step-down the OnSteppedDown observer still fires.
fn may be nil. Then Run just keeps this instance elected — acquiring, renewing, and releasing the lock — while the caller does its leader work from another goroutine, gated on Elector.Token. This is the token-driven style; the callback style and this one are interchangeable, pick whichever fits.
Run must not be called more than once concurrently on the same Elector; the two calls would corrupt shared leadership state. Use one Elector per Run.
func (*Elector) Token ¶
Token returns the current leadership term's fencing token and true while this instance is the leader, or 0 and false otherwise. It is safe to call from any goroutine, so code outside the LeaderFunc can perform fenced writes:
if token, ok := e.Token(); ok {
e.FenceHSet(ctx, token, "state", "key", "value")
}
The returned token is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fence* helpers — so unlike Elector.IsLeader there is no time-of-check-to-time-of-use hazard in acting on it.
type Fencer ¶ added in v0.3.0
type Fencer struct {
// contains filtered or unexported fields
}
Fencer binds a leadership term's fencing token to the Elector that minted it, so a single value carries everything a fenced write needs. It is what a LeaderFunc receives, and what you pass down into the code that writes shared state.
A Fencer is a small value; copy it freely. Its methods are the bound equivalents of the Elector's Fence* helpers: each applies its write only when the bound token is at least the highest already applied, and returns applied == false when the token is stale (a newer term has written), so the caller can stop. A Fencer is valid only for its term; once leadership is lost, its writes are fenced out like any other stale token.
func NewFencer ¶ added in v0.3.0
NewFencer binds an explicit token to e, for callers that already hold a token — for example one persisted from a prior leadership term, or supplied in a test. Most code should instead take the Fencer a LeaderFunc receives or call Elector.Fencer; reach for this only when the token comes from outside the elector. The token is not validated against the elector's current term: the fence is still enforced at write time, so a stale token is rejected there.
func (Fencer) Eval ¶ added in v0.3.0
func (f Fencer) Eval(ctx context.Context, body string, writeKeys []string, args ...any) (applied bool, err error)
Eval fences an arbitrary Lua write. See Elector.FenceEval.
func (Fencer) HSet ¶ added in v0.3.0
HSet performs a fenced HSET of field=value into hashKey. See Elector.FenceHSet for the full semantics.
func (Fencer) Set ¶ added in v0.3.0
Set performs a fenced SET of key=value. See Elector.FenceSet.
type LeaderFunc ¶
LeaderFunc is the work run while an instance is leader. It receives a context cancelled when leadership is lost or Run's context is cancelled, and a Fencer bound to this leadership term — use it for every fenced write, or pass it down to the code that writes shared state. LeaderFunc must return promptly once its context is cancelled; Run does not release the lock or re-contend until it does.
type Observer ¶
type Observer struct {
// OnElected fires when this instance wins a leadership term, with that
// term's fencing token. It fires before the LeaderFunc is invoked.
OnElected func(token int64)
// OnSteppedDown fires when this instance loses or relinquishes leadership,
// after the LeaderFunc has returned.
OnSteppedDown func()
// OnFollower fires when this instance is, for now, a follower: an acquire
// attempt found the lock held by another instance. It fires once per
// transition into the follower role — on the first lost attempt, and again
// only after an intervening leadership term — not on every retry. A consumer
// can use it to learn its initial role at startup without waiting to win.
OnFollower func()
}
Observer is a set of optional callbacks invoked on leadership transitions. A nil field is simply not called. Callbacks run on the Run goroutine and must not block; offload slow work (I/O, network) to a goroutine of your own.
Only role transitions are reported, mirroring the design of k8s.io/client-go/tools/leaderelection. Transient Redis errors during acquire or renewal are handled internally; their only consequence the caller cares about — losing leadership — surfaces through OnSteppedDown. Monitor Redis health through your Redis client, not through this Observer.
Source Files