redlease

package module
v0.4.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 27, 2026 License: MIT Imports: 11 Imported by: 0

README

redlease

CI Go Reference Go Report Card

Lease-based leader election on Redis, with fencing tokens.

go get github.com/nijatsdev/redlease

Pre-1.0: the API may change between v0.x releases.

redlease elects one long-lived leader among instances and keeps it elected — for a singleton background job, a scheduler, a cron-like task, or the single writer in a one-writer-many-readers system. It manages the whole leadership lifecycle: acquire, renew, step down, release, and fail over.

It is not a general-purpose mutex. The litmus test:

Is the lock held for the duration of a role or the duration of an operation? Role (be the leader, own the schedule, be the one writer) → redlease. Operation (guard this critical section, update this counter safely) → a distributed mutex.

One instance holds a Redis lock with a TTL and runs your work while it is leader. If it cannot renew the lock, it steps down so another instance takes over. What redlease adds on top:

Every leadership term is assigned a strictly increasing fencing token, and writes routed through the Fence* helpers reject any token older than the latest applied. A paused, GC-stalled, or clock-skewed leader that still believes it holds the lock cannot overwrite newer state — its writes are refused at Redis.

Why fencing

Lease-based leader election has an unavoidable window. The lock has a TTL; if the leader pauses (GC, CPU starvation) or is partitioned, the lock can expire and a second instance can be elected while the first still thinks it is leader. For a few seconds, two leaders exist. This is inherent to any lease-based lock — shortening the TTL only shrinks the window, it never closes it.

Fencing makes that window safe. Each term gets a token; the protected resource only accepts writes whose token is at least the highest it has already seen. The stale leader's writes carry an old token and are rejected. This is the mitigation Martin Kleppmann describes in How to do distributed locking.

Do you actually need it?

Fencing matters only when a stale leader's write to shared state would be harmful:

Your leader… Need fencing?
writes a value that must not regress (sequence number, counter, monotonic state) Yes
does no writes (runs a cron, sends notifications) No — at most you want idempotency/dedup
writes self-healing last-writer-wins state that the next correct write repairs No — a plain lock is enough

If you are in the "No" rows, a simpler lock will do. redlease is for the first row.

Usage

rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})

e, err := redlease.New(rc, redlease.Config{Name: "report-builder", TTL: 5 * time.Second})
if err != nil {
    log.Fatal(err)
}

// Run blocks until ctx is cancelled. The callback runs only while leader; its
// context is cancelled the instant leadership is lost. The Fencer carries this
// term's fencing token — use it for every fenced write, or pass it down to the
// code that writes shared state.
e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
    applied, err := f.HSet(leaderCtx, "jobs:report", "status", "running")
    if err != nil {
        // Redis error.
    }
    if !applied {
        // A newer leader has taken over; this term is stale. Stop working.
        return
    }
})

All instances that should contend for the same leadership must share the same Config.Name.

Fenced writes

A Fencer binds the term's token to the elector that minted it, so you carry one value instead of threading a token and a client separately. Each fenced write checks the token and performs the write atomically in a single Lua script, so there is no window in which the token could go stale between the check and the write. All methods share one high-water mark, so a token advanced by any of them fences every later, lower-token write:

f.HSet(ctx, hashKey, field, value) // fenced HSET
f.Set(ctx, key, value)             // fenced SET

// Escape hatch for any other Redis write (ZADD, XADD, multi-key, ...).
// KEYS[1]/ARGV[1] are reserved for the fence; address yours from index 2.
f.Eval(ctx,
    "redis.call('zadd', KEYS[2], ARGV[2], ARGV[3])",
    []string{"board"}, "100", "alice")

All three return (applied bool, err error): applied == false means your token is stale — a newer leader has taken over — and you should stop writing. f.Token() returns the raw token for fencing a resource that isn't Redis (see below).

The same writes are available on the elector itself — e.FenceHSet(ctx, token, ...), e.FenceSet, e.FenceEval — taking the token explicitly. Use those in the token-driven style, where you hold a token from e.Token() rather than a Fencer from the callback.

Observability

redlease emits no logs of its own — a library should not impose a log format, level, or destination on its caller. Instead it exposes leadership transitions through an optional Observer; wire them to your own logger, metrics, or tracing. Every field is optional and a nil one is simply not called.

e, _ := redlease.New(rc, redlease.Config{
    Name: "report-builder",
    Observer: redlease.Observer{
        OnElected:     func(token int64) { slog.Info("leader elected", "fence", token) },
        OnSteppedDown: func()            { slog.Info("leader stepped down") },
        OnFollower:    func()            { slog.Info("running as follower") },
    },
})

OnFollower fires when an acquire attempt finds the lock held by another instance — once per transition into the follower role, not on every retry. It lets a follower learn its initial role at startup without waiting to win.

Only role transitions are reported. Transient Redis errors during acquire or renewal are handled internally; the consequence the caller cares about — losing leadership — surfaces through OnSteppedDown. Monitor Redis health through your Redis client, not through the Observer. Callbacks run on the Run goroutine and must not block.

Checking leadership outside the callback

The Fencer reaches your LeaderFunc directly, but sometimes another goroutine — an HTTP handler, say — needs to act as the leader too. Fencer(), Token(), and IsLeader() expose the current state from any goroutine:

if f, ok := e.Fencer(); ok {
    // We are the leader. The fenced write is safe even if leadership changes
    // right now: a stale token is rejected at write time.
    f.HSet(ctx, "state", "key", "value")
}

Prefer Token() for anything that writes. IsLeader() exists for display/metrics, but it is advisory — leadership can be lost the instant after it returns, so never gate a correctness-sensitive write on if e.IsLeader() { write() }. That is the split-brain race fencing exists to prevent; carry the token through a Fence* helper instead.

If all your leader work happens this way — outside the callback — pass nil for the LeaderFunc. Run then just keeps the instance elected (acquire, renew, release) while your other goroutines act via Token():

go e.Run(ctx, nil) // hold leadership; do the work elsewhere via e.Token()

The callback style and this token-driven style are interchangeable — pick whichever fits your app.

How it works

  • Acquire — a single Lua script does SET name:leader <id> NX PX <ttl-ms> and, on success, INCR name:fence to mint the token. Doing both atomically guarantees every term's token is strictly greater than any prior term's. The TTL is set in milliseconds so sub-second values are honored exactly.
  • Hold — the leader renews the lock on RenewInterval via an ownership-checked script (it only extends a lock whose value is still its own id). A renewal that returns 0 means the lock was lost; a transient Redis error is tolerated until TTL would have lapsed, then the leader steps down.
  • Release — on graceful step-down the leader deletes the lock (ownership-checked, so it never deletes a successor's lock), letting the next instance take over without waiting for the TTL.
  • Fence — each fenced write runs a Lua script that stores the highest applied token in name:fence:applied and rejects any write carrying a lower token, performing the write only when the token is current.

Fencing writes that don't go to Redis

Fencing is not a Redis concept — it applies to any shared resource a stale leader could corrupt (a Postgres row, an S3 object). The catch is that the fence must be enforced at the resource itself, atomically with the write, because that is the only place the check and the write can happen together.

This package enforces the fence for Redis writes, because Redis is the resource it can reach into (via Lua). If your leader writes elsewhere, redlease still gives you the universal half — the monotonic token — but you must enforce it at your resource. For example, in Postgres:

UPDATE state SET value = $1, fence = $2 WHERE key = $3 AND fence <= $2;
-- rows affected == 0  ->  your token was stale; you were fenced out

So: use the Fence* helpers when you write to Redis; use the token with a conditional write (a WHERE fence <= token, a compare-and-swap, an If-Match precondition) when you write anywhere else.

Correctness boundary

Read this before using it for anything that matters.

The fencing token is generated and stored in Redis. On a single Redis instance, this gives a strict, monotonic guarantee: tokens never go backward, and the fence is sound.

On a replicated Redis deployment (Sentinel or Cluster), Redis replication is asynchronous. A primary can acknowledge the acquire — and the token INCR — before it has propagated to a replica, and a failover to that replica can lose it. In that window the monotonicity the fence depends on can be violated, and two leaders could in principle obtain non-ordered tokens. This is the same limitation that affects every Redis-based lock, fencing or not.

On Redis Cluster there is also a separate, non-negotiable requirement: the lock, fence, and applied keys all derive from Config.Name, and a single Lua script touches more than one of them, so they must hash to the same slot. Wrap Name in a hash tag — e.g. "{report-builder}" — so every derived key shares it. Without one they scatter across slots and the acquire script fails with CROSSSLOT.

So redlease is the right tool when:

  • you run Redis single-instance, or
  • a brief, rare token regression on Redis failover is acceptable for your workload.

If you need a fencing guarantee that survives failover, source the token from a linearizable store and apply it against your resource. redlease deliberately does not pretend Redis can provide that.

License

MIT

Documentation

Overview

Package redlease implements lease-based leader election on Redis with fencing tokens.

One instance holds a Redis lock with a TTL and runs the caller's work while it is leader; if it cannot renew, it steps down so another instance takes over. Unlike a plain Redis lock, every leadership term is assigned a strictly increasing fencing token. Writes routed through the Elector's Fence* helpers are rejected when they carry a token older than the latest applied, so a paused or clock-skewed leader that still believes it holds the lock cannot overwrite newer state.

When you need this

Fencing matters only when a stale leader's write to shared state would be harmful — for example a value that must not regress, a sequence number, or a counter. If the leader does no writes, or writes self-healing last-writer-wins state, a plain lock is enough and you do not need this package.

Correctness boundary

The fence token is generated and stored in Redis. On a single Redis instance this gives a strict, monotonic guarantee. On a replicated deployment (Sentinel or Cluster), Redis replication is asynchronous: a primary can acknowledge the acquire — and the token increment — before it reaches a replica, and a failover to that replica can lose them. In that window the monotonicity the fence relies on can be violated. For strict correctness across failover, source the fencing token from a linearizable store (etcd, ZooKeeper) instead of Redis. This package is the right tool when you run Redis single-instance, or when a brief, rare token regression on failover is acceptable.

On Redis Cluster, also wrap Config.Name in a hash tag (e.g. "{name}") so the lock, fence, and applied keys hash to the same slot; otherwise the acquire script fails with CROSSSLOT.

Example

This example runs a periodic job on exactly one instance at a time. Several instances call Run with the same Config.Name; only the elected leader executes the job. Each write the leader makes carries its leadership term's fencing token, so a stale leader that has not yet noticed it lost the lock cannot overwrite a newer leader's state. 

package main

import (
	"context"
	"log"
	"time"

	goredis "github.com/redis/go-redis/v9"

	"github.com/nijatsdev/redlease"
)

func main() {
	rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})

	e, err := redlease.New(rc, redlease.Config{
		Name: "report-builder",
		TTL:  5 * time.Second,
	})
	if err != nil {
		log.Fatal(err)
	}

	ctx, cancel := context.WithCancel(context.Background())
	defer cancel()

	// Run blocks until ctx is cancelled. The callback runs only while this
	// instance is the leader; its context is cancelled the moment leadership
	// is lost. The Fencer carries this term's fencing token — use it for every
	// write to shared state.
	e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
		ticker := time.NewTicker(time.Second)
		defer ticker.Stop()

		for {
			select {
			case <-leaderCtx.Done():
				return
			case <-ticker.C:
				// Record progress under the fencing token. If a newer leader
				// has taken over, applied is false: this term is stale and
				// must stop working.
				applied, err := f.HSet(leaderCtx, "jobs:report", "status", "running")
				if err != nil {
					log.Printf("write failed: %v", err)
					continue
				}

				if !applied {
					log.Printf("fenced out by a newer leader; stopping")
					return
				}
			}
		}
	})
}
Example (DatabaseFencing)

This example fences a write to a resource other than Redis — here a SQL database. redlease enforces the fence for you only on Redis writes; for any other store it gives you the term's fencing token and you enforce it at the resource, atomically with your write. In SQL that means a conditional UPDATE that only applies when the row's stored fence is not newer than your token.

The table is assumed to hold a fence column alongside the value: 

CREATE TABLE state (id text PRIMARY KEY, value text, fence bigint NOT NULL DEFAULT 0);

package main

import (
	"context"
	"database/sql"
	"log"
	"time"

	goredis "github.com/redis/go-redis/v9"

	"github.com/nijatsdev/redlease"
)

func main() {
	rc := goredis.NewClient(&goredis.Options{Addr: "localhost:6379"})

	db, err := sql.Open("postgres", "postgres://localhost/app")
	if err != nil {
		log.Fatal(err)
	}

	defer func() { _ = db.Close() }()

	e, err := redlease.New(rc, redlease.Config{Name: "report-builder", TTL: 5 * time.Second})
	if err != nil {
		log.Print(err)
		return
	}

	ctx := context.Background()

	e.Run(ctx, func(leaderCtx context.Context, f redlease.Fencer) {
		// Take the term's token from the Fencer and enforce it yourself in SQL.
		// The UPDATE applies only when our token is at least the stored fence,
		// then advances it; a stale leader's lower token matches no row.
		const q = `UPDATE state
		             SET value = $1, fence = $2
		           WHERE id = $3 AND fence <= $2`

		res, err := db.ExecContext(leaderCtx, q, "running", f.Token(), "report")
		if err != nil {
			log.Printf("write failed: %v", err)
			return
		}

		if n, _ := res.RowsAffected(); n == 0 {
			// No row updated: a newer leader has advanced the fence past our
			// token. We are stale and must stop working.
			log.Printf("fenced out by a newer leader; stopping")
			return
		}
	})
}

Index

Examples

Constants

View Source 
const DefaultTTL = 5 * time.Second

DefaultTTL is the lock TTL used when Config.TTL is unset. Override via Config.

Variables

This section is empty.

Functions

This section is empty.

Types

type Config 

type Config struct {
	// Name identifies the lock. All instances contending for the same leadership
	// must use the same Name; different Names are independent locks.
	//
	// The lock, fence, and applied keys all derive from Name, and a single Lua
	// script touches more than one of them. On Redis Cluster they must therefore
	// hash to the same slot: wrap Name in a hash tag, e.g. "{report-builder}",
	// so all derived keys share it. Without one they scatter across slots and the
	// acquire script fails with CROSSSLOT.
	Name string

	// TTL is how long the lock lives without renewal. A leader that cannot renew
	// within TTL loses leadership. Shorter TTL means faster failover but less
	// tolerance for pauses. Must be at least 100ms so a renewal can complete
	// before expiry. Defaults to DefaultTTL.
	TTL time.Duration

	// RenewInterval is how often the leader renews the lock. Must be at most
	// TTL/2, so at least two renewal attempts fit before expiry and a single
	// dropped renewal does not cost leadership. Defaults to TTL/3.
	RenewInterval time.Duration

	// AcquireInterval is how often a follower retries acquiring the lock. Must be
	// less than TTL, so a follower polls at least once per lock lifetime.
	// Defaults to TTL/2.
	AcquireInterval time.Duration

	// InstanceID uniquely identifies this instance; it is the lock's value and
	// guards renewal and release so an instance never affects another's lock. If
	// empty, a hostname + random suffix is generated.
	InstanceID string

	// Observer receives lifecycle events. All fields are optional; the library
	// emits no output of its own, so callers wire these to their own logger,
	// metrics, or tracing as they see fit.
	Observer Observer
}

Config configures an Elector. Only Name is required (the client is passed separately to New); zero-valued timing fields fall back to the Default* constants.

type Elector 

type Elector struct {
	// contains filtered or unexported fields
}

Elector runs leader election for a single lock and mints fencing tokens.

func New 

func New(client Redis, cfg Config) (*Elector, error)

New returns an Elector from cfg. It returns an error if required fields are missing or timing parameters are inconsistent: TTL below 100ms, RenewInterval greater than TTL/2, or AcquireInterval not less than TTL.

func (*Elector) FenceEval 

func (e *Elector) FenceEval(ctx context.Context, token int64, body string, writeKeys []string, args ...any) (applied bool, err error)

FenceEval fences an arbitrary Redis write supplied as a Lua body, for writes the typed helpers do not cover (ZADD, XADD, multi-key updates, and so on). The body runs atomically only when token is current; the fence returns 1 when applied and 0 when fenced out.

The body must not contain its own return statement: FenceEval appends the fence's "return 1", so a return inside body shadows it and makes the result report applied == false even on a write that ran. Just perform the write and let FenceEval supply the return.

Within body, the protected token check is already done. Address your own keys and arguments starting at index 2 — KEYS[1] and ARGV[1] are reserved for the fence (the high-water key and the token). Pass your keys in writeKeys and your arguments in args; they appear as KEYS[2..] and ARGV[2..] in that order.

Example — fence a ZADD: 

applied, err := e.FenceEval(ctx, token,
    "redis.call('zadd', KEYS[2], ARGV[2], ARGV[3])",
    []string{"myset"}, "1.0", "member")

func (*Elector) FenceHSet 

func (e *Elector) FenceHSet(ctx context.Context, token int64, hashKey, field, value string) (applied bool, err error)

FenceHSet performs a fenced HSET of field=value into hashKey: the write is applied only if token is at least the highest token already applied through this elector. It returns true when applied and false when token is stale — a newer leadership term has since written — so the caller can stop emitting derived events. token must be the value passed to the LeaderFunc for the current term. A non-nil error means the write could not be attempted.

func (*Elector) FenceSet 

func (e *Elector) FenceSet(ctx context.Context, token int64, key, value string) (applied bool, err error)

FenceSet performs a fenced SET of key=value, applied only if token is current. Semantics match Elector.FenceHSet.

func (*Elector) Fencer added in v0.3.0 

func (e *Elector) Fencer() (f Fencer, ok bool)

Fencer returns a Fencer bound to the current leadership term and true while this instance is the leader, or a zero Fencer and false otherwise. It is the token-driven-style counterpart to the Fencer a LeaderFunc receives: code outside the callback can take a Fencer and pass it down to its writers.

Like Elector.Token, the returned Fencer is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fencer's methods.

func (*Elector) InstanceID 

func (e *Elector) InstanceID() string

InstanceID returns this elector's instance identity.

func (*Elector) IsLeader 

func (e *Elector) IsLeader() bool

IsLeader reports whether this instance currently holds leadership. It is advisory only: leadership can be lost the instant after it returns, so never gate a correctness-sensitive write on it. For writes, carry the token from Elector.Token through a Fence* helper, which rejects a stale token at write time.

func (*Elector) Run 

func (e *Elector) Run(ctx context.Context, fn LeaderFunc)

Run contends for leadership until ctx is cancelled. Each time this instance wins, it invokes fn (a LeaderFunc) with a Fencer for the term, then steps down and re-contends when leadership is lost. Run blocks until ctx is cancelled and fn (if running) has returned; on the final shutdown step-down the OnSteppedDown observer still fires.

fn may be nil. Then Run just keeps this instance elected — acquiring, renewing, and releasing the lock — while the caller does its leader work from another goroutine, gated on Elector.Token. This is the token-driven style; the callback style and this one are interchangeable, pick whichever fits.

Run must not be called more than once concurrently on the same Elector; the two calls would corrupt shared leadership state. Use one Elector per Run.

func (*Elector) Token 

func (e *Elector) Token() (token int64, ok bool)

Token returns the current leadership term's fencing token and true while this instance is the leader, or 0 and false otherwise. It is safe to call from any goroutine, so code outside the LeaderFunc can perform fenced writes: 

if token, ok := e.Token(); ok {
    e.FenceHSet(ctx, token, "state", "key", "value")
}

The returned token is a snapshot; leadership can change immediately after. That is safe because the fence is enforced at write time — a stale token is rejected by the Fence* helpers — so unlike Elector.IsLeader there is no time-of-check-to-time-of-use hazard in acting on it.

type Fencer added in v0.3.0 

type Fencer struct {
	// contains filtered or unexported fields
}

Fencer binds a leadership term's fencing token to the Elector that minted it, so a single value carries everything a fenced write needs. It is what a LeaderFunc receives, and what you pass down into the code that writes shared state.

A Fencer is a small value; copy it freely. Its methods are the bound equivalents of the Elector's Fence* helpers: each applies its write only when the bound token is at least the highest already applied, and returns applied == false when the token is stale (a newer term has written), so the caller can stop. A Fencer is valid only for its term; once leadership is lost, its writes are fenced out like any other stale token.

func NewFencer added in v0.3.0 

func NewFencer(e *Elector, token int64) Fencer

NewFencer binds an explicit token to e, for callers that already hold a token — for example one persisted from a prior leadership term, or supplied in a test. Most code should instead take the Fencer a LeaderFunc receives or call Elector.Fencer; reach for this only when the token comes from outside the elector. The token is not validated against the elector's current term: the fence is still enforced at write time, so a stale token is rejected there.

func (Fencer) Eval added in v0.3.0 

func (f Fencer) Eval(ctx context.Context, body string, writeKeys []string, args ...any) (applied bool, err error)

Eval fences an arbitrary Lua write. See Elector.FenceEval.

func (Fencer) HSet added in v0.3.0 

func (f Fencer) HSet(ctx context.Context, hashKey, field, value string) (applied bool, err error)

HSet performs a fenced HSET of field=value into hashKey. See Elector.FenceHSet for the full semantics.

func (Fencer) Set added in v0.3.0 

func (f Fencer) Set(ctx context.Context, key, value string) (applied bool, err error)

Set performs a fenced SET of key=value. See Elector.FenceSet.

func (Fencer) Token added in v0.3.0 

func (f Fencer) Token() int64

Token returns the fencing token this Fencer carries — the value the elector assigned to this leadership term.

type LeaderFunc 

type LeaderFunc func(ctx context.Context, f Fencer)

LeaderFunc is the work run while an instance is leader. It receives a context cancelled when leadership is lost or Run's context is cancelled, and a Fencer bound to this leadership term — use it for every fenced write, or pass it down to the code that writes shared state. LeaderFunc must return promptly once its context is cancelled; Run does not release the lock or re-contend until it does.

type Observer 

type Observer struct {
	// OnElected fires when this instance wins a leadership term, with that
	// term's fencing token. It fires before the LeaderFunc is invoked.
	OnElected func(token int64)

	// OnSteppedDown fires when this instance loses or relinquishes leadership,
	// after the LeaderFunc has returned.
	OnSteppedDown func()

	// OnFollower fires when this instance is, for now, a follower: an acquire
	// attempt found the lock held by another instance. It fires once per
	// transition into the follower role — on the first lost attempt, and again
	// only after an intervening leadership term — not on every retry. A consumer
	// can use it to learn its initial role at startup without waiting to win.
	OnFollower func()
}

Observer is a set of optional callbacks invoked on leadership transitions. A nil field is simply not called. Callbacks run on the Run goroutine and must not block; offload slow work (I/O, network) to a goroutine of your own.

Only role transitions are reported, mirroring the design of k8s.io/client-go/tools/leaderelection. Transient Redis errors during acquire or renewal are handled internally; their only consequence the caller cares about — losing leadership — surfaces through OnSteppedDown. Monitor Redis health through your Redis client, not through this Observer.

type Redis 

type Redis interface {
	goredis.Scripter
}

Redis is the subset of the go-redis client this package needs: the script runner used by all lock and fence operations. *goredis.Client and *goredis.ClusterClient both satisfy it, as does any compatible wrapper.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.