hoplock

README

hoplock

Lease-based leader election in Go, backed by S3 (or any S3-compatible store with conditional writes). Zero external dependencies, ~1k lines of code.

Why

Distributed coordination usually means running etcd, Consul, or a Raft cluster. That is operational overhead you may not want for a single elected leader. S3 gives you a strongly-consistent compare-and-swap primitive — If-None-Match: * and If-Match: <ETag> on PUT — which is enough to elect one writer at a time without any side infrastructure.

hoplock is the smallest reusable wrapper around that idea: a Backend interface with three methods, an Elector that manages the acquire/renew loop, and an S3 backend that talks plain HTTPS with an in-package sigv4 signer.

The pattern was popularised by Litestream's S3 leaser; this library is a clean-room re-implementation with a different API shape and no AWS SDK dependency.

Install

go get github.com/xinix00/hoplock

Pure Go, no cgo, no transitive dependencies.

Use

import (
    "context"
    "time"

    "github.com/xinix00/hoplock"
    "github.com/xinix00/hoplock/s3"
)

elector := &hoplock.Elector{
    Backend: &s3.Backend{
        Endpoint:        "https://s3.us-east-1.amazonaws.com",
        Bucket:          "my-bucket",
        Key:             "my-app/lease.json",
        Region:          "us-east-1",
        AccessKeyID:     os.Getenv("AWS_ACCESS_KEY_ID"),
        SecretAccessKey: os.Getenv("AWS_SECRET_ACCESS_KEY"),
    },
    Owner: "node-1",
    TTL:   30 * time.Second,
}

err := elector.Lead(ctx, func(leaderCtx context.Context, lease hoplock.Lease) error {
    // Run your leader-only work here. leaderCtx is cancelled when the
    // lease is lost; return promptly when that happens.
    return doLeaderWork(leaderCtx, lease.Generation)
})

Lease.Generation is a strictly monotonic counter that increments every time the lease changes hands. Use it as a fencing token in downstream systems that must reject writes from a stale leader (e.g. a node that thinks it is leader because its renewal is delayed).

Backends

S3 — hoplock/s3

Works against AWS S3 and any S3-compatible service that honours the conditional-write headers and offers strong read-after-write consistency on the lease key. Tested against AWS; tested-by-design against Cloudflare R2, MinIO, Tigris, Backblaze B2 (S3 API).

Provider	Region	UsePathStyle
AWS S3	us-east-1 (etc.)	false (default)
Cloudflare R2	auto	usually false
MinIO	us-east-1	true
Backblaze B2	endpoint-specific	true

Conditional writes on AWS S3 became generally available in November 2024. Older buckets, older S3 emulators, or third-party tools that strip preconditions will silently let two leaders coexist — verify before deploying.

In-memory — hoplock/mem

For tests and single-process use. Two Electors sharing one *mem.Backend will compete for the lease in the same way they would across machines, which is useful for unit-testing logic that depends on leader transitions.

Bring your own

hoplock.Backend is three methods (Read, Write, Delete). Anything that exposes a strongly-consistent compare-and-swap can implement it: Postgres advisory locks, etcd, ZooKeeper, the local filesystem with flock, your in-house metadata store. The leader-election logic in Elector does not change.

type Backend interface {
    Read(ctx context.Context) (*State, string, error)
    Write(ctx context.Context, prevHandle string, state *State) (string, error)
    Delete(ctx context.Context, handle string) error
}

Write is a CAS keyed on prevHandle. The empty string means "must not exist" (used for a fresh acquire); any other value must equal what the most recent Read returned. On precondition failure, return hoplock.ErrLeaseHeld.

Two API levels

Lead(ctx, fn) — the common case

Calls fn with a context that is cancelled when leadership is lost. If fn returns nil, Lead waits for the lease to be acquired again and calls fn once more. If fn returns an error, Lead returns it. On outer-context cancellation, Lead returns ctx.Err().

Run(ctx) <-chan Event — when you need finer control

Emits EventAcquired, EventRenewed, EventLost events. Use this if you need to react to renewal heartbeats, surface leadership state in a UI, or run multiple leadership-bound subsystems with different lifecycle rules.

Tuning

Field	Default	Notes
TTL	30s	Maximum gap between leaders. Worst-case failover time after a leader crash.
Renew	TTL/3	How often the leader writes a fresh ExpiresAt.
Poll	TTL/15	How often a follower checks whether the lease is free.
ReleaseTimeout	2s	Bound for the best-effort Delete issued on graceful shutdown.

If you tighten TTL below a few seconds you start running into clock skew between machines; the Elector logs a warning when it observes a lease state more than 2*TTL ahead of its local clock.

What this is not

• Not a consensus protocol. No quorum, no log replication. One leader at a time, and you trust the backend to enforce that.
• Not a fencing-token registry. Lease.Generation is a monotonic number, but you have to wire it into downstream systems yourself.
• Not a distributed lock service. It elects exactly one holder per Backend. To coordinate multiple resources, run multiple Electors against different keys.
• Not safe across S3-compatible services that fake conditional writes. Test your backend before trusting it.

Status

Pre-1.0. The Backend interface and Elector field set are likely stable; the s3.Backend configuration surface may grow (extra knobs for proxying, custom signing, etc.) without breaking existing fields.

Documentation

Overview

Package hoplock provides lease-based leader election over a pluggable backend.

A Backend persists a single lease record. Mutual exclusion relies on the backend's compare-and-swap semantics: every write either succeeds because the previous handle still matches, or fails with ErrLeaseHeld. The Handle is opaque (e.g. an S3 ETag) and serves as proof that no concurrent writer has touched the record since the caller last observed it.

Callers typically use Elector, which runs the acquire/renew loop and notifies on leadership changes. Backend can also be used directly for one-shot or custom flows.

Index

• Variables
• type Backend
• type Elector
  • func (e *Elector) Lead(ctx context.Context, fn func(ctx context.Context, lease Lease) error) error
  • func (e *Elector) Run(ctx context.Context) <-chan Event
• type Event
• type EventKind
• type Lease
• type State
  • func (s *State) IsExpired() bool
  • func (s *State) TTL() time.Duration

Variables

var (
	// ErrNoLease means no lease record exists at the configured key.
	ErrNoLease = errors.New("hoplock: no lease")

	// ErrLeaseHeld means the conditional write failed because the stored
	// handle differs from the one supplied. Either someone else holds the
	// lease, or the caller's view is stale.
	ErrLeaseHeld = errors.New("hoplock: lease held by another owner")
)

Errors returned by Backend implementations.

Types

type Backend

type Backend interface {
	// Read returns the current lease state and the handle observed at read
	// time. Returns (nil, "", ErrNoLease) if no record exists.
	Read(ctx context.Context) (*State, string, error)

	// Write commits state. If prevHandle is empty, succeeds only if no
	// record exists. Otherwise, succeeds only if the stored handle equals
	// prevHandle. Returns ErrLeaseHeld on precondition failure.
	Write(ctx context.Context, prevHandle string, state *State) (handle string, err error)

	// Delete removes the record only if handle still matches. Returns
	// ErrNoLease if the record is already gone, ErrLeaseHeld if a different
	// holder has taken over.
	Delete(ctx context.Context, handle string) error
}

Backend is the storage abstraction. Implementations must guarantee that Write and Delete are linearisable: at most one concurrent caller can succeed against the same prior state.

All mutations go through Write, which is a compare-and-swap keyed on prevHandle. The empty string means "no record must exist" (used to create a fresh lease); any other value must equal the handle returned by the most recent successful Read or Write (used to renew or take over). On precondition mismatch, Write returns ErrLeaseHeld.

type Elector

type Elector struct {
	// Backend is the storage abstraction. Required.
	Backend Backend

	// Owner is the identifier written into State.Owner. Defaults to
	// "<hostname>:<pid>" if empty.
	Owner string

	// TTL is the duration each acquired or renewed lease is valid for.
	// Defaults to 30s.
	TTL time.Duration

	// Renew is how often the loop attempts to refresh a held lease.
	// Defaults to TTL/3. Should be safely below TTL to absorb transient
	// backend errors.
	Renew time.Duration

	// Poll is how often the loop retries while waiting for a lease held
	// by someone else (or while recovering from transient errors).
	// Defaults to TTL/15.
	Poll time.Duration

	// ReleaseTimeout bounds the best-effort Delete attempt issued when the
	// Run context is cancelled while holding. Defaults to 2s. Set to a
	// negative value to skip release on shutdown.
	ReleaseTimeout time.Duration

	// Now overrides the wall-clock source for ExpiresAt computations and
	// expiry checks. Defaults to time.Now. Timer-based sleeping (Renew,
	// Poll) still uses real time — Now injection is for deterministic
	// decision-logic tests, not full clock virtualisation.
	Now func() time.Time

	// Logger receives debug/warning messages. Defaults to slog.Default().
	Logger *slog.Logger
}

Elector runs a single leader-election loop against a Backend. It is goroutine-safe to construct, but Run should be called at most once per Elector instance.

func (*Elector) Lead

func (e *Elector) Lead(ctx context.Context, fn func(ctx context.Context, lease Lease) error) error

Lead acquires the lease and calls fn with a context that is cancelled when leadership is lost. fn must return promptly when its context is cancelled. If fn returns a non-nil error, Lead stops trying and returns that error. If fn returns nil, Lead waits and tries to re-acquire, looping until the outer ctx is cancelled. Lead returns ctx.Err() on graceful shutdown.

func (*Elector) Run

func (e *Elector) Run(ctx context.Context) <-chan Event

Run starts the election loop and returns a channel that receives Events until ctx is cancelled. The channel is closed after the loop exits.

Send semantics: EventAcquired and EventLost block on send (callers must observe transitions); EventRenewed is non-blocking and dropped on a full buffer (idempotent — the next renewal will catch you up).

type Event

type Event struct {
	Kind  EventKind
	Lease Lease
	// Err is populated on EventLost when the loss was caused by a renewal
	// failure or context cancellation. Nil for EventAcquired/EventRenewed.
	Err error
}

Event describes a leadership transition or renewal observed by an Elector.

type EventKind

type EventKind int

EventKind classifies an Event emitted by an Elector.

const (
	// EventAcquired fires when this Elector starts holding the lease,
	// either by creating it from scratch or by taking over an expired one.
	EventAcquired EventKind = iota + 1

	// EventRenewed fires after each successful renewal of an already-held
	// lease. Useful for surfacing the new ExpiresAt.
	EventRenewed

	// EventLost fires when this Elector stops holding the lease, either
	// because renewal failed (Err is set) or because Run is exiting due
	// to context cancellation.
	EventLost
)

type Lease

type Lease struct {
	State
	Handle string
}

Lease is a State together with a Handle proving ownership. The Handle is backend-specific (e.g. an S3 ETag) and must be supplied to Replace or Delete to authorise the write.

type State

type State struct {
	// Generation is incremented every time leadership changes. Strictly
	// monotonic across the lifetime of a lease key. Use this as a fencing
	// token for downstream operations that must reject stale leaders.
	Generation int64 `json:"generation"`

	// ExpiresAt is the absolute time at which the lease becomes invalid
	// if not renewed. Compared against the local clock, so all participants
	// must have reasonably synchronised time.
	ExpiresAt time.Time `json:"expires_at"`

	// Owner is a free-form identifier of the holder. Informational only —
	// mutual exclusion does not depend on it.
	Owner string `json:"owner,omitempty"`
}

State is the lease record persisted in a Backend.

func (*State) IsExpired

func (s *State) IsExpired() bool

IsExpired reports whether the lease has passed its expiry time according to the local clock.

func (*State) TTL

func (s *State) TTL() time.Duration

TTL is the remaining time until expiry. Negative if already expired.