README
¶
replica-ha
The high-availability control plane for go-volumes
replicated block volumes. It drives a
replica.Engine (the data plane that
mirrors a volume across N synchronous replicas) under leader election so that
exactly one node is the active writer for a volume at any instant — the safe
single-active-writer guarantee that makes failover non-destructive.
It is deliberately vendor-neutral: leader election and fencing are
interfaces you implement, not a baked-in dependency. Bring etcd, Consul,
ZooKeeper, or a SQL advisory lock; bring micro-VM STONITH, IPMI, or a cloud
detach call. The safety-critical reconcile core depends only on the Go standard
library and go-volumes/{replica,interface} — no consensus store is linked
in — and is exhaustively unit-tested (100% coverage, 6 arches) against an
in-memory fake.
The safety property it guarantees
A replicated volume is only safe if at most one writer is live at a time. If two nodes both believe they own the volume (split-brain — the classic outcome of a partition where a stale leader keeps writing while a new leader is elected on the other side), they mirror divergent writes into the same replicas and corrupt the volume irrecoverably.
replica-ha prevents this with fence-before-promote (STONITH): a node that
wins leadership MUST fence the previous writer — prove it can no longer write —
before it activates its own engine as the writer. If the fence fails or
times out, the node refuses to activate and stays passive. The control plane
never writes while a possibly-live old writer might also be writing.
Coordinator (lease election) Fencer
│ Observe: IsSelf=true ▲
▼ │ Fence(prevWriter)
follower ─▶ fence-pending ──(fence OK)──▶ leader │ MUST isolate the old
│ (fence FAILS → stay fence-pending, │ writer; nil ONLY if
│ NEVER write — no split-brain) │ that is truly done
▼ Observe: IsSelf=false (lease lost) │
demote ◀──────────────────────────────────────── ActiveDevice gate:
WriteAt/Sync → ErrNotLeader when passive
│
▼
replica.Engine → replica A, replica B, … (synchronous mirror)
Pieces
Coordinator— the coordination seam: lease-based leader election + membership. An interface, so the reconcile core is dependency-free and fully testable with an in-memory fake.Controller— the reconcile state machine:RoleFollower→RoleFencePending(won the lease, not yet writing) →RoleLeader(fence confirmed, gate open); demotes immediately on lease loss.ActiveDevice— wraps thereplica.EnginesoWriteAt/SyncreturnErrNotLeaderwhile passive (ReadAt/Sizealways pass through — a follower may still serve reads). This is how "active writer" is enforced on the data path; the engine itself has no leadership awareness.replica.Fencer— the fencing seam (defined ingo-volumes/replica).
Usage
import (
"github.com/go-volumes/replica"
replicaha "github.com/go-volumes/replica-ha"
)
eng, _ := replica.New(replicas, replica.Config{MinInSync: 2, Local: "node-a"})
ctrl, dev, err := replicaha.New(eng, myCoordinator, myFencer, logger)
if err != nil {
return err
}
// dev is the gated volume.Device the data path writes through — hand it to an
// NBD server, a filesystem-format driver, or pool.OpenWith. It rejects writes
// with ErrNotLeader until this node is the confirmed leader.
go ctrl.Run(ctx) // campaign, fence-before-promote, demote
defer ctrl.Stop(context.Background()) // resign + deactivate
Implementing a Coordinator
A Coordinator exposes a time-bounded lease for one volume. The lease is the
safety hinge: while you hold it you may (after fencing) write; the instant you
stop renewing it — stall, partition, being fenced — it expires and you are no
longer leader. Implement it over any store with sessions/leases and a
compare-and-swap election: etcd concurrency.Election, Consul sessions,
ZooKeeper ephemeral znodes, a Postgres advisory lock with a TTL, …
| Method | Contract |
|---|---|
Campaign(ctx) |
Block until this node acquires and holds the lease, or ctx is done. Returning nil means leadership is held now. Re-campaigning while leader may return immediately. |
Resign(ctx) |
Release the lease so a peer can win. A no-op (nil) when not leader. |
Observe(ctx) |
Stream a Leadership on every change. CRITICAL: a lost lease (expiry/partition) MUST surface here as IsSelf == false so the Controller demotes before the lease grace window elapses. An implementation that cannot observe its own lease loss is unsafe and unusable here. Deliver the current state promptly so a late subscriber is not blind; close the channel when ctx is done or the session dies. |
Members(ctx) |
The node IDs holding a live membership lease — fenced/partitioned nodes drop out automatically as their lease expires. |
NodeID() |
This node's stable identity (matches Leadership.Leader when leader; the id a peer passes to Fence). |
Leadership.Term must be monotonically non-decreasing, strictly greater on
every leader change — an epoch / fencing token. All methods must be
concurrency-safe and honour ctx.
Implementing a Fencer
type Fencer interface { Fence(ctx context.Context, writer string) error }
Fence must isolate the named writer so it can no longer issue a single write
to the shared replicas, and return nil only once that is definitively true.
The Controller opens the write gate the moment Fence returns nil — so a
Fencer that returns nil without actually stopping the old writer silently
re-introduces split-brain. When in doubt, return an error: a failed fence keeps
the new leader passive (safe); a falsely-successful one corrupts data.
Back it with whatever your substrate offers, strongest first:
- Power / STONITH — hard-stop the old writer's machine or micro-VM (this is
what weft does, via its agent:
StopVM, then poll until confirmed-stopped). A stopped node cannot write — the gold standard. - Cloud control plane — force-detach the volume / power off the instance via the provider API.
- Fabric isolation — an IPMI/PDU power cut, or a network ACL that severs the old writer from the replicas.
- Lease / credential revocation — revoke the token the old writer presents to the replicas (only safe if the replicas actually enforce it on every write).
Fence should be idempotent (fencing an already-dead node returns nil) and
honour ctx (a timeout → return an error → the new leader stays passive and
retries on the next observation).
Known limitation (honest)
If a node loses its lease during a slow Fence and then activates, there is a
brief window before it processes the demotion. That window is closed by STONITH
itself (the new leader hard-stops this node) and, ultimately, by threading a
fencing token (the lease Term) onto every replica write so a stale writer's
writes are rejected at the replica — the activate(token) hook is reserved for
exactly this. This is the standard state of the art (Longhorn and Patroni rely on
the same fencing model); replica-ha is explicit that watertightness depends on
the Fencer being real.
Status
Core control plane complete — Coordinator seam, Controller state machine,
ActiveDevice gate — 100% test coverage on all 6 supported 64-bit
architectures. An etcd-backed Coordinator (mirroring weft-ha-postgresql's
EtcdDCS, with an embedded-etcd integration test) and weft's STONITH Fencer
are integrator-side follow-ups that plug into these seams.
License
BSD-3-Clause © the go-volumes/replica-ha authors.
Documentation
¶
Overview ¶
Package replicaha is the high-availability control plane for go-volumes replicated block volumes. It drives a replica.Engine (the data plane that mirrors a volume across N synchronous replicas) under leader election so that exactly one node is the active writer for a volume at any instant — the safe single-active-writer guarantee that makes failover non-destructive.
The safety property ¶
A replicated volume is only safe if at most one writer is live at a time. If two nodes both believe they own the volume (split-brain — the classic outcome of a network partition where a stale leader keeps writing while a new leader is elected on the other side), they will mirror divergent writes into the same replicas and corrupt the volume irrecoverably. This package prevents that with fence-before-promote (STONITH): a node that wins leadership MUST fence the previous writer — prove it can no longer write — BEFORE it activates its own engine as the writer. If the fence fails or times out, the node refuses to activate and stays passive. The control plane never writes while a possibly-live old writer might also be writing.
Pieces ¶
- Coordinator is the coordination seam: lease-based leader election and membership. It is an interface so this core is dependency-free and fully testable with an in-memory fake; an etcd-backed implementation (mirroring weft-ha-postgresql's EtcdDCS) is a deferred follow-up that lives outside this package so the safety-critical state machine carries no etcd weight.
- Controller is the reconcile state machine. It campaigns for leadership, fences the prior writer before promoting, gates the engine between ACTIVE (writable) and PASSIVE (writes rejected), and demotes immediately when leadership is lost.
- ActiveDevice wraps the replica.Engine so that WriteAt/Sync return ErrNotLeader whenever the Controller is not the active writer. This is how "active writer" is modelled: the engine itself has no active/passive gate, so the gate lives here, in front of it, on the data path.
The control plane USES the engine and a replica.Fencer; it does not reimplement replication.
Index ¶
Constants ¶
This section is empty.
Variables ¶
var ( // ErrNotLeader is returned by the write path ([ActiveDevice.WriteAt], // [ActiveDevice.Sync]) when the [Controller] is not the active writer — it // is a follower, or a freshly-elected leader that has not yet confirmed the // prior writer is fenced. Reads are always permitted; only the write path is // gated. ErrNotLeader = errors.New("replica-ha: not the active writer (not leader)") // ErrFenceFailed is returned (and logged) when fence-before-promote could // not confirm the prior writer is fenced. On this error the Controller // refuses to activate and stays passive: it never promotes into a // maybe-live old writer. It wraps the underlying [replica.Fencer] error. ErrFenceFailed = errors.New("replica-ha: fence failed — refusing to promote") // ErrStopped is returned by [Controller.Run] when the controller has been // stopped via [Controller.Stop], and by operations attempted after Stop. ErrStopped = errors.New("replica-ha: controller stopped") // ErrNoEngine is returned by [New] when no [replica.Engine] is supplied. ErrNoEngine = errors.New("replica-ha: nil engine") // ErrNoCoordinator is returned by [New] when no [Coordinator] is supplied. ErrNoCoordinator = errors.New("replica-ha: nil coordinator") // ErrNoFencer is returned by [New] when no [replica.Fencer] is supplied. A // control plane with no way to fence cannot promote safely, so a nil fencer // is rejected at construction rather than discovered at the first failover. ErrNoFencer = errors.New("replica-ha: nil fencer") )
Functions ¶
func New ¶
func New(engine *replica.Engine, coord Coordinator, fencer replica.Fencer, log *slog.Logger) (*Controller, *ActiveDevice, error)
New builds a Controller over engine, coord and fencer. It returns the Controller and the ActiveDevice the data path must write through — that device starts PASSIVE and only becomes writable once the Controller has promoted (fenced the prior writer). All three dependencies are required.
If log is nil a discarding logger is used.
Types ¶
type ActiveDevice ¶
type ActiveDevice struct {
// contains filtered or unexported fields
}
ActiveDevice wraps a replica.Engine with an active/passive write gate. It is how this package models "active writer": the engine itself has no notion of leadership, so the gate lives in front of it. While the owning Controller is the active writer the gate is open and every call passes straight through to the engine; while it is passive the WRITE path (ActiveDevice.WriteAt, ActiveDevice.Sync) returns ErrNotLeader without touching any replica, and the READ path (ActiveDevice.ReadAt, ActiveDevice.Size) is always allowed (a follower may serve reads).
An ActiveDevice satisfies volume.Device, so a filesystem-format driver or an NBD server can write its on-disk image through it unchanged and will be transparently rejected with ErrNotLeader the instant this node stops being the leader — the data path needs no awareness of leadership beyond handling that error.
The gate is a single atomic bool: flipping it is wait-free and safe to call from the Controller's run loop while writes are in flight on other goroutines. A write that observes "active" and then races a demotion is the normal in-flight-write case; the lease's grace window plus the prior-writer fence on the NEXT promotion are what bound safety, not per-write locking.
func (*ActiveDevice) Close ¶
func (d *ActiveDevice) Close() error
Close releases the underlying engine. After Close the device must not be used again.
func (*ActiveDevice) IsActive ¶
func (d *ActiveDevice) IsActive() bool
IsActive reports whether the write gate is currently open.
func (*ActiveDevice) ReadAt ¶
func (d *ActiveDevice) ReadAt(p []byte, off int64) (int, error)
ReadAt always passes through to the engine: a follower may serve reads.
func (*ActiveDevice) Size ¶
func (d *ActiveDevice) Size() (int64, error)
Size reports the replicated volume size; always allowed.
func (*ActiveDevice) Sync ¶
func (d *ActiveDevice) Sync() error
Sync flushes the engine when active, and returns ErrNotLeader otherwise.
func (*ActiveDevice) WriteAt ¶
func (d *ActiveDevice) WriteAt(p []byte, off int64) (int, error)
WriteAt mirrors p at off to the engine when this node is the active writer, and returns (0, ErrNotLeader) otherwise without touching any replica.
type Controller ¶
type Controller struct {
// contains filtered or unexported fields
}
Controller is the per-node reconcile state machine that keeps exactly one node the active writer for a volume. It campaigns for leadership through a Coordinator, fences the prior writer through a replica.Fencer BEFORE activating its replica.Engine as the writer, and demotes immediately when leadership is lost. Construct it with New and drive it with Controller.Run.
func (*Controller) Device ¶
func (c *Controller) Device() *ActiveDevice
Device returns the gated ActiveDevice the data path writes through.
func (*Controller) Members ¶
func (c *Controller) Members(ctx context.Context) ([]string, error)
Members returns the currently-live node IDs from the Coordinator.
func (*Controller) Run ¶
func (c *Controller) Run(ctx context.Context) error
Run drives the control loop until ctx is cancelled or Controller.Stop is called. The shape:
- Subscribe to leadership changes via Coordinator.Observe.
- Campaign for leadership in the background; when it returns we believe we hold the lease, but we wait for the observed Leadership to confirm us before promoting — the observation is the authoritative signal.
- On each observed Leadership: - IsSelf true → promote: FENCE the prior writer, then (only on a confirmed fence) open the write gate. A failed/timed-out fence leaves us in RoleFencePending (not writing); the next observation retries. - IsSelf false → demote immediately: close the write gate so no further writes are accepted (lost lease / a peer won / partition).
Run returns ctx.Err() on cancellation, or ErrStopped after Stop.
func (*Controller) Status ¶
func (c *Controller) Status() Status
Status returns a snapshot of the controller's role, term, prior-writer identity, write-gate state, and the engine's replica status.
func (*Controller) Stop ¶
func (c *Controller) Stop(ctx context.Context) error
Stop deactivates the write gate and resigns leadership through the Coordinator. It is safe to call once; subsequent calls are no-ops that return nil. Stop does not cancel a running Controller.Run; cancel its context for that. Stop ensures that after it returns this node is not the active writer and has released its lease.
type Coordinator ¶
type Coordinator interface {
// Campaign blocks until this node becomes the leader (acquires and holds the
// write lease) or ctx is done. It returns nil once leadership is acquired,
// or ctx.Err() if ctx is cancelled first. It is not an error to call
// Campaign while already leader; implementations may return immediately.
Campaign(ctx context.Context) error
// Resign voluntarily gives up leadership (releases the lease), allowing a
// peer to win the next election. Resigning when not leader is a no-op and
// returns nil.
Resign(ctx context.Context) error
// Observe returns a channel that streams a [Leadership] on every leadership
// change for as long as ctx is live. The channel is closed when ctx is done
// (or the underlying session ends). A lost lease (expiry / partition) MUST
// be reported here as a Leadership with IsSelf == false so the Controller
// stops writing. Implementations should deliver the current leadership
// promptly after the call so a late subscriber is not left blind.
Observe(ctx context.Context) (<-chan Leadership, error)
// Members lists the node IDs that currently hold a live membership lease.
// A fenced or partitioned node drops out automatically as its lease expires,
// so the returned set reflects only reachable, live peers.
Members(ctx context.Context) ([]string, error)
// NodeID is this node's stable identity within the cluster. It matches the
// Leader field of a [Leadership] when this node is the leader, and the
// identity a peer passes to the [replica.Fencer] to fence this node.
NodeID() string
}
Coordinator is the coordination seam between the Controller and a lease-based consensus store (etcd, in the deferred production implementation). Keeping it an interface is what lets the safety-critical state machine in Controller stay dependency-free and exhaustively testable with an in-memory fake.
Lease semantics ¶
Leadership is backed by a time-bounded lease that the implementation renews in the background. The lease is the safety hinge of the whole design:
- While this node holds the lease it is the leader and MAY write (once it has fenced the prior writer — see Controller).
- If the node stops renewing the lease for any reason — process stall, network partition, the node being fenced by a peer — the lease EXPIRES at the store and leadership is lost. A lost lease MUST surface on the Coordinator.Observe channel as a Leadership whose IsSelf is false, so the Controller demotes and stops writing before the lease's grace window elapses. An implementation that cannot observe its own lease loss is unsafe and unusable here.
- Because the lease expires automatically, a partitioned-away old leader loses its claim without cooperating; the new leader still fences it (it cannot assume the old leader has already noticed) but the lease bounds the window of double ownership.
All methods must be safe for concurrent use and must honour ctx cancellation.
type Leadership ¶
type Leadership struct {
// Leader is the node ID of the current lease holder. It is empty when no
// node holds the lease (e.g. during an election after a leader's lease
// expired). An empty Leader with IsSelf false means "no leader right now".
Leader string
// IsSelf is true when Leader is this node ([Coordinator.NodeID]). When it
// becomes false after having been true, this node has LOST leadership and
// the [Controller] must stop writing immediately.
IsSelf bool
// Term is the monotonically non-decreasing election term (a.k.a. epoch or
// fencing token). Every successful campaign that changes the leader yields a
// strictly greater Term. The Controller records it so a new leader can be
// distinguished from a stale duplicate observation, and so a fencing token
// could be threaded to the data plane in a future revision.
Term int64
}
Leadership is a single observation of who currently holds the write lease for a volume. It is streamed by Coordinator.Observe on every leadership change.
type Role ¶
type Role int
Role is the Controller's role for a volume at a point in time.
const ( // RoleFollower means this node is not the leader: the write gate is closed // and [ActiveDevice.WriteAt] returns [ErrNotLeader]. RoleFollower Role = iota // RoleFencePending means this node has WON leadership but has not yet // confirmed the prior writer is fenced. It is NOT writing — this is the // safety state that prevents split-brain. It becomes [RoleLeader] only after // a successful fence, or falls back to follower behaviour (still not // writing) if the fence fails. RoleFencePending // RoleLeader means this node is the confirmed active writer: it holds the // lease AND has fenced the prior writer, so the write gate is open. RoleLeader )
type Status ¶
type Status struct {
// Role is the current control-plane role.
Role Role
// Term is the election term of the current (or most recent) leadership
// observation.
Term int64
// PrevWriter is the node ID this controller fenced (or attempted to fence)
// on its most recent promotion. Empty if it has never promoted.
PrevWriter string
// Active reports whether the write gate is currently open.
Active bool
// Replicas is the engine's per-replica replication state
// ([replica.Engine.Status]).
Replicas []replica.ReplicaStatus
}
Status is a snapshot of the Controller's state, returned by Controller.Status.
Source Files