replica

README

replica

A pure-Go (CGO_ENABLED=0), standard-library-only replication engine for highly-available block volumes in the go-volumes family. It fronts N synchronous replicas — each any volume.Device (in production an NBD client to a remote pool volume, in tests an in-memory device) — behind a single Engine that itself satisfies volume.Device. A filesystem-format driver or an NBD server writes through the Engine exactly as it would a local device.

Model

Single-active-writer with synchronous replication (RPO 0):

• WriteAt mirrors every write to all in-sync replicas concurrently and acks only once they all ack. A replica that errors is marked out-of-sync and dropped from the write set; the write still succeeds as long as at least MinInSync in-sync replicas remain, otherwise it fails (ErrNoInSync).
• ReadAt serves from one in-sync replica, preferring the configured Local one, failing over to the next on a read error.
• Sync flushes every in-sync replica concurrently (same degrade rule).
• Size is the replicas' agreed size, validated equal at construction (ErrSizeMismatch otherwise).
• Close closes every replica.

Per-replica state is InSync | OutOfSync | Rebuilding, exposed via Status().

Rebuild(ctx, name, chunkSize) resyncs an out-of-sync (or new) replica by streaming the volume from a healthy in-sync source (ReadAt → WriteAt, chunked). The target is flipped to Rebuilding first, so live writes are mirrored to it alongside the in-sync set while the stream runs; it flips to InSync once caught up. A live write that fails on the target aborts the rebuild.

Failover and fencing of a stale writer (STONITH before promotion) are a future control-plane concern. The engine leaves a Fencer seam for it but implements no consensus here.

Durability

The Engine adds no buffering of its own: durability between the engine and its replicas relies on each replica's own Sync (a pool fsync, an S3 PUT, an NBD FLUSH to a remote backing).

Usage

e, err := replica.New([]replica.Replica{
    {Name: "local",  Dev: localDev},   // e.g. a same-host pool volume
    {Name: "remote", Dev: nbdClient},  // e.g. an NBD client to another host
}, replica.Config{MinInSync: 1, Local: "local"})
if err != nil { /* ... */ }
defer e.Close()

var dev volume.Device = e             // *Engine satisfies volume.Device
_, _ = dev.WriteAt(buf, off)          // mirrored to all in-sync replicas
_, _ = dev.ReadAt(buf, off)           // served from the local replica
_ = dev.Sync()                        // flush all in-sync replicas

// Heal a degraded replica.
_ = e.Rebuild(context.Background(), "remote", 0) // 0 = DefaultChunkSize

Install

go get github.com/go-volumes/replica

Dependencies: the Go standard library and github.com/go-volumes/interface only.

License

BSD-3-Clause — see LICENSE.

Documentation

Overview

Package replica is the data plane for replicated, highly-available block volumes in the go-volumes family. It fronts N synchronous replicas — each any volume.Device (in production an NBD client to a remote pool volume, in tests an in-memory device) — behind a single Engine that itself satisfies volume.Device, so a filesystem-format driver or an NBD server can write through it unchanged.

The model is single-active-writer with synchronous replication (RPO 0): one Engine owns a volume and mirrors every write to all in-sync replicas, acking only once they all ack. A replica that errors is marked out-of-sync and dropped from the write set; the write still succeeds as long as at least MinInSync replicas remain in sync, otherwise it fails. Reads are served from one healthy replica (preferring the configured local one) with failover.

Failover and fencing of a stale writer (STONITH before promotion) are a future control-plane concern; this package leaves a Fencer seam for it but implements no consensus here.

Durability between the Engine and its replicas relies on each replica's own Sync: the Engine's Sync flushes every in-sync replica, and each replica is responsible for making those bytes durable (a pool fsync, an S3 PUT, an NBD FLUSH to a remote backing). The Engine adds no buffering of its own.

Index

• Constants
• Variables
• type Config
• type Engine
  • func New(replicas []Replica, cfg Config) (*Engine, error)
  • func (e *Engine) Close() error
  • func (e *Engine) Fencer() Fencer
  • func (e *Engine) ReadAt(p []byte, off int64) (int, error)
  • func (e *Engine) Rebuild(ctx context.Context, name string, chunkSize int) error
  • func (e *Engine) Size() (int64, error)
  • func (e *Engine) Status() []ReplicaStatus
  • func (e *Engine) Sync() error
  • func (e *Engine) WriteAt(p []byte, off int64) (int, error)
• type Fencer
• type Replica
• type ReplicaStatus
• type State
  • func (s State) String() string

Constants

const DefaultChunkSize = 1 << 20 // 1 MiB

DefaultChunkSize is the rebuild streaming granularity when Rebuild does not specify one.

Variables

var (
	// ErrNoReplicas is returned by New when no replicas are supplied.
	ErrNoReplicas = errors.New("replica: no replicas")
	// ErrSizeMismatch is returned by New when replicas disagree on size.
	ErrSizeMismatch = errors.New("replica: replica size mismatch")
	// ErrNoInSync is returned when an operation cannot meet the minimum number
	// of in-sync replicas (the volume has lost too many replicas).
	ErrNoInSync = errors.New("replica: not enough in-sync replicas")
	// ErrUnknownReplica is returned by Rebuild for an unknown replica name.
	ErrUnknownReplica = errors.New("replica: unknown replica")
	// ErrDuplicateName is returned by New when two replicas share a name.
	ErrDuplicateName = errors.New("replica: duplicate replica name")
	// ErrClosed is returned by operations after the Engine is closed.
	ErrClosed = errors.New("replica: engine closed")
)

Errors returned by the Engine.

Types

type Config

type Config struct {
	// MinInSync is the minimum number of in-sync replicas a write/flush must
	// reach to succeed. It defaults to 1 (any single surviving replica keeps the
	// volume writable). Set it higher for quorum-style durability. It is clamped
	// to [1, len(replicas)].
	MinInSync int
	// Local names the replica reads prefer (e.g. a same-host replica). Empty (or
	// unmatched) prefers the first replica.
	Local string
	// Fencer, if set, is exposed via [Engine.Fencer] for the control plane.
	Fencer Fencer
}

Config tunes an Engine. The zero value is valid: MinInSync defaults to 1 and Local defaults to the first replica.

type Engine

type Engine struct {
	// contains filtered or unexported fields
}

Engine fronts a set of replicas as a single replicated volume.Device. It is safe for concurrent use. Construct it with New.

func New

func New(replicas []Replica, cfg Config) (*Engine, error)

New builds an Engine over replicas. It validates that the replica set is non-empty, that names are unique, and that every replica reports the same size (returning ErrSizeMismatch otherwise). All replicas start in-sync.

func (*Engine) Close

func (e *Engine) Close() error

Close closes every replica device once and marks the Engine closed. It joins the per-replica Close errors. Subsequent operations return ErrClosed.

func (*Engine) Fencer

func (e *Engine) Fencer() Fencer

Fencer returns the configured Fencer seam (nil if none). The control plane uses it to fence a stale writer before promotion; the Engine never calls it.

func (*Engine) ReadAt

func (e *Engine) ReadAt(p []byte, off int64) (int, error)

ReadAt serves from one in-sync replica, preferring the configured local one, then trying the others in order on error. It returns the first successful read; if every in-sync replica errors, it returns the last error, and if there are no in-sync replicas it returns ErrNoInSync.

func (*Engine) Rebuild

func (e *Engine) Rebuild(ctx context.Context, name string, chunkSize int) error

Rebuild resynchronizes the named replica from a healthy in-sync source by streaming the whole volume (source ReadAt → target WriteAt) in chunks, then marks the target in-sync.

Concurrency: the target is first flipped to Rebuilding, which makes Engine's write path mirror live writes to it alongside the in-sync set (see mirror). Each chunk is then read from a live in-sync source — which is itself receiving those same live writes — and written to the target. Because both source and target see every concurrent write, and the source is the authority for each chunk's current contents, the target converges to the source once the full span has been copied. A live write that demotes the target back to OutOfSync aborts the rebuild.

chunkSize <= 0 uses DefaultChunkSize. Rebuild honors ctx cancellation between chunks.

func (*Engine) Size

func (e *Engine) Size() (int64, error)

Size reports the replicated volume size (validated equal across replicas at construction).

func (*Engine) Status

func (e *Engine) Status() []ReplicaStatus

Status returns a snapshot of every replica's state, in configuration order.

func (*Engine) Sync

func (e *Engine) Sync() error

Sync flushes every in-sync (and rebuilding) replica concurrently, applying the same degrade rule as WriteAt. Durability of the bytes is each replica's own responsibility.

func (*Engine) WriteAt

func (e *Engine) WriteAt(p []byte, off int64) (int, error)

WriteAt mirrors p at off to every in-sync replica (and any rebuilding one) concurrently. It returns len(p) once at least MinInSync in-sync replicas ack; a replica that errors is marked out-of-sync. The write fails only if fewer than MinInSync in-sync replicas remain.

type Fencer

type Fencer interface {
	// Fence must isolate the named writer so it can no longer issue a single
	// write to the shared replicas, and return nil ONLY once that is definitively
	// true. A control plane opens the new leader's write gate the moment Fence
	// returns nil, so a Fencer that returns nil WITHOUT actually stopping the old
	// writer silently re-introduces split-brain and corrupts the volume. When in
	// doubt, return an error: a failed fence keeps the new leader passive (safe);
	// a falsely-successful one is unsafe.
	//
	// Implementations back this with whatever the substrate offers, strongest
	// first: power/STONITH (hard-stop the node or micro-VM), a cloud force-detach
	// or power-off, an IPMI/PDU cut or network ACL, or revoking the credential the
	// writer presents to the replicas. Fence should be idempotent (fencing an
	// already-dead writer returns nil) and must honour ctx (a timeout → error).
	Fence(ctx context.Context, writer string) error
}

Fencer is the seam a control plane uses to STONITH a stale writer before promoting a new one. The Engine does not call it (single-active-writer fencing is a control-plane decision — see github.com/go-volumes/replica-ha); it is held so a higher layer can fence through the same object that owns the replicas.

type Replica

type Replica struct {
	// Name uniquely identifies the replica within an Engine (e.g. the pool or
	// host it lives on). Names must be unique.
	Name string
	// Dev is the backing device. In production this is typically an NBD client
	// to a remote pool volume; in tests it is an in-memory device.
	Dev volume.Device
}

Replica names a backing volume.Device that an Engine mirrors a volume to.

type ReplicaStatus

type ReplicaStatus struct {
	Name  string
	State State
}

ReplicaStatus is a snapshot of one replica's state, returned by Engine.Status.

type State

type State int

State is a replica's replication state within an Engine.

const (
	// InSync means the replica holds the current data and participates in every
	// write and read.
	InSync State = iota
	// OutOfSync means the replica errored (or is new) and is excluded from
	// writes and reads until a [Engine.Rebuild] brings it current.
	OutOfSync
	// Rebuilding means the replica is being resynced: live writes are applied to
	// it alongside the in-sync set, but it is not yet trusted for reads.
	Rebuilding
)

func (State) String

func (s State) String() string

String renders a State for logs and status output.