libraft

README

libraft

libraft (s3raft) replaces etcd's raft consensus with an S3-compatible object store. S3 conditional writes (If-None-Match / CAS) build a shared, totally-ordered append log — and that log is the raft log. Every node runs its own etcd apply loop over the same log, so anything computed at apply time (revision, consistent index, auth state) is a deterministic function of the log and stays identical across nodes. Ships with CI, CodeQL, OpenSSF Scorecard, cosign-signed releases, Dependabot, an example, and an e2e harness.

It installs into etcd by machine-code monkey-patch — no edits to etcd source — triggered by a blank import plus the ETCD_S3LOG_URL environment variable. See The install seam below for the mechanism and v3/LIMITATIONS.md for the behavioral edges.

Quick Start

go get github.com/cnuss/libraft

Blank-import libraft in the etcd binary's main (or any program embedding etcd); it patches (*bootstrappedRaft).newRaftNode and serverstorage.OpenBackend at init when ETCD_S3LOG_URL is set (a no-op otherwise):

package main

import (
	_ "github.com/cnuss/libraft"
)

// ... build/run etcd as usual ...

ETCD_S3LOG_URL is the http(s) endpoint of any S3-compatible store, followed by the bucket (lowercase) and an optional prefix:

export ETCD_S3LOG_URL=https://s3.us-east-1.amazonaws.com/my-bucket/my-prefix
export AWS_ACCESS_KEY_ID=…  AWS_SECRET_ACCESS_KEY=…
etcd --data-dir /var/lib/etcd

Or try it locally against MinIO with the bundled example (credentials default to minioadmin/minioadmin):

docker run -d -p 9000:9000 minio/minio server /data
cd examples/basic
ETCD_S3LOG_URL=http://127.0.0.1:9000/libraft-demo/basic go run .

Run it twice: the second run starts from a brand-new data directory yet reads back the first run's value, restored from S3.

Layout

Three packages:

github.com/cnuss/libraft             — the import seam: blank-import this to
                                     install s3raft (re-exports EnvURL).
github.com/cnuss/libraft/v3          — s3raft core: S3 CAS log, raft.Node over
                                     the log, bbolt checkpoint/restore, notify,
                                     batch, metrics.
github.com/cnuss/libraft/v3/reflect  — the installer: monkey-patches etcd's
                                     raft construction into the core.

Hosts blank-import the root package (which pulls in v3/reflect); the core exports the seam the installer calls (Start, NewRaftNode, S3OpenBackend, ActiveNS, EnvURL, Logger). For the file-by-file map, see CONTRIBUTING.md → Where to find things.

The install seam

The installer rewrites the machine-code prologue of two functions with an unconditional jump into the core:

Patched target	Replacement
(*bootstrappedRaft).newRaftNode	v3.NewRaftNode → v3.Start
serverstorage.OpenBackend	v3.S3OpenBackend

newRaftNode is etcd's sole raft-construction site — the one call that also carries the snapshotter, WAL, and *membership.RaftCluster (whose ID() is the etcd cluster ID). It is unexported, so its code address is reached with //go:linkname and its unexported argument/return types are reconstructed as byte-identical layout mirrors in v3.NewRaftNode; etcd calls its own methods on the returned pointer, so only the memory layout must match. Platform primitives: mach_vm_protect on darwin, mprotect on linux, VirtualProtect on windows; amd64 and arm64.

Patching an unexported symbol inside etcdserver was once reverted for a build-layout-dependent SIGBUS (the target's text page could share with the patcher's own helpers). It is safe here because the patcher lives in a separate package that the linker places pages away from etcdserver; verify with go tool nm when touching the installer (see CONTRIBUTING).

When to use it

s3raft trades write latency for operational simplicity: every write is an S3 round-trip, so the latency floor is object-store RTT — physics, not tuning. That makes it a fit for low-write control planes (configuration, service discovery, CI locks) that want etcd's API without managing raft quorum, disks, and membership. It is not an etcd replacement for kube-apiserver-class write loads, and it is a mode with documented semantic differences (v3/LIMITATIONS.md) — not a silent drop-in. The CAS ordering that makes the shared log safe is modeled in TLA+ (v3/tla/).

Example

Self-contained program in ./examples:

Example	Demonstrates
basic	Embedded etcd + the blank import; with ETCD_S3LOG_URL set, the raft log lives in S3 and state survives a wiped data dir.

Run it locally:

make run basic

Testing

make test   # library unit + fuzz tests (fast, in-package)
make e2e    # builds and runs every example binary, asserts its output

make e2e runs go test -C e2e -count=1 -v . — the harness lives in its own module so its deps (docker SDK, etcdmain) stay out of the library's graph. The -count=1 defeats the test cache, since the harness builds the example binaries at runtime and the cache key wouldn't otherwise pick up example source changes. The s3raft legs need a real S3-compatible store: with AWS_REGION set the ambient AWS env is used (ETCD_S3LOG_URL must point at the store); without it the harness starts a throwaway MinIO container via docker, and skips those legs where no linux-container daemon is reachable.

Contributing

See CONTRIBUTING.md for the local dev loop, release process, and what makes a good example.

License

MIT

Documentation

Overview

Package libraft installs s3raft — etcd's raft consensus replaced by an S3-compatible object store — into the importing binary.

Blank-import it from the main package of any binary that embeds or builds etcd:

import _ "github.com/cnuss/libraft"

The installer (github.com/cnuss/libraft/v3/reflect) monkey-patches etcd's raft entry points at init when the ETCD_S3LOG_URL environment variable is set, and is a no-op otherwise. See the README for the install seam and v3/LIMITATIONS.md for the behavioral edges.

Index

• Constants
• type BootstrappedRaft
• type RaftNode
• type RaftNodeConfig
• type ToApply

Constants

const EnvURL = v3.EnvURL

EnvURL is the environment variable that activates s3raft. Its value is the http(s) endpoint of an S3-compatible store followed by the bucket and an optional prefix, e.g. https://s3.us-east-1.amazonaws.com/my-bucket/my-prefix (bucket names must be lowercase).

Types

type BootstrappedRaft

type BootstrappedRaft = v3.BootstrappedRaft

BootstrappedRaft mirrors etcd's (*bootstrappedRaft) receiver.

type RaftNode

type RaftNode = v3.RaftNode

RaftNode mirrors etcd's raftNode, the value NewRaftNode returns.

type RaftNodeConfig

type RaftNodeConfig = v3.RaftNodeConfig

RaftNodeConfig mirrors etcd's raftNodeConfig.

type ToApply

type ToApply = v3.ToApply

ToApply mirrors etcd's toApply.