v3

Documentation

Overview

package v3 is a proof-of-concept replacement for the raft consensus algorithm backed by an S3-compatible object store. It relies on S3 conditional writes (`If-None-Match: *`) to implement compare-and-swap appends to a shared log, which provides a total order of proposals without leader election or quorum replication: the object store itself is the single strongly-consistent authority.

This file contains a minimal, dependency-free S3 REST client (SigV4).

Index

• Constants
• Variables
• func Logger() *zap.Logger
• func NewRaftNode(bp, ssp, walp, clp unsafe.Pointer) unsafe.Pointer
• func S3OpenBackend(cfg config.ServerConfig, hooks backend.Hooks) backend.Backend
• func SetNotifier(n LogNotifier)
• func Start(lg *zap.Logger, rawurl string, id uint64, nsKey string, peers []raft.Peer, ...) (raft.Node, error)
• type BootstrappedRaft
• type LogNotifier
• type PushNotifier
  • func (p *PushNotifier) ServeHTTP(w http.ResponseWriter, r *http.Request)
  • func (p *PushNotifier) Watch(ctx context.Context, wake func()) error
• type RaftNode
• type RaftNodeConfig
• type SQSNotifier
  • func (s *SQSNotifier) Watch(ctx context.Context, wake func()) error
• type ToApply

Constants

const EnvURL = "ETCD_S3LOG_URL"

envURL is the switch: when non-empty, s3raft takes over consensus.

Variables

var (
	ActiveNS string
)

backend capture: OpenBackend (hijacked in hijack.go) runs during bootstrap, before the raft node is constructed, so it stashes the opened backend here for the checkpointer to snapshot. activeNS is the bucket namespace derived from the (identical-across-members, restart-stable) initial-cluster configuration; the node reuses it so log and snapshot objects share one prefix.

Functions

func Logger

func Logger() *zap.Logger

func NewRaftNode

func NewRaftNode(bp, ssp, walp, clp unsafe.Pointer) unsafe.Pointer

NewRaftNode replaces etcd's (*bootstrappedRaft).newRaftNode. It mirrors its body but builds the raft.Node from the S3 log via Start (instead of raft.StartNode/RestartNode). The arguments arrive as opaque pointer words so the signature is expressible without etcd's unexported types; the real ABI is receiver + 3 pointer args → 1 pointer result, all word-sized, so the register layout lines up. This call site — unlike the raft.StartNode seam — carries the *membership.RaftCluster, whose ID() is the etcd cluster ID.

func S3OpenBackend

func S3OpenBackend(cfg config.ServerConfig, hooks backend.Hooks) backend.Backend

s3OpenBackend replaces serverstorage.OpenBackend. It reproduces the stock backend configuration (via exported APIs), restores the bbolt file from the bucket when the local one is missing (disk-wiped recovery), opens the backend, and captures it for the checkpointer.

func SetNotifier

func SetNotifier(n LogNotifier)

SetNotifier installs a custom log-change notifier. Call before the node starts (e.g. from a deployment's init). A nil argument restores the default.

func Start

func Start(lg *zap.Logger, rawurl string, id uint64, nsKey string, peers []raft.Peer, ms *raft.MemoryStorage) (raft.Node, error)

start creates (or joins) the shared S3 log and returns a raft.Node backed by it. peers is non-empty when bootstrapping a new cluster, in which case the initial ConfChange entries are CAS-written to the log (idempotently: an existing log wins and is adopted). ms is the WAL-seeded MemoryStorage; entries already present there are not re-emitted through Ready.Entries.

nsKey namespaces all objects (`<nsKey>/...`) so multiple etcd clusters can share one bucket without seeing each other's logs. The hijack layer derives it (see nsFromConfig) identically across members and stably across membership changes and disk-wiped restarts.

Types

type BootstrappedRaft

type BootstrappedRaft struct {
	// contains filtered or unexported fields
}

type LogNotifier

type LogNotifier interface {
	Watch(ctx context.Context, wake func()) error
}

LogNotifier delivers "the shared log changed" signals so the node can sync its tail immediately instead of waiting for the poll fallback. This is the seam that decouples *where* change notifications come from:

• minioNotifier (default): MinIO's ListenBucketNotification streaming GET, for local/self-hosted clusters.
• An SQS-backed implementation long-polling a queue fed by S3 Event Notifications, for AWS.
• A Lambda entrypoint subscribed to S3 object-created events that holds a node reference and calls wake() per delivered event — no long-poll at all.

Watch blocks until ctx is cancelled, invoking wake once per detected change (coalescing is fine: the node re-reads the whole tail). It returns an error if the source is unavailable, and the node falls back to polling.

type PushNotifier

type PushNotifier struct {
	// contains filtered or unexported fields
}

PushNotifier is a LogNotifier fed by inbound S3 event deliveries instead of polling — the shape a Lambda deployment uses, where object notifications arrive as an HTTP POST. Wire its ServeHTTP onto the "POST /s3" route and install it with SetNotifier before the node starts:

pn := &s3raft.PushNotifier{}
s3raft.SetNotifier(pn)
http.Handle("POST /s3", pn)

Watch parks until ctx is done, holding the node's wake callback; each delivered S3 event that creates a log object invokes it.

func (*PushNotifier) ServeHTTP

func (p *PushNotifier) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP parses an S3 event notification body (POST /s3) and wakes the node if any record creates a log object. It coalesces to a single wake per delivery: the node re-reads the whole tail regardless.

func (*PushNotifier) Watch

func (p *PushNotifier) Watch(ctx context.Context, wake func()) error

type RaftNode

type RaftNode struct {
	RaftNodeConfig
	// contains filtered or unexported fields
}

type RaftNodeConfig

type RaftNodeConfig struct {
	raft.Node
	// contains filtered or unexported fields
}

type SQSNotifier

type SQSNotifier struct {
	QueueURL     string // https://sqs.<region>.amazonaws.com/<acct>/<queue>
	Region       string
	AccessKey    string
	SecretKey    string
	SessionToken string
	// WaitSeconds is the SQS long-poll wait (0..20); defaults to 20.
	WaitSeconds int
	// contains filtered or unexported fields
}

SQSNotifier is a LogNotifier for AWS deployments that are not Lambda: an SQS queue subscribed (directly or via SNS) to the bucket's S3 Event Notifications. It long-polls the queue and wakes the node for each event that creates a log object, deleting handled messages. This is the pull analog of the Lambda PushNotifier — same interface, no polling of S3 itself.

Construct from the environment with newSQSNotifierFromEnv (ETCD_S3LOG_SQS_URL) and install with SetNotifier before the node starts, or build one directly.

e2e requires a live SQS queue wired to the bucket; the event parsing is unit tested (notify_sqs_test.go). SigV4 for the "sqs" service is hand-rolled to keep s3raft dependency-free, mirroring the S3 signer in client.go.

func (*SQSNotifier) Watch

func (s *SQSNotifier) Watch(ctx context.Context, wake func()) error

Watch long-polls the queue until ctx is cancelled, calling wake once per batch that contains at least one log-object-created event. It returns the first connection error so the node can fall back to polling; once running, transient receive errors are retried after a short pause.

type ToApply

type ToApply struct {
	// contains filtered or unexported fields
}