warden

Documentation

Overview

Package warden defines the shared contracts for the candacenet warden service: core types, the wire protocol, and the interfaces that the election, watchdog, notification, dashboard, and configuration packages implement or consume.

This package is frozen. Every other package under services/warden is written against the names here, so callers may rely on the types, the route path constants, and the eight interfaces keeping their meaning: a change that redefines one of them is a change to the whole service, not to one subsystem. The package is types, constants, pure helpers (Quorum, SortNodes, SortPeers, NewIncidentID) and the one real-time Clock the production binary runs on; it opens no connection, reads no file, and owns no shared mutable state, which is what lets every other package import it without a cycle. Behaviour lives in the packages that implement these interfaces.

Architecture overview

Every node in the fleet runs the same warden binary. Nodes perform Raft-style leader election (terms + votes, no log replication) over a static, configured peer set. A leader is elected only with a majority quorum, which makes split-brain impossible: at most one leader can exist per term, and a minority partition remains leaderless (correct and intentional).

The elected leader sends periodic heartbeats to all peers. Heartbeats carry the leader's authoritative ClusterView so that every follower can render a correct cluster dashboard without extra gossip. The leader also acts as the fleet watchdog: it tracks peer liveness transitions (alive -> suspect -> dead and recovery) and emits Incidents to a Notifier (SMTP email in production, mock/file/log in tests) with per-incident deduplication and a cooldown so flapping nodes do not spam the operator.

Transport is the gRPC WardenService over h2c on the tailnet (WireGuard is the node-to-node transport-security boundary). A single bound port per node serves the cluster RPCs (Vote, Heartbeat, Identify, and the WatchCluster stream) over gRPC, multiplexed by cmux with the HTTP surface: the SSR dashboard (PathDashboard), the JSON API (PathAPIStatus), and Prometheus metrics (PathMetrics).

Package layout. The reusable pieces live beside this package under services/warden; app/warden is the runnable composition on top of them.

warden         - this package: types, wire protocol, interfaces
election       - election state machine + peer liveness tracking
grpctransport  - gRPC client implementing Transport (h2c, pooled conns)
grpcserver     - gRPC WardenService server: unary RPCs + WatchCluster
grpcmux        - single-port cmux multiplexing the gRPC and HTTP surfaces
testclock      - fake clock for deterministic tests
watchdog       - leader-only incident engine with dedup/cooldown
notify         - Notifier implementations: SMTP, log, file, mock
dashboard      - SSR dashboard (HTMX + embedded assets) + JSON API
metrics        - Prometheus collectors + /metrics handler
config         - YAML + environment configuration loading
proto/warden/v1 - the candacenet.warden.v1 schema and its bindings
app/warden/cmd - main.go wiring (the runnable composition)

Index

• Constants
• func NewIncidentID(t IncidentType, peer NodeID, at time.Time) string
• func Quorum(n int) int
• func SortNodes(ns []Node)
• func SortPeers(peers []PeerView)
• type Clock
  • func NewRealClock() Clock
• type ClusterView
• type HeartbeatRequest
• type HeartbeatResponse
• type IdentifyResponse
• type Incident
• type IncidentLog
• type IncidentType
• type MemberKind
• type Membership
  • func (m Membership) Clone() Membership
  • func (m Membership) HasVoter(id NodeID) bool
  • func (m Membership) Supersedes(other Membership) bool
• type Node
• type NodeID
• type Notifier
• type PeerDiscoverer
• type PeerStatus
• type PeerView
• type PersistentState
• type RPCHandler
• type Role
• type Roster
• type Store
• type Term
• type Ticker
• type Timer
• type Transport
• type ViewSource
• type VoteRequest
• type VoteResponse

Constants

const (
	PathVote      = "/warden/v1/vote"
	PathHeartbeat = "/warden/v1/heartbeat"
	PathIdentify  = "/warden/v1/identify"
	PathAPIStatus = "/api/status"
	PathMetrics   = "/metrics"
	PathDashboard = "/"
	// PathClusterPartial is the HTMX partial the dashboard polls to
	// refresh the cluster table without a full page reload.
	PathClusterPartial = "/partials/cluster"
)

PathVote/PathHeartbeat/PathIdentify are retired: no HTTP handler registers them (those RPCs are gRPC methods now). They and the JSON message types above are kept only because wire_contract_test.go freezes them as the legacy-compat encoding; PathAPIStatus, PathMetrics, PathDashboard, and PathClusterPartial remain live — they still name the current HTTP surface.

Functions

func NewIncidentID

func NewIncidentID(t IncidentType, peer NodeID, at time.Time) string

NewIncidentID builds the canonical incident ID.

func Quorum

func Quorum(n int) int

Quorum returns the majority threshold for a cluster of n nodes (n/2 + 1). A candidate needs at least this many votes, counting its own.

func SortNodes

func SortNodes(ns []Node)

SortNodes sorts a Node slice by ID, the canonical order for Membership.Voters and Roster.Nodes.

func SortPeers

func SortPeers(peers []PeerView)

SortPeers sorts a PeerView slice by node ID, the canonical order for ClusterView.Peers.

Types

type Clock

type Clock interface {
	Now() time.Time
	After(d time.Duration) <-chan time.Time
	NewTimer(d time.Duration) Timer
	NewTicker(d time.Duration) Ticker
}

Clock abstracts time so the election and watchdog state machines can be tested deterministically with a simulated clock (services/warden/testclock). Production code uses NewRealClock.

func NewRealClock

func NewRealClock() Clock

NewRealClock returns a Clock backed by the time package.

type ClusterView

type ClusterView struct {
	// Self is the node rendering/returning this view.
	Self NodeID `json:"self"`
	// Role is Self's current election role.
	Role Role `json:"role"`
	// Term is Self's current term.
	Term Term `json:"term"`
	// LeaderID is the current known leader, or "" if unknown/leaderless.
	LeaderID NodeID `json:"leader_id"`
	// Source is the node whose observations produced Peers (the leader
	// for authoritative views; Self for local fallback views).
	Source NodeID `json:"source"`
	// Authoritative is true when Peers reflects the current leader's
	// liveness tracking (either Self is the leader, or the view was
	// received from the leader within the freshness window).
	Authoritative bool `json:"authoritative"`
	// UpdatedAt is when Source produced the peer observations.
	UpdatedAt time.Time `json:"updated_at"`
	// Peers contains every cluster member including Self and Source,
	// sorted by Node.ID (see SortPeers).
	Peers []PeerView `json:"peers"`
	// ElectionsStarted counts elections Self has started since boot
	// (exported as the warden_elections_total metric).
	ElectionsStarted uint64 `json:"elections_started"`
	// Membership is the effective voting configuration this view was
	// rendered under.
	Membership Membership `json:"membership"`
}

ClusterView is a point-in-time snapshot of the cluster as known by one node. The leader produces authoritative views from its own liveness tracking and piggybacks them on heartbeats; followers cache the most recent leader view so every node's dashboard shows the same cluster state. When no leader view is fresh (leaderless, or partitioned away from the leader), a node falls back to its own local observations with Authoritative == false.

type HeartbeatRequest

type HeartbeatRequest struct {
	Term     Term   `json:"term"`
	LeaderID NodeID `json:"leader_id"`
	// View is the leader's current authoritative cluster view. May be nil
	// (followers then keep their previous cached view).
	View *ClusterView `json:"view,omitempty"`
	// Membership is the leader's effective voting configuration. Receivers
	// with a lower local version persist-then-adopt it (only from the
	// leader they currently accept). May be nil (no change conveyed).
	Membership *Membership `json:"membership,omitempty"`
}

HeartbeatRequest is sent by the leader to every peer at the configured heartbeat interval. It asserts leadership for Term and carries the leader's authoritative ClusterView for follower dashboards.

type HeartbeatResponse

type HeartbeatResponse struct {
	Term   Term   `json:"term"`
	OK     bool   `json:"ok"`
	NodeID NodeID `json:"node_id"`
}

HeartbeatResponse acknowledges a heartbeat. OK is false when the receiver rejects the sender as leader (stale term; Term then tells the stale leader the newer term so it can step down), OR when the heartbeat carried a membership change (discovery mode) that this node failed to durably persist — the leader's one-at-a-time settle accounting (election/membership.go) must never count such a response as an ack, since doing so would let a change be declared committed before a real quorum has stored it.

OK is NOT a liveness signal: any response that arrives at all (regardless of OK) proves the sender is reachable, and the leader's peer-liveness bookkeeping (lastContact/latencyMS, election/heartbeat.go) updates on receipt independent of OK. Only the absence of a response (an RPC error) is evidence of non-liveness. A reachable follower that merely fails to persist a membership change must never be misclassified as dead.

type IdentifyResponse

type IdentifyResponse struct {
	ClusterID string `json:"cluster_id"`
	NodeID    NodeID `json:"node_id"`
	Version   string `json:"version"`
}

IdentifyResponse answers the GET PathIdentify handshake. Discovery treats a node as a same-cluster warden observer candidate only when ClusterID matches the local configuration.

type Incident

type Incident struct {
	// ID uniquely identifies the incident, e.g. "peer_dead/node-a/1721433600".
	ID   string       `json:"id"`
	Type IncidentType `json:"type"`
	// Peer is the affected node.
	Peer Node `json:"peer"`
	// Term is the reporting leader's term when the incident was detected.
	Term Term `json:"term"`
	// ReportedBy is the leader that detected the incident.
	ReportedBy NodeID `json:"reported_by"`
	// DetectedAt is when the leader detected the transition.
	DetectedAt time.Time `json:"detected_at"`
	// LastSeen is the leader's last successful contact with the peer
	// before the incident (zero if never seen).
	LastSeen time.Time `json:"last_seen"`
	// Message is a human-readable summary suitable for an email body line.
	Message string `json:"message"`
}

Incident is a single watchdog event that (subject to dedup/cooldown) results in exactly one operator notification.

type IncidentLog

type IncidentLog interface {
	Incidents() []Incident
}

IncidentLog exposes the incident history for the dashboard. Implemented by the watchdog. Returned slices are copies, most recent first.

type IncidentType

type IncidentType string

IncidentType classifies watchdog incidents.

const (
	// IncidentPeerDead is raised by the leader when a peer transitions
	// to StatusDead, or when a newly elected leader first observes an
	// already-dead peer.
	IncidentPeerDead IncidentType = "peer_dead"
	// IncidentPeerRecovered is raised by the leader when a peer that had
	// a peer_dead incident becomes StatusAlive again.
	IncidentPeerRecovered IncidentType = "peer_recovered"
)

type MemberKind

type MemberKind string

MemberKind classifies a node's relationship to the voting cluster.

const (
	// MemberVoter is a full member: counted in quorum, may vote and lead.
	MemberVoter MemberKind = "voter"
	// MemberObserver is an identify-verified warden node awaiting
	// admission: it receives heartbeats and views but never votes, never
	// counts toward quorum, and never starts elections.
	MemberObserver MemberKind = "observer"
	// MemberDiscovered is a node reported by the discovery source that has
	// not (yet) been identify-verified as part of this cluster.
	MemberDiscovered MemberKind = "discovered"
)

type Membership

type Membership struct {
	// Version increases by one per membership change.
	Version uint64 `json:"version"`
	// CreatedInTerm is the term of the leader that minted this membership.
	// It disambiguates sibling configurations: a leader that persists a new
	// version and is deposed before disseminating it leaves a config with
	// the same Version as the one the next leader mints. Identity is the
	// (Version, CreatedInTerm) pair — see Supersedes.
	CreatedInTerm Term `json:"created_in_term"`
	// Voters is the full voting member set, sorted by ID.
	Voters []Node `json:"voters"`
}

Membership is the effective voting configuration. It is persisted, changed ONLY by the leader, strictly one node at a time (single-server changes keep any old-majority/new-majority pair overlapping, which preserves election safety), and disseminated via heartbeats. Quorum is ALWAYS computed over Voters — never over a discovery roster and never over "currently reachable peers" — so unreachability can never shrink the quorum denominator.

func (Membership) Clone

func (m Membership) Clone() Membership

Clone deep-copies the Membership (its Voters slice) so a snapshot can be persisted, disseminated, or handed out without aliasing the owner's state.

func (Membership) HasVoter

func (m Membership) HasVoter(id NodeID) bool

HasVoter reports whether id is in the voting set.

func (Membership) Supersedes

func (m Membership) Supersedes(other Membership) bool

Supersedes reports whether m is strictly newer than other under the lexicographic (Version, CreatedInTerm) order. Adoption uses this — never a bare Version comparison — so a higher-term leader's config replaces a stale sibling of equal Version, and ack accounting can distinguish a node holding a divergent config from one holding the config being settled.

type Node

type Node struct {
	ID NodeID `json:"id" yaml:"id"`
	// Addr is the host:port the node's warden HTTP server listens on,
	// reachable over the tailnet (e.g. "203.0.113.10:7717").
	Addr string `json:"addr" yaml:"addr"`
}

Node is a member of the static cluster peer set.

type NodeID

type NodeID string

NodeID uniquely identifies a node in the cluster (e.g. "node-a").

type Notifier

type Notifier interface {
	Notify(ctx context.Context, inc Incident) error
}

Notifier delivers a watchdog Incident to the operator. Production uses SMTP email; tests use a recording mock; the e2e harness uses a file sink. Implementations must be safe for concurrent use. Notify should return an error only on delivery failure; the watchdog logs failures and retries on the next evaluation (dedup still prevents duplicate notifications once one delivery succeeds).

type PeerDiscoverer

type PeerDiscoverer interface {
	Discover(ctx context.Context) (<-chan Roster, error)
}

PeerDiscoverer reports candidate cluster nodes. Discover returns a channel delivering roster snapshots until ctx ends (the implementation closes the channel when done and must send an initial snapshot promptly). Snapshots are advisory: the election manager's event loop consumes them, verifies candidates via Transport.Identify, and only the LEADER turns stable, verified candidates into one-at-a-time membership changes. When the discovery source is unavailable, implementations should keep the channel open and simply not send (consumers fall back to the last known roster and the persisted membership — never to an empty set).

type PeerStatus

type PeerStatus string

PeerStatus is the liveness classification of a peer, derived from how long ago it was last seen (heartbeat response for the leader; heartbeat receipt for followers observing the leader).

const (
	// StatusUnknown means the peer has never been seen since this
	// observer started.
	StatusUnknown PeerStatus = "unknown"
	// StatusAlive means the peer responded within SuspectAfter.
	StatusAlive PeerStatus = "alive"
	// StatusSuspect means no contact for at least SuspectAfter but less
	// than DeadAfter.
	StatusSuspect PeerStatus = "suspect"
	// StatusDead means no contact for at least DeadAfter. The leader's
	// watchdog raises an Incident on this transition.
	StatusDead PeerStatus = "dead"
)

type PeerView

type PeerView struct {
	Node   Node       `json:"node"`
	Status PeerStatus `json:"status"`
	// LastSeen is the Source's last successful contact with this peer
	// (zero time if never seen).
	LastSeen time.Time `json:"last_seen"`
	// LatencyMS is the most recent heartbeat round-trip time in
	// milliseconds as measured by the Source (0 if unknown).
	LatencyMS float64 `json:"latency_ms"`
	// Member classifies the node's membership standing (voter, observer,
	// discovered). Producers must always set it; an empty value is treated
	// as MemberVoter for backward compatibility.
	Member MemberKind `json:"member,omitempty"`
}

PeerView is one row of a ClusterView: a single peer as observed by the view's Source node.

type PersistentState

type PersistentState struct {
	CurrentTerm Term   `json:"current_term"`
	VotedFor    NodeID `json:"voted_for"`
	// Membership is the effective voting configuration, persisted so a
	// restart resumes with the same quorum denominator (nil in state files
	// written before membership support; the config seed applies then).
	Membership *Membership `json:"membership,omitempty"`
}

PersistentState is the durable election state, per Raft: persisting the current term and the vote cast in it guarantees a node can never vote twice in the same term across restarts, which is what makes a majority quorum imply at most one leader per term.

type RPCHandler

type RPCHandler interface {
	HandleVote(ctx context.Context, req VoteRequest) VoteResponse
	HandleHeartbeat(ctx context.Context, req HeartbeatRequest) HeartbeatResponse
	// HandleIdentify serves the cluster-identity handshake.
	HandleIdentify(ctx context.Context) IdentifyResponse
}

RPCHandler is the server side of the wire protocol, implemented by the election manager and served as the gRPC WardenService by services/warden/grpcserver (multiplexed onto the single node port with the HTTP surface by services/warden/grpcmux). Implementations must be safe for concurrent use.

type Role

type Role string

Role is a node's current role in the election state machine.

const (
	RoleFollower  Role = "follower"
	RoleCandidate Role = "candidate"
	RoleLeader    Role = "leader"
)

type Roster

type Roster struct {
	Nodes []Node `json:"nodes"`
}

Roster is a discovery snapshot: candidate cluster nodes as reported by a PeerDiscoverer. It is advisory only — it never directly changes voting membership.

type Store

type Store interface {
	Save(st PersistentState) error
	Load() (st PersistentState, ok bool, err error)
}

Store persists PersistentState. Save must be atomic and durable before returning (write-then-rename for the file implementation). Load returns ok == false when no state has ever been saved.

type Term

type Term uint64

Term is a monotonically increasing election term, Raft-style. A node persists its current term (and vote) via Store so terms never regress across restarts.

type Ticker

type Ticker interface {
	C() <-chan time.Time
	Stop()
}

Ticker mirrors time.Ticker behind an interface.

type Timer

type Timer interface {
	C() <-chan time.Time
	// Stop prevents the timer from firing; it reports whether it stopped
	// a pending fire (same semantics as time.Timer.Stop).
	Stop() bool
	// Reset re-arms the timer with duration d (same semantics as
	// time.Timer.Reset: only call on stopped/drained timers).
	Reset(d time.Duration) bool
}

Timer mirrors time.Timer behind an interface.

type Transport

type Transport interface {
	// RequestVote sends a VoteRequest to peer and returns its response.
	// An error means the peer was unreachable or replied malformed; the
	// caller treats it as a vote not granted.
	RequestVote(ctx context.Context, peer Node, req VoteRequest) (VoteResponse, error)
	// SendHeartbeat sends a HeartbeatRequest to peer and returns its
	// response. An error means the peer was unreachable; the leader's
	// liveness tracker records the failed contact.
	SendHeartbeat(ctx context.Context, peer Node, req HeartbeatRequest) (HeartbeatResponse, error)
	// Identify asks peer for its cluster identity (used to verify that a
	// discovered node is a warden of the same cluster before treating it
	// as an observer). An error means unreachable or not a warden.
	Identify(ctx context.Context, peer Node) (IdentifyResponse, error)
}

Transport sends cluster RPCs to peers. The production implementation (services/warden/grpctransport) carries them over the gRPC WardenService (h2c on the tailnet); tests substitute in-memory fakes to simulate partitions, delays, and node death.

type ViewSource

type ViewSource interface {
	// View returns the current cluster view snapshot. Safe for
	// concurrent use; the returned value is a copy the caller may keep.
	View() ClusterView
	// Subscribe returns a channel that receives view snapshots after
	// state changes (role/term/leader changes, peer status transitions).
	// Delivery is best-effort: when the buffered channel is full,
	// intermediate updates are dropped, so consumers should treat a
	// receive as a change signal and may re-read View() for the latest
	// state. cancel unsubscribes and closes the channel.
	Subscribe(buf int) (ch <-chan ClusterView, cancel func())
}

ViewSource provides cluster view snapshots and change notifications. Implemented by the election manager; consumed by the watchdog, dashboard, and metrics packages.

type VoteRequest

type VoteRequest struct {
	Term        Term   `json:"term"`
	CandidateID NodeID `json:"candidate_id"`
}

VoteRequest asks a peer for its vote in CandidateID's election for Term.

type VoteResponse

type VoteResponse struct {
	Term    Term   `json:"term"`
	Granted bool   `json:"granted"`
	VoterID NodeID `json:"voter_id"`
}

VoteResponse is the reply to a VoteRequest. Term is the voter's current term after processing the request (a candidate seeing a higher term must step down to follower and adopt it).