README
¶
pagurus — one copy of a service, failing over across an Incus cluster
A hermit crab lives in one shell at a time and moves when it has to. pagurus keeps a container
workload (an incus-compose stack, or plain Incus containers) running on at most one member of an
Incus cluster at any instant, and on some member whenever a quorum is up. When the member
running it dies, hangs or is cut off, the copy there is fenced before another member starts its own.
It is for clusters without shared storage, where Incus' own cluster healing cannot move an instance off a dead member. Each member keeps its own copy of the stack in its own project; pagurus decides which one may run.
etcd (3 nodes) the office: a conch election on a lease
▲ ▲ ▲
┌─────────────┴─┐ ┌────┴──────┐ ┌┴──────────────┐
│ pagurus@node1 │ │ @node2 │ │ @node3 │ one agent per member, under systemd's watchdog
│ leads: runs │ │ standby │ │ standby │
└──────┬────────┘ └───────────┘ └───────────────┘
│ incus-compose up / incus stop / cgroup.kill
project whoami-node1 whoami-node2 whoami-node3 (each pinned to its own member)
The leader holds a lease in etcd (conch runs the election).
Before its lease can expire it stops its copy; if its agent is gone, peers stop that copy through the
Incus API, and an optional watchdog reboots a host whose agent wedges. The safety argument, and the
gap it leaves, are in docs/00_design.md.
What it assumes — read these before relying on it:
- at most one member impaired at a time, and a working etcd quorum;
- containers only for the self-fence's last resort (
cgroup.kill); VMs only with theinstancesstarter and an Incus-API fence; - stateless workloads, or the opt-in volume replication in
docs/02_replication.md, which can lose the writes since the last sync; - a residual gap: with no reboot-armed watchdog and
fence.unverified=proceed, a leader cut off from both etcd and Incus whose agent is also dead, on a host that stays up, can overlap a new leader.docs/00_design.md§ The residual gap states it exactly.
Requirements
- An Incus cluster of ≥ 3 members (or 2 members plus an etcd member elsewhere as a witness), cgroup2, systemd.
- etcd v3, three nodes.
- conch on
PATH(the version ingo.mod), and incus-compose for the compose starter.
Try it
lab/ builds a throwaway three-member cluster on one Incus host and runs the
failure drills against it, refereed by pagurus audit:
lab/up.sh && lab/drill.sh # ~3 min up, ~4 min of drills; lab/down.sh removes it
The same lab runs in CI on every push (.github/workflows/lab.yml).
Docs
00_design.md |
process shape, the safety argument, the residual gap, timing, architecture |
01_operations.md |
provisioning, reaching Incus and etcd, starters, updates, metrics, the CLI |
02_replication.md |
stateful services: replicated volumes and their limits |
03_fencing.md |
fence policies, VM and peer fences, watchdog modes |
04_comparison.md |
vs Incus cluster healing, Proxmox HA, Pacemaker, keepalived, Nomad |
05_results.md |
drill results from the author's cluster |
06_lab.md |
the lab |
Status
In production on the author's three-member cluster since October 2026 (stateless and replicated
services). Releases: GitHub releases (static
linux amd64/arm64). Interfaces may still change before 1.0; see CHANGELOG.md.
License: see LICENSE. Security reports: SECURITY.md.
Directories
¶
| Path | Synopsis |
|---|---|
|
cmd
|
|
|
blackhole-proxy
command
Command blackhole-proxy is a drill tool, never part of a real deployment: it forwards local ports to etcd endpoints so one agent can be cut off from etcd without touching the host firewall.
|
Command blackhole-proxy is a drill tool, never part of a real deployment: it forwards local ports to etcd endpoints so one agent can be cut off from etcd without touching the host firewall. |
|
pagurus
command
Command pagurus runs one incus-compose stack on at most one member of an Incus cluster, failing it over between members.
|
Command pagurus runs one incus-compose stack on at most one member of an Incus cluster, failing it over between members. |
|
internal
|
|
|
agent
Package agent is the per-member daemon.
|
Package agent is the per-member daemon. |
|
backend
Package backend holds what the core knows about a workload without knowing how it runs: an instance, as a member's site reports it.
|
Package backend holds what the core knows about a workload without knowing how it runs: an instance, as a member's site reports it. |
|
backend/incus
Package incus is the runtime driver: it runs the stack with incus-compose in this member's project, and stops (fences) a member's project through the Incus cluster API or, locally, through cgroup.kill.
|
Package incus is the runtime driver: it runs the stack with incus-compose in this member's project, and stops (fences) a member's project through the Incus cluster API or, locally, through cgroup.kill. |
|
backend/incus/provision
Package provision prepares the Incus side of a pagurus service: one cluster group per member holding exactly that member (host-<member>), and one project per member pinned to it (restricted.cluster.groups), and, for a service with a replicate block, each replicated custom volume in every member's project on that member.
|
Package provision prepares the Incus side of a pagurus service: one cluster group per member holding exactly that member (host-<member>), and one project per member pinned to it (restricted.cluster.groups), and, for a service with a replicate block, each replicated custom volume in every member's project on that member. |
|
backend/incus/start
Package start holds the ways a member brings its copy of the workload up in its own pinned project.
|
Package start holds the ways a member brings its copy of the workload up in its own pinned project. |
|
config
Package config loads and validates a pagurus service definition.
|
Package config loads and validates a pagurus service definition. |
|
coord
Package coord holds the etcd schema of one pagurus service.
|
Package coord holds the etcd schema of one pagurus service. |
|
fence
Package fence holds pagurus's fences: the ways one member's copy of the workload is stopped, each a plugin with its own budget, run as an ordered chain that stops at the first fence that confirms.
|
Package fence holds pagurus's fences: the ways one member's copy of the workload is stopped, each a plugin with its own budget, run as an ordered chain that stops at the first fence that confirms. |
|
fence/fencetest
Package fencetest is a fake environment for testing fences without a real Incus, cgroupfs or host: fake binaries on a temp PATH, a fake cgroup2 tree, a real child process with a pidfile, and a contract every fence kind must pass.
|
Package fencetest is a fake environment for testing fences without a real Incus, cgroupfs or host: fake binaries on a temp PATH, a fake cgroup2 tree, a real child process with a pidfile, and a contract every fence kind must pass. |
|
metrics
Package metrics is pagurus's Prometheus exposition: a few gauges and counters, written by hand in the text format (no client library).
|
Package metrics is pagurus's Prometheus exposition: a few gauges and counters, written by hand in the text format (no client library). |
|
replica
Package replica holds the pure rules of volume replication (docs/02_replication.md): how pagurus names its snapshots, which ones it may prune, and rule 4, whether a member's volume holds writes the source lacks.
|
Package replica holds the pure rules of volume replication (docs/02_replication.md): how pagurus names its snapshots, which ones it may prune, and rule 4, whether a member's volume holds writes the source lacks. |
|
testutil
Package testutil runs throwaway etcds (one member, or a cluster) and a TCP proxy that can blackhole traffic to simulate a network partition.
|
Package testutil runs throwaway etcds (one member, or a cluster) and a TCP proxy that can blackhole traffic to simulate a network partition. |
|
versions
Package versions records which Incus and incus-compose an agent drives, and warns when they are outside what pagurus was built and tested against.
|
Package versions records which Incus and incus-compose an agent drives, and warns when they are outside what pagurus was built and tested against. |
|
watchdog
Package watchdog turns "this agent is a fit custodian of the workload" into a watchdog pet (the sbd / Proxmox-LRM pattern, stonith_exploration.md §7).
|
Package watchdog turns "this agent is a fit custodian of the workload" into a watchdog pet (the sbd / Proxmox-LRM pattern, stonith_exploration.md §7). |
Click to show internal directories.
Click to hide internal directories.