README
¶
Sproutfs
Sproutfs is storage and managed memory for virtual machines. The VMs can move between hosts and fork without copying their inherited disk and memory data. Sproutfs combines whole-VM checkpoints in object storage with a shared memory pager and a Firecracker integration.
The project is under active development. It provides the host and orchestrator
commands that the demo deployment runs, and the Rust library through which the
VMM maps its guest memory. A program that embeds a host can import the runtime
packages at the module root (host, volume, vmmemory, vmmigrate,
vmmachine, api/host and what they build on). Helpers and test harnesses are
under internal. No production deployment is recorded yet.
There is no garbage collector, and the collector is deferred indefinitely. Until one exists, the object store grows without bound. It keeps these objects permanently:
- every checkpoint at which a VM was ever forked;
- the checkpoints that checkpoint's index names;
- the pinned checkpoints that a deleted VM leaves behind.
Deleting a VM reclaims only checkpoints that no fork was taken from. See TASK-24 in the backlog.
What it does
- Serves VM volume reads and writes from memory. It makes them durable with a periodic whole-VM checkpoint: pause the vCPUs, save VMM state, seal the dirty pages by write protection, resume, then upload the sealed pages and an index.
- Forks a running VM as a handoff from the parent. One pause gives one fork point, which serves any number of children on this host or on another host. Taking the fork point publishes nothing.
- Shares resident memory pages of the same identity within a pager.
- Moves a running VM between hosts post-copy. The destination resumes first and pulls the pages that no checkpoint holds from the source's peer server.
A VM's durable state is one checkpoint, which the VM's control record in the object store selects. A guest write never waits on object-store latency. Losing a host rewinds its VMs' disks to their last checkpoint. That is normally at most one checkpoint interval old (60 seconds by default), and the loss window (five minutes by default) bounds it when publishing fails. The guest then cold boots, because an interval checkpoint holds no RAM. Read the loss model before integrating.
See it run
One disposable GCE VM runs a single-node cluster with two host pods. Five flows run against it non-interactively: boot, fork, live migration, recovery after a host is killed, and a command run inside a guest across a fork and a move.
scripts/demo-gce.sh create
scripts/demo-gce.sh run
scripts/demo-gce.sh delete
See the demo guide for each command and its timings.
Build and test
Install Go 1.26 or newer, then run from the repository root:
go build ./...
go test ./...
go test -race ./...
The core Go tests use simulated disks, networks and object storage and need no cloud account. The Linux VM integration tests need KVM and userfaultfd, among other things. Follow the managed-memory qualification guide for those tests and the Firecracker build.
If you change protocol definitions, install Buf and run just generate
(or buf lint followed by buf generate). Do not edit generated files.
Documentation
- Architecture: components, the loss model and the VM lifecycle.
- Volumes and checkpoints: the dirty overlay, publication and forks.
- Host assembly: assembly, draining, budgets and shutdown.
- Managed VM memory: pager and Firecracker integration.
- Live migration: post-copy migration and fork handoff.
- Testing: simulation, failure injection and integration tests.
- The demo: boot, fork, migration and host loss on one GCE VM.
See the documentation index for the full guide.
License
Apache-2.0; see LICENSE. The Firecracker fork under
third_party/firecracker keeps its own Apache-2.0 license.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
guest
Package guest is the channel between a host and the agent inside one of its guests: the JSON the agent serves, and the dialer that reaches it.
|
Package guest is the channel between a host and the agent inside one of its guests: the JSON the agent serves, and the dialer that reaches it. |
|
host
Package host is the wire protocol of the demo host binary: the JSON a sproutfs-host serves on its API port, and the client the orchestrator and a draining host reach it with.
|
Package host is the wire protocol of the demo host binary: the JSON a sproutfs-host serves on its API port, and the client the orchestrator and a draining host reach it with. |
|
orch
Package orch is the wire protocol of the demo orchestrator: what the CLI asks for, what a draining host asks for, and the client both reach it with.
|
Package orch is the wire protocol of the demo orchestrator: what the CLI asks for, what a draining host asks for, and the client both reach it with. |
|
Package checkpoint stores VM checkpoints in shared object storage as one index object of metadata and the parts holding the guest bytes.
|
Package checkpoint stores VM checkpoints in shared object storage as one index object of metadata and the parts holding the guest bytes. |
|
internal/part
Package part lays out one checkpoint's parts, which are its data: the VMM state and the guest pages it published.
|
Package part lays out one checkpoint's parts, which are its data: the VMM state and the guest pages it published. |
|
cmd
|
|
|
sproutfs-guest-agent
command
Command sproutfs-guest-agent runs inside a demo guest.
|
Command sproutfs-guest-agent runs inside a demo guest. |
|
sproutfs-guest-chase
command
Command sproutfs-guest-chase loads a Valkey server in a guest with data whose reads depend on each other, and later walks it one request at a time, timing each request.
|
Command sproutfs-guest-chase loads a Valkey server in a guest with data whose reads depend on each other, and later walks it one request at a time, timing each request. |
|
sproutfs-guest-witness
command
Command sproutfs-guest-witness is what a guest answers with when it is asked whether its memory and its disk are what it wrote.
|
Command sproutfs-guest-witness is what a guest answers with when it is asked whether its memory and its disk are what it wrote. |
|
sproutfs-host
command
Command sproutfs-host runs VMs.
|
Command sproutfs-host runs VMs. |
|
sproutfs-orchestrator
command
Command sproutfs-orchestrator drives the demo.
|
Command sproutfs-orchestrator drives the demo. |
|
sproutfs-peerbench
command
Command sproutfs-peerbench measures a guest fault's latency while a post-copy stream fills the link between two hosts, and stripe reads beside it, over the peer server (docs/migration.md#the-peer-server).
|
Command sproutfs-peerbench measures a guest fault's latency while a post-copy stream fills the link between two hosts, and stripe reads beside it, over the peer server (docs/migration.md#the-peer-server). |
|
sproutfs-publishbench
command
Command sproutfs-publishbench measures one publication of a large guest's memory to the object store: what a `stop --suspend` of a guest with a multi-GiB heap spends uploading it (docs/measurements/gce-publication-throughput-2026-10-04.md).
|
Command sproutfs-publishbench measures one publication of a large guest's memory to the object store: what a `stop --suspend` of a guest with a multi-GiB heap spends uploading it (docs/measurements/gce-publication-throughput-2026-10-04.md). |
|
sproutfs-restorebench
command
Command sproutfs-restorebench measures a guest's memory read back on another host from the cluster's disk cache and from the object store (plans/disk-cache-2026-10-02.md, "How it is proved", measurement 3).
|
Command sproutfs-restorebench measures a guest's memory read back on another host from the cluster's disk cache and from the object store (plans/disk-cache-2026-10-02.md, "How it is proved", measurement 3). |
|
sproutfs-shardbench
command
Command sproutfs-shardbench times a shard of the cluster's cache moving between hosts on Compute Engine (docs/hosting.md, "Shards on network disks"), with the real host, membership and attach API.
|
Command sproutfs-shardbench times a shard of the cluster's cache moving between hosts on Compute Engine (docs/hosting.md, "Shards on network disks"), with the real host, membership and attach API. |
|
sproutfs-startbench
command
Command sproutfs-startbench measures how long a VM takes to start: from the request to the guest running, and to the guest's agent answering, for a cold start, a restore, a fork on the same host and a fork to another host.
|
Command sproutfs-startbench measures how long a VM takes to start: from the request to the guest running, and to the guest's agent answering, for a cold start, a restore, a fork on the same host and a fork to another host. |
|
sproutfs-stripebench
command
Command sproutfs-stripebench measures whether reading k of k+m stripes from several servers beats reading one whole object from one server.
|
Command sproutfs-stripebench measures whether reading k of k+m stripes from several servers beats reading one whole object from one server. |
|
sproutfsctl
command
Command sproutfsctl drives a sproutfs demo deployment: create a VM, attach to its console, fork it, move it between hosts, kill a host and recover the VMs it was running.
|
Command sproutfsctl drives a sproutfs demo deployment: create a VM, attach to its console, fork it, move it between hosts, kill a host and recover the VMs it was running. |
|
Package control holds the one mutable durable object a VM owns: the record that selects which published checkpoint the VM's state is.
|
Package control holds the one mutable durable object a VM owns: the record that selects which published checkpoint the VM's state is. |
|
Package host is everything one host of the deployment does: it opens VMs from their control records, runs their VMM processes, checkpoints them on an interval, captures and forks them, hands them to other hosts and takes them over.
|
Package host is everything one host of the deployment does: it opens VMs from their control records, runs their VMM processes, checkpoints them on an interval, captures and forks them, hands them to other hosts and takes them over. |
|
internal
|
|
|
blob
Package blob encodes independently readable, bounded raw or Zstandard blobs.
|
Package blob encodes independently readable, bounded raw or Zstandard blobs. |
|
ctxsync
Package ctxsync provides context-aware synchronization primitives built on channels.
|
Package ctxsync provides context-aware synchronization primitives built on channels. |
|
handover
Package handover is how a deployment carries one handoff to a destination that takes it.
|
Package handover is how a deployment carries one handoff to a destination that takes it. |
|
jsonhttp
Package jsonhttp is the JSON transport the demo control plane is made of: one request shape, one error shape, and one client call.
|
Package jsonhttp is the JSON transport the demo control plane is made of: one request shape, one error shape, and one client call. |
|
knobs
Package knobs is the deployment's tunables in one place: the size a checkpoint part fills to, the budgets a pager admits pages against, the intervals a host checkpoints and re-reads its records on, and the bounds a drain and a publication work within.
|
Package knobs is the deployment's tunables in one place: the size a checkpoint part fills to, the budgets a pager admits pages against, the intervals a host checkpoints and re-reads its records on, and the bounds a drain and a publication work within. |
|
latency
Package latency is latency instrumentation: the live histograms the fault path writes without taking the host lock, and the snapshots a stats record carries.
|
Package latency is latency instrumentation: the live histograms the fault path writes without taking the host lock, and the snapshots a stats record carries. |
|
simtest
Package simtest generates a deployment and the failures that happen to it from one seed.
|
Package simtest generates a deployment and the failures that happen to it from one seed. |
|
testarena
Package testarena is the arena mode a test suite builds its pagers in.
|
Package testarena is the arena mode a test suite builds its pagers in. |
|
testbacking
Package testbacking wraps a volume so a simulated run decides the order in which memory regions reach it.
|
Package testbacking wraps a volume so a simulated run decides the order in which memory regions reach it. |
|
testjail
Package testjail runs a copy of the test binary as another user, which is how a test plays a jailed VMM.
|
Package testjail runs a copy of the test binary as another user, which is how a test plays a jailed VMM. |
|
testnet
Package testnet provides a loopback platform.Network for tests that need production TCP framing rather than a simulated transport.
|
Package testnet provides a loopback platform.Network for tests that need production TCP framing rather than a simulated transport. |
|
testpager
Package testpager is a simulated arena and a simulated process memory region for suites that run a pager without a kernel: the simulation's campaigns, and the host and migration suites.
|
Package testpager is a simulated arena and a simulated process memory region for suites that run a pager without a kernel: the simulation's campaigns, and the host and migration suites. |
|
testrepro
Package testrepro checks scheduled workloads across independent Go processes.
|
Package testrepro checks scheduled workloads across independent Go processes. |
|
testresource
Package testresource supplies roomy host resource allotments for tests whose subject is another invariant.
|
Package testresource supplies roomy host resource allotments for tests whose subject is another invariant. |
|
testsoak
Package testsoak runs one seed of a soak campaign and reports what it cost.
|
Package testsoak runs one seed of a soak campaign and reports what it cost. |
|
vmtest
Package vmtest qualifies the Rust memory client against a real Linux Go pager fixture.
|
Package vmtest qualifies the Rust memory client against a real Linux Go pager fixture. |
|
vmwire
Package vmwire carries the host side of the managed-memory control protocol described in docs/vm-memory.md: fixed 56-byte little-endian frames over a Unix socket, SCM_RIGHTS descriptor passing, sealed memfd arenas, and the userfaultfd ioctls that resolve one fault.
|
Package vmwire carries the host side of the managed-memory control protocol described in docs/vm-memory.md: fixed 56-byte little-endian frames over a Unix socket, SCM_RIGHTS descriptor passing, sealed memfd arenas, and the userfaultfd ioctls that resolve one fault. |
|
Package journal is a host's write-ahead log of flushed blocks, kept on a network disk of its own (plans/fsync-journal-2026-10-06.md).
|
Package journal is a host's write-ahead log of flushed blocks, kept on a network disk of its own (plans/fsync-journal-2026-10-06.md). |
|
Package membership is which hosts are in the cluster, and which cache disk each one serves: one object in the object store, the membership.
|
Package membership is which hosts are in the cluster, and which cache disk each one serves: one object in the object store, the membership. |
|
Package peer is the peer server: the one channel hosts talk to each other over.
|
Package peer is the peer server: the one channel hosts talk to each other over. |
|
internal/previous
Package previous is the peer protocol as the release before this one spoke it, frozen: main at 6af0d3de, when the channel was the page server.
|
Package previous is the peer protocol as the release before this one spoke it, frozen: main at 6af0d3de, when the channel was the page server. |
|
internal/wire
Package wire encodes typed protobuf control messages and keeps optional bulk data in a separate raw payload frame.
|
Package wire encodes typed protobuf control messages and keeps optional bulk data in a separate raw payload frame. |
|
peertest
Package peertest reads and rewrites peer-server frames for tests that stand between a host and the wire: a reply held back, a reply turned into BUSY, a reply whose bytes are damaged on the way.
|
Package peertest reads and rewrites peer-server frames for tests that stand between a host and the wire: a reply held back, a reply turned into BUSY, a reply whose bytes are damaged on the way. |
|
adapters
Package adapters constructs the platform adapters that talk to real machines: the host filesystem, TCP, and the object stores a deployment runs on.
|
Package adapters constructs the platform adapters that talk to real machines: the host filesystem, TCP, and the object stores a deployment runs on. |
|
bounded
Package bounded bounds every request a process makes of an object store.
|
Package bounded bounds every request a process makes of an object store. |
|
internal/framer
Package framer turns byte streams into the framed connections of platform.Network.
|
Package framer turns byte streams into the framed connections of platform.Network. |
|
sim
Package sim provides in-memory platform adapters intended to be constructed and used entirely inside a testing/synctest bubble.
|
Package sim provides in-memory platform adapters intended to be constructed and used entirely inside a testing/synctest bubble. |
|
Package rank says which cache disks hold a window's stripes.
|
Package rank says which cache disks hold a window's stripes. |
|
Package resource accounts reservations against one host allotment of RAM, which is what the pager's resident pages and the page cache's retained objects are taken from.
|
Package resource accounts reservations against one host allotment of RAM, which is what the pager's resident pages and the page cache's retained objects are taken from. |
|
Package stripe splits an envelope into the stripes of an erasure code and rebuilds it from any k of them.
|
Package stripe splits an envelope into the stripes of an erasure code and rebuilds it from any k of them. |
|
Package vmmachine drives Firecracker processes over host-managed volumes.
|
Package vmmachine drives Firecracker processes over host-managed volumes. |
|
Package vmmemory manages bounded shared backing for mapped volume memory regions.
|
Package vmmemory manages bounded shared backing for mapped volume memory regions. |
|
internal/pageranges
Package pageranges is the interval map the Linux connection keeps its client's mapping generations in: a state constant over half-open runs of pages, held so that replacing one run costs the depth of its boundaries rather than its pages.
|
Package pageranges is the interval map the Linux connection keeps its client's mapping generations in: a state constant over half-open runs of pages, held so that replacing one run costs the depth of its boundaries rather than its pages. |
|
internal/slots
Package slots is the pager's arena addresses: which of its offsets hold no page, the consecutive runs of them that let consecutive pages become one mapping command, the extents a memory region places its private pages in, and how many pages it may hold at once.
|
Package slots is the pager's arena addresses: which of its offsets hold no page, the consecutive runs of them that let consecutive pages become one mapping command, the extents a memory region places its private pages in, and how many pages it may hold at once. |
|
internal/zirconvm
Package zirconvm is a Go port of the page layer of Zircon's VM, which is replacing the pager's own (plans/zircon-pager-port-2026-10-05.md).
|
Package zirconvm is a Go port of the page layer of Zircon's VM, which is replacing the pager's own (plans/zircon-pager-port-2026-10-05.md). |
|
Package vmmigrate moves a running VM between hosts with a pause of the VMM state capture and nothing else.
|
Package vmmigrate moves a running VM between hosts with a pause of the VMM state capture and nothing else. |
|
Package volume serves a VM's writable volumes from the one checkpoint its control record selects.
|
Package volume serves a VM's writable volumes from the one checkpoint its control record selects. |
Click to show internal directories.
Click to hide internal directories.