fleetbox

README

fleetbox

Real Linux VMs as Go test fixtures — on macOS and Linux.

fleetbox boots stock Linux cloud images and hands them to your tests over SSH. On macOS (Apple Silicon) it drives Apple's Virtualization.framework; on Linux, cloud-hypervisor. The Go API is identical on both. Think testcontainers, except instead of a container you get a whole machine: real kernel, real systemd, real /dev/kvm.

func TestAgainstRealLinux(t *testing.T) {
	vm := fleetboxtest.Start(t, "debian-12")

	out, err := vm.SSH(context.Background(), "uname -a")
	if err != nil {
		t.Fatal(err)
	}
	t.Log(out) // Linux ... aarch64 GNU/Linux
}

One line gets you a booted Debian box, reachable over SSH with a keypair fleetbox generates itself (it never touches your ~/.ssh). When the test returns, the VM is destroyed.

Status: v0. It works, but the API will change and there are no compatibility promises yet. See Limitations.

Why

Containers are wonderful right up until you need to test something a container can't give you: a kernel module, a systemd unit, an nftables ruleset, kubeadm, a VPN, anything that wants /dev/kvm. The usual fallback is a VM tool that arrives with a yaml dialect, a background agent forwarding SSH ports, and patched images. That is a lot of machinery, and all of it has to keep working.

fleetbox keeps the machinery small and refuses to grow it:

• Real VMs, not containers. It EFI-boots unmodified cloud images through their own bootloader; on M3+ you also get nested virtualization, so KVM works inside the guest.
• Every VM gets a routable IP. A vmnet SharedMode network on macOS 26+, a shared bridge with static addresses on Linux. No port forwarding to set up, no tunnel daemon to babysit: call vm.IP() and connect.
• Nothing of ours runs in the guest. No agent, no helper binary, no host/guest protocol. cloud-init configures the VM once at first boot; after that it's a plain distro you reach over SSH.
• Library-first. The Go package is the product; the CLI is a thin wrapper over the same calls. Test fixtures clean themselves up through t.Cleanup.

No yaml, no templates, no per-distro code paths. Flags and sane defaults.

Requirements

• macOS on Apple Silicon. Clusters (VM↔VM networking) need macOS 26+; below that you get a single VM at a time. Nested virtualization (/dev/kvm in the guest) needs M3 or newer. Intel Macs are not supported.
• Linux, amd64 or arm64. Needs /dev/kvm (be in the kvm group) and CAP_NET_ADMIN for the bridge and taps. fleetbox checks both before booting anything and fails with a one-line fix, not a cryptic boot error.

Plus Go 1.24+. The module compiles on darwin/arm64 and linux/{amd64,arm64}; other targets build but return a clear "unsupported platform" error at runtime.

Install

go get github.com/pilat/fleetbox

That's the whole install: nothing to build, nothing to codesign. Your test binary stays pure Go. On macOS, all the Virtualization.framework work lives in a small signed fleetbox-helper that fleetbox downloads, checksum-pinned, into ~/.fleetbox/bin/ the first time you boot a VM. Linux is the same story with different binaries: the cloud-hypervisor VMM and its firmware are fetched and pinned on first use.

Prefer the CLI to the library? On macOS (Apple Silicon) it ships as a Homebrew cask:

brew tap pilat/fleetbox
brew install --cask fleetbox

On Linux (Homebrew has no casks there), install it straight from source:

go install github.com/pilat/fleetbox/cmd/fleetbox@latest

Either way you get the pure-Go fleetbox binary; the helper and VMM still auto-download on first boot, exactly as they do for the library.

The first boot also downloads the cloud image (a few hundred MB, cached in ~/.fleetbox/images/) and prints a progress line so it doesn't look like a hung test.

Hacking on fleetbox itself, or running on an air-gapped host? Build the helper locally and point fleetbox at it:

make helper                                    # builds + ad-hoc-signs ./bin/fleetbox-helper
FLEETBOX_HELPER=$PWD/bin/fleetbox-helper go test ./...

Usage

As a test fixture

import (
	"context"
	"testing"

	"github.com/pilat/fleetbox"
	"github.com/pilat/fleetbox/fleetboxtest"
)

func TestNeedsARealKernel(t *testing.T) {
	vm := fleetboxtest.Start(t, "debian-12",
		fleetbox.WithCPUs(2),
		fleetbox.WithMemoryGB(4),
	)

	// What no container can give you: systemd as PID 1, your own kernel,
	// a working /dev/kvm.
	out, err := vm.SSH(context.Background(),
		"cat /proc/1/comm && uname -r && test -e /dev/kvm && echo ok")
	if err != nil {
		t.Fatalf("%v\n%s", err, out)
	}
	t.Log(out) // systemd / 6.1.0-… / ok
}

fleetboxtest.Start registers t.Cleanup to destroy the VM, derives a collision-safe name from the test name so parallel tests don't fight, and skips the test automatically when the hardware can't run it. SkipIfShort opts a test out under go test -short. StartN boots several VMs on one shared network — a real cluster whose members reach each other by IP (see Limitations for where clustering is supported).

As a library (no testing.T)

vm, err := fleetbox.Start(ctx, "builder",
	fleetbox.WithImage("ubuntu-24.04"),
	fleetbox.WithCPUs(4),
	fleetbox.WithMemoryGB(8),
	fleetbox.WithDiskGB(40),
)
if err != nil {
	log.Fatal(err)
}

fmt.Println(vm.IP())              // net.IP — directly reachable, no forwarding

out, err := vm.SSH(ctx, "sudo apt-get install -y nginx")  // user has passwordless sudo
if err != nil {
	log.Fatalf("%v\n%s", err, out)
}

_ = vm.Stop(ctx)                  // graceful shutdown, disk preserved
// vm.Destroy(ctx) deletes it entirely

Start is idempotent: call it again with the same name and it boots the existing VM instead of recreating it. State lives under ~/.fleetbox/clusters/<cluster>/<name>/ and survives reboots; Destroy (or fleetbox rm) is the only thing that deletes a disk. Full API on pkg.go.dev.

Handing a VM a directory

WithFixture packs a host directory into the guest as a read-only fixture — the way to hand a VM your test data, config, or build output. It works identically on macOS and Linux: at boot the directory is snapshotted into an ext4 image, attached read-only, and mounted in the guest at the path you give:

dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "input.json"), payload, 0o644)

vm := fleetboxtest.Start(t, "debian-12", fleetbox.WithFixture(dir, "/work"))
out, _ := vm.SSH(context.Background(), "cat /work/input.json")  // reads the snapshot

From the CLI it's a repeatable --fixture host:guest flag:

./bin/fleetbox up dev --fixture ./src:/work --fixture ./fixtures:/data

Fixtures are read-only and world-readable: every file 0444, every directory 0555, owned by root. Host permission and exec bits are not preserved. The set of fixtures is frozen when the VM is first created (rm and recreate to change it), but the content is re-snapshotted from the host directory on every boot, so a reboot picks up host-side changes. To get data back out of the guest, use fleetbox cp or scp. The guest path must be absolute, and host paths must not contain colons (the value splits on the last colon).

From the command line

The CLI wraps the same library for manual work:

make build                                     # compiles ./bin/fleetbox (pure Go, nothing to sign)

./bin/fleetbox up web                          # create & boot a VM
./bin/fleetbox up node -n 3                    # boot a cluster: node-1, node-2, node-3
./bin/fleetbox ssh node-2                      # members are addressed by name
./bin/fleetbox ssh node-1 -- ping -c1 node-2   # ...and reach each other by IP
./bin/fleetbox cp ./mybinary web:/usr/local/bin/
./bin/fleetbox ls                              # NAME  IP  STATE  CPUS  MEM  DISK  AGE
./bin/fleetbox ssh-config >> ~/.ssh/config     # then plain `ssh web` works
./bin/fleetbox down node-1                     # stop one member; the rest keep running
./bin/fleetbox rm node                         # destroy the whole cluster (prefix match)
./bin/fleetbox version                         # print version, commit, build date

A cluster's VMs live in one holder process sharing one network (vmnet on macOS, a bridge on Linux), which is what lets them reach each other by IP; ssh/down/rm still address each member by name.

Images

Pass a built-in alias or any direct URL to a raw / qcow2 cloud image:

Alias	Image
debian-11	Debian 11 generic cloud (amd64/arm64, per host)
debian-12 (default)	Debian 12 generic cloud
debian-13	Debian 13 generic cloud
ubuntu-22.04	Ubuntu 22.04 server cloud
ubuntu-24.04	Ubuntu 24.04 server cloud
ubuntu-26.04	Ubuntu 26.04 server cloud

fleetboxtest.Start(t, "debian-12")
fleetboxtest.Start(t, "https://example.com/my-cloud-image.qcow2")

Each alias is pinned to a dated upstream snapshot and verified by SHA256 on download, so a given alias boots the same image on every machine and every run. A direct URL is taken as-is (unverified) — the escape hatch for a custom or bleeding-edge image. Images are downloaded and cached once in ~/.fleetbox/images/, with qcow2 converted to raw on the way in. Adding a distro is adding a catalog entry; there are no per-distro code paths.

How it works

Start is a short pipeline: make sure the SSH keypair exists, download and cache the image, write a cloud-init seed ISO, boot the VM, wait for SSH to answer. The platform-specific step is the IP. On macOS the VM's address is read out of /var/db/dhcpd_leases, looked up by hostname; on Linux fleetbox assigns a static address on the bridge and hands it to the guest through cloud-init.

Where that pipeline runs is the part worth knowing. On Linux it runs in your process. On macOS it runs inside the downloaded fleetbox-helper, the only binary that links Virtualization.framework. The library spawns the helper bound to the test process, so the helper and its VMs are reaped when the test exits, even on kill -9. The CLI does the opposite: each up group (a single VM, or a cluster sharing one network) gets a detached holder that outlives the command, which is what keeps VMs running between invocations. On macOS that holder is the helper; on Linux it's the CLI re-exec'd. Either way, SSH and cp dial the VM's IP directly — the helper protocol never proxies them.

For the full picture, read ARCHITECTURE.md; for why it's built this way, the decision log lives in docs/adr/.

Limitations

• Fixtures are read-only. There is no live read-write share: edits inside the guest don't flow back to the host, and host-side edits only show up after a reboot. The exact semantics are under Handing a VM a directory; for the guest→host direction, use cp / scp.
• A CLI cluster is one process. Members share one holder so they can share one network, so a holder crash takes the whole cluster down (a lone VM is unaffected). On Linux a SIGKILL'd holder also leaves its bridge and taps behind; the next up or down sweeps them. Members started by separate up commands sit on separate networks and can't be merged afterwards — bring a cluster up together.
• First run downloads, then caches. The cloud image (both platforms) and, on macOS, the signed helper are fetched once into ~/.fleetbox and reused. FLEETBOX_HELPER swaps in a locally built helper (development, offline). In CI, cache ~/.fleetbox/{bin,images} so cold runs don't re-download.
• Platform matrix. Clusters need macOS Apple Silicon 26+ or Linux amd64/arm64; macOS below 26 runs a single VM; Intel macOS is unsupported. On Linux, a stopped VM brought back up needs its /24 to still be free — on a contended host the auto-picked subnet can shift and the rebooted VM won't be reachable; bring clusters up fresh. arm64 Linux boot via rust-hypervisor-firmware is not yet validated on hardware.
• v0 API. Expect breaking changes until it stabilizes.

CI

GitHub-hosted macOS runners can't nest virtualization, so the macOS workflow does lint, build, and unit tests; VM-boot tests run locally via make test-vm. GitHub-hosted x86-64 Linux runners, on the other hand, expose /dev/kvm, so the Linux backend's VM-boot tests run in CI after a one-time udev tweak (plus a cache, so the image and VMM aren't re-downloaded every run):

- run: |
    echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666"' | sudo tee /etc/udev/rules.d/99-kvm.rules
    sudo udevadm control --reload-rules && sudo udevadm trigger
- uses: actions/cache@v4
  with:
    path: |
      ~/.fleetbox/bin
      ~/.fleetbox/images
    key: fleetbox-${{ runner.os }}

arm64 hosted Linux runners do not have KVM ("not supported for this sku"); use an x86-64 runner for VM-boot CI. This is the "develop on a Mac, test in cheap x86-64 hosted Linux CI" story.

Roadmap

Roughly in priority order:

• Programmatic file copy — a library-side copy in/out for cases a fixture doesn't fit (the CLI already has cp over scp).
• Preserve host permissions in fixtures (they currently arrive world-readable, uid 0).

Recently landed: read-only host→guest fixtures (WithFixture / --fixture, identical on macOS and Linux), VM-to-VM networking over a real network (vmnet SharedMode), and CLI clustering (fleetbox up node -n 3) — so a kubeadm cluster, an etcd quorum, or a Raft group runs on real interconnected nodes, not mocks.

License

MIT.

Documentation

Overview

Package fleetbox provides Linux VMs as Go test fixtures.

On macOS (Apple Silicon) fleetbox boots stock Linux cloud images via Apple Virtualization.framework; on Linux it boots them via cloud-hypervisor. Either way it configures the guest once with cloud-init and provides SSH access for testing, through the same backend-neutral Go API.

The orchestration runs client-side on both platforms and drives a VM helper over a unix socket; the helper holds only the live VMs/network (ADR-0020). On macOS that helper is a separately distributed, ad-hoc-signed fleetbox-helper subprocess the library downloads at first use, so the importable package — and therefore the user's test binary — is pure Go and needs neither cgo nor codesign (ADR-0017). On Linux the single binary self-reexecs into the helper. The public API below is identical on both.

Basic usage:

vm, err := fleetbox.Start(ctx, "myvm")
if err != nil {
	log.Fatal(err)
}
defer vm.Stop(ctx)

out, err := vm.SSH(ctx, "uname -a")

Index

• Variables
• func NestedVirtSupported() bool
• func Prune() error
• type Cluster
  • func NewCluster(options ...Option) (*Cluster, error)
  • func StartCluster(ctx context.Context, names []string, options ...Option) (*Cluster, error)
  • func (c *Cluster) Add(ctx context.Context, name string) (*VM, error)
  • func (c *Cluster) Close() error
  • func (c *Cluster) VMs() []*VM
• type Fixture
• type Option
  • func WithCPUs(n int) Option
  • func WithDiskGB(n int) Option
  • func WithFixture(hostDir, guestPath string) Option
  • func WithImage(img string) Option
  • func WithMemoryGB(n int) Option
• type Options
• type VM
  • func Start(ctx context.Context, name string, options ...Option) (*VM, error)
  • func StartN(ctx context.Context, prefix string, n int, options ...Option) ([]*VM, error)
  • func (v *VM) Destroy(ctx context.Context) error
  • func (v *VM) IP() net.IP
  • func (v *VM) Name() string
  • func (v *VM) SSH(ctx context.Context, cmd string) (string, error)
  • func (v *VM) State() string
  • func (v *VM) Stop(ctx context.Context) error

Variables

var ErrClustersUnsupported = errors.New("fleetbox: clusters require macOS 26+")

ErrClustersUnsupported is returned when a second cluster member is requested on a backend that cannot interconnect VMs — macOS older than 26, where VZ NAT isolates VMs from one another (ADR-0008, ADR-0012). A single VM still works.

Functions

func NestedVirtSupported

func NestedVirtSupported() bool

NestedVirtSupported returns true if nested virtualization is available — what consumers that run KVM inside guests need. On macOS it requires Apple Silicon M3+ and macOS 15+; on Linux it requires /dev/kvm and the KVM nested parameter enabled. On macOS it is a pure-Go host heuristic (so deciding to skip a test never downloads the helper); the helper performs the authoritative VZ check at boot (ADR-0017, R7).

func Prune

func Prune() error

Prune reclaims the inert host resources a fleetbox holder leaves behind if it dies without running its teardown — on Linux, orphaned bridges, taps, and firewall rules, plus restoring ip_forward once nothing of ours remains. It runs automatically on the CLI's down, so cleanup is never the user's job; this exported form is for library callers that want to sweep explicitly. On macOS it is a no-op: vmnet owns its own state and the in-process VMs of a dead helper die with it (ADR-0013, ADR-0017).

Types

type Cluster

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster is a set of VMs sharing one network, so every member reaches the others by IP — a vmnet SharedMode network on macOS, a Linux bridge on Linux (ADR-0008, ADR-0011). The shared network is a runtime object tied to the Cluster's lifetime — never persisted, so a Cluster is a runtime handle, not on-disk state. Members can be added after creation, which is what lets a CLI holder process grow a live cluster without recreating its network.

func NewCluster

func NewCluster(options ...Option) (*Cluster, error)

NewCluster creates a cluster client. The helper is spawned on the first Add (it is launched on the first member's name) and then owns every member on one shared network. Shared client prep (store, SSH key, image) runs once and is reused for every Add.

func StartCluster

func StartCluster(ctx context.Context, names []string, options ...Option) (*Cluster, error)

StartCluster boots the named VMs on one shared network and returns the Cluster. On any member's failure it destroys the members already started and closes the cluster, then returns the error (all-or-nothing, like StartN).

func (*Cluster) Add

func (c *Cluster) Add(ctx context.Context, name string) (*VM, error)

Add boots an additional VM on the cluster's shared network and registers it as a member. The new VM reaches every existing member by IP. Adding a second member on a backend that cannot interconnect VMs (macOS < 26) is rejected with ErrClustersUnsupported before any boot work — the same guard StartCluster applies up front, here also covering a node re-joining a live cluster (ADR-0012). The first member is a lone VM and is always allowed.

func (*Cluster) Close

func (c *Cluster) Close() error

Close releases the cluster's resources. On Linux it tears down the shared bridge and egress rules; on macOS it stops every remaining member and waits for the helper to exit. Call it once every member has been stopped or destroyed. It is idempotent.

func (*Cluster) VMs

func (c *Cluster) VMs() []*VM

VMs returns a snapshot of the cluster's current members in the order they were added.

type Fixture

type Fixture = opts.Fixture

Fixture is a read-only host directory packed into the guest at boot. At first creation HostPath is snapshotted into an ext4 image, attached to the VM as a read-only block device, and mounted by the stock guest at GuestPath via cloud-init. Fixtures are a property the VM is born with: the set is frozen at first creation and persisted, but the content is rebuilt from HostPath on every boot (so the guest sees the directory as it is at that boot, never live within a boot). Files arrive world-readable (0444), directories traversable (0555), owned by root; host permission and executable bits are not preserved. It works identically on macOS and Linux (ADR-0015).

type Option

type Option = opts.Option

Option is a functional option for configuring a VM.

func WithCPUs

func WithCPUs(n int) Option

WithCPUs sets the number of CPUs.

func WithDiskGB

func WithDiskGB(n int) Option

WithDiskGB sets the disk size in gigabytes.

func WithFixture

func WithFixture(hostDir, guestPath string) Option

WithFixture packs the host directory hostDir into the guest at guestPath as a read-only fixture: at boot the directory is snapshotted into an ext4 image, attached as a read-only block device, and mounted by the stock guest at guestPath. Call it more than once to add several fixtures. guestPath must be absolute; hostDir must exist and be a directory, and is resolved to an absolute path. The fixture set is frozen when the VM is first created and ignored when passed to an already-existing VM (exactly as cpu/memory/disk options are), but the content is rebuilt from hostDir on every boot. Files arrive world-readable, owned by root. In a StartN or StartCluster every member receives the same fixtures (ADR-0015).

func WithImage

func WithImage(img string) Option

WithImage sets the image to use (alias or URL).

func WithMemoryGB

func WithMemoryGB(n int) Option

WithMemoryGB sets the memory in gigabytes.

type Options

type Options = opts.Options

Options configures VM creation.

type VM

type VM struct {
	// contains filtered or unexported fields
}

VM represents a running virtual machine.

func Start

func Start(ctx context.Context, name string, options ...Option) (*VM, error)

Start creates and boots a new VM with the given name. The orchestration runs client-side (image resolve, disk/seed/fixture build, store) and drives a VM helper over a unix socket; the helper is the downloaded, signed binary on macOS and a self-reexec of this binary on Linux (ADR-0020). If the VM already exists, the helper boots the existing VM. SSH dials the VM's IP directly.

func StartN

func StartN(ctx context.Context, prefix string, n int, options ...Option) ([]*VM, error)

StartN creates and boots N VMs with the given prefix (prefix-1, prefix-2, ...). All N VMs share ONE network, so they reach each other by IP — the cluster is interconnected (ADR-0008 macOS, ADR-0011 Linux). It is a thin wrapper over StartCluster with generated names.

func (*VM) Destroy

func (v *VM) Destroy(ctx context.Context) error

Destroy stops the VM and removes all its files.

func (*VM) IP

func (v *VM) IP() net.IP

IP returns the VM's IP address.

func (*VM) Name

func (v *VM) Name() string

Name returns the VM name.

func (*VM) SSH

func (v *VM) SSH(ctx context.Context, cmd string) (string, error)

SSH executes a command on the VM via SSH and returns the output.

func (*VM) State

func (v *VM) State() string

State returns the current VM state.

func (*VM) Stop

func (v *VM) Stop(ctx context.Context) error

Stop gracefully shuts down the VM. The disk is preserved.