fleetbox

README

fleetbox

Real Linux VMs as Go test fixtures — on macOS and Linux.

fleetbox boots stock Linux cloud images — on macOS (Apple Silicon) through Apple's Virtualization.framework, on Linux through cloud-hypervisor — hands them to your tests over SSH, and tears them down when the test ends. Think testcontainers — except instead of a container you get a whole machine: real kernel, real systemd, real /dev/kvm. The Go API is the same on both platforms.

func TestAgainstRealLinux(t *testing.T) {
	vm := fleetboxtest.Start(t, fleetbox.Debian12)

	out, err := vm.SSH(context.Background(), "uname -a")
	if err != nil {
		t.Fatal(err)
	}
	t.Log(out) // Linux ... aarch64 GNU/Linux
}

One line gets you a booted Debian box, reachable over SSH (fleetbox generates its own keypair — it never touches your ~/.ssh). The VM is destroyed automatically when the test returns.

Status: v0. It works, but the API will change and there are no compatibility promises yet. See Limitations.

Why

Containers are wonderful right up until you need to test something a container can't give you — a kernel module, a systemd unit, an nftables ruleset, kubeadm, a VPN, anything that wants /dev/kvm. The usual fallback is a VM tool that comes with a yaml file, a background agent forwarding SSH ports, and patched images. That's a lot of moving parts to keep working.

fleetbox takes the opposite line:

• Real VMs, not containers. It EFI-boots unmodified cloud images through their own bootloader. Real kernel, real init, and nested virtualization on M3+ — you can run KVM inside the guest.
• Every VM gets a routable IP. On macOS 26+ each VM joins a vmnet SharedMode network; on Linux it gets a static address on a shared bridge — a real, directly reachable IP either way. No port forwarding, no -p flags, no tunnel daemon — call vm.IP() and connect.
• Nothing of ours runs in the guest. No agent, no helper binary, no host↔guest protocol. A VM is configured exactly once by cloud-init; after that it's a plain distro you reach over SSH.
• Library-first. The Go package is the product; the CLI is a thin wrapper over the same calls. Fixtures clean themselves up through t.Cleanup.

It's opinionated on purpose: no yaml, no templates, no per-distro code paths — just flags and sane defaults.

Requirements

One of:

• macOS, Apple Silicon — clusters (VM↔VM) need macOS 26+ (vmnet SharedMode); macOS below 26 runs a single VM via VZ NAT. Nested virtualization (/dev/kvm in the guest) needs M3 or newer. Intel Macs are not supported. All VM work runs in a small signed fleetbox-helper that fleetbox downloads to ~/.fleetbox/bin/ on first use — your test binary stays pure Go and needs no codesign (see First run).
• Linux, amd64 or arm64 — needs /dev/kvm (be in the kvm group) and CAP_NET_ADMIN (to create the bridge and taps). fleetbox preflight-checks both and fails with a one-line fix if either is missing, rather than a cryptic boot error. The cloud-hypervisor binary and firmware are downloaded and checksum-pinned to ~/.fleetbox/bin/ on first use.

Plus Go 1.24+. The module compiles on darwin/arm64 and linux/{amd64,arm64}; other targets build but return a clear "unsupported platform" error.

Install

go get github.com/pilat/fleetbox

First run

Your binary that boots a VM links no hypervisor and needs no codesign — that is the whole point. The Virtualization.framework work lives in a small, separately signed fleetbox-helper, fetched once and cached, the same way the Linux backend already fetches cloud-hypervisor.

• macOS: the helper is downloaded to ~/.fleetbox/bin/ (checksum-pinned) on first use. Until the first tagged release publishes that artifact, build it locally and point fleetbox at it with FLEETBOX_HELPER:

  make helper                                    # builds + ad-hoc-signs ./bin/fleetbox-helper
  FLEETBOX_HELPER=$PWD/bin/fleetbox-helper go test ./...
  

  FLEETBOX_HELPER is also the offline / air-gapped escape hatch once the download exists.

• Linux: nothing to sign; cloud-hypervisor downloads to ~/.fleetbox/bin/ on first use.

On both platforms the cloud image downloads once to ~/.fleetbox/images/. A first run that pulls a multi-hundred-MB image (and, on macOS, the helper) prints a progress line so it doesn't look like a hung test.

Usage

As a test fixture

import (
	"context"
	"testing"

	"github.com/pilat/fleetbox"
	"github.com/pilat/fleetbox/fleetboxtest"
)

func TestNeedsARealKernel(t *testing.T) {
	vm := fleetboxtest.Start(t, fleetbox.Debian12,
		fleetbox.WithCPUs(2),
		fleetbox.WithMemoryGB(4),
	)

	// A real machine, not a container: real init (systemd as PID 1), real
	// kernel, nested KVM available.
	out, err := vm.SSH(context.Background(),
		"cat /proc/1/comm && uname -r && test -e /dev/kvm && echo ok")
	if err != nil {
		t.Fatalf("%v\n%s", err, out)
	}
	t.Log(out) // systemd / 6.1.0-… / ok
}

fleetboxtest.Start registers t.Cleanup to destroy the VM, derives a collision-safe name from the test name (parallel-test friendly), and skips automatically when the hardware can't run it. SkipIfShort opts a test out under go test -short. StartN boots an interconnected cluster — several VMs on one shared network that reach each other by IP (where clustering is supported; see Limitations).

As a library (no testing.T)

vm, err := fleetbox.Start(ctx, "builder",
	fleetbox.WithImage(fleetbox.Ubuntu2404),
	fleetbox.WithCPUs(4),
	fleetbox.WithMemoryGB(8),
	fleetbox.WithDiskGB(40),
)
if err != nil {
	log.Fatal(err)
}

fmt.Println(vm.IP())              // net.IP — directly reachable, no forwarding

out, err := vm.SSH(ctx, "sudo apt-get install -y nginx")  // user has passwordless sudo
if err != nil {
	log.Fatalf("%v\n%s", err, out)
}

_ = vm.Stop(ctx)                  // graceful shutdown, disk preserved
// vm.Destroy(ctx) deletes it entirely

Start is idempotent: call it again with the same name and it boots the existing VM instead of recreating it. State lives under ~/.fleetbox/clusters/<cluster>/<name>/ and survives reboots — Destroy (or fleetbox rm) is the only thing that deletes a disk. Full API on pkg.go.dev.

Handing a VM a directory

WithFixture packs a host directory into the guest as a read-only fixture — the natural way to hand a VM your test data, config, or build output without a daemon. It works identically on macOS and Linux: at boot the directory is snapshotted into an ext4 image, attached read-only, and mounted by the guest at the path you give:

dir := t.TempDir()
os.WriteFile(filepath.Join(dir, "input.json"), payload, 0o644)

vm := fleetboxtest.Start(t, fleetbox.Debian12, fleetbox.WithFixture(dir, "/work"))
out, _ := vm.SSH(context.Background(), "cat /work/input.json")  // reads the snapshot

From the CLI it's a repeatable --fixture host:guest flag:

./bin/fleetbox up dev --fixture ./src:/work --fixture ./fixtures:/data

The fixture is read-only and world-readable (every file 0444, every dir 0555, owned by root), so any guest user can read it; host permission and exec bits are not preserved. The set of fixtures is frozen when the VM is first created (change it with rm + recreate), but the content is re-snapshotted from the host directory on every boot — so a reboot picks up host-side changes, though never live within a single boot. To get data back out of the guest, use fleetbox cp / scp. The guest path must be absolute, and host paths must not contain colons (the value is split on the last colon).

From the command line

The CLI wraps the same library for manual work:

make build                                   # compiles ./bin/fleetbox (pure Go, nothing to sign)

./bin/fleetbox up web                        # create & boot a VM
./bin/fleetbox up node -n 3                  # interconnected cluster: node-1, node-2, node-3
./bin/fleetbox ssh node-2                     # address a cluster member by name
./bin/fleetbox ssh node-1 -- ping -c1 node-2 # nodes reach each other by IP
./bin/fleetbox ssh web -- systemctl status   # …or run a command
./bin/fleetbox cp ./mybinary web:/usr/local/bin/
./bin/fleetbox ls                            # NAME  IP  STATE  CPUS  MEM  DISK  AGE
./bin/fleetbox ssh-config >> ~/.ssh/config   # then plain `ssh web` works
./bin/fleetbox down node-1                    # stop one member; the rest keep running
./bin/fleetbox rm node                        # destroy the whole cluster (prefix match)

A cluster runs in one holder process sharing one network (a vmnet network on macOS, a Linux bridge on Linux), so its VMs reach each other by IP; down/ssh/rm still address each member by name.

Images

Use a built-in alias or any direct URL to a raw / qcow2 cloud image:

Alias	Image
debian-12 (default)	Debian 12 generic cloud (amd64/arm64, per host)
ubuntu-24.04	Ubuntu 24.04 server cloud (amd64/arm64, per host)

fleetboxtest.Start(t, fleetbox.Debian12)
fleetboxtest.Start(t, "https://example.com/my-cloud-image.qcow2")

Images are downloaded and cached once in ~/.fleetbox/images/, with qcow2 converted to raw on the way in. Adding a distro is adding a catalog entry — there are no per-distro code paths.

How it works

Start runs a short, boring pipeline: ensure the SSH key → download and cache the image → generate a cloud-init seed ISO → boot the VM (macOS: EFI on a vmnet SharedMode NIC; Linux: cloud-hypervisor with a tap on a shared bridge) → get the VM's IP (macOS: from /var/db/dhcpd_leases by hostname; Linux: the statically assigned address) → wait for SSH. On Linux that pipeline runs in-process; on macOS it runs in the downloaded, signed fleetbox-helper subprocess — the only thing that links Virtualization.framework — which the library spawns bound to the test process, so the helper and its VMs are reaped when the test exits (even on kill -9). In CLI mode a detached holder per up group (a single VM, or a whole cluster sharing one network) outlives the command: on macOS that holder is the helper, on Linux the re-exec'd CLI. SSH and cp dial the VM's IP directly either way — the helper protocol never proxies them.

For the full picture, read ARCHITECTURE.md; for why it's built this way, the decision log lives in docs/adr/.

Limitations

Both the library (StartN) and the CLI (up -n N) boot interconnected clusters whose VMs reach each other by IP. Mind the sharp edges:

• Fixtures are read-only and frozen at creation. WithFixture / --fixture copies a host directory into the guest read-only (an ext4 image), on both macOS and Linux. There is no live read-write share — edits inside the guest don't flow back to the host; use cp / scp for the output direction. The set of fixtures is fixed when the VM is created (rm and recreate to change it), though the content re-snapshots on every boot. Files arrive world-readable, owned by root; host permission and exec bits aren't preserved.
• A CLI cluster shares one holder process. All members of a cluster live in one process to share one network, so a holder crash takes the whole cluster down (a single VM is unaffected); on Linux a SIGKILL'd holder also leaves its bridge/taps behind (swept on the next up/down). On macOS the holder is the downloaded fleetbox-helper. Members started by separate up commands have separate networks and can't be merged into one cluster afterwards — bring a cluster up together.
• First run downloads, then caches. The cloud image (both platforms) and, on macOS, the signed fleetbox-helper are fetched once into ~/.fleetbox and reused. Until the first tagged release publishes the helper, build it locally (make helper) and set FLEETBOX_HELPER — which is also the offline override. In CI, cache ~/.fleetbox/{bin,images} so cold runs don't re-download.
• Platform matrix. macOS Apple Silicon 26+ (clusters), macOS Apple Silicon <26 (single VM only), Linux amd64/arm64 (clusters); Intel macOS unsupported. On Linux, a stopped VM brought back up needs its /24 to still be free — on a contended host the auto-picked subnet can shift and the rebooted VM won't be reachable; bring clusters up fresh. arm64 Linux boot via rust-hypervisor-firmware is not yet validated on hardware.
• v0 API. Expect breaking changes until it stabilizes.

CI

GitHub-hosted macOS runners can't nest-virtualize, so the macOS CI does lint, build, and unit tests; VM-boot tests run locally via make test-vm. But GitHub-hosted x86-64 Linux runners do expose /dev/kvm, so the Linux backend's VM-boot tests can run in CI after a one-time udev tweak that lets the runner user open it (and a cache so the image + VMM aren't re-downloaded every run):

- run: |
    echo 'KERNEL=="kvm", GROUP="kvm", MODE="0666"' | sudo tee /etc/udev/rules.d/99-kvm.rules
    sudo udevadm control --reload-rules && sudo udevadm trigger
- uses: actions/cache@v4
  with:
    path: |
      ~/.fleetbox/bin
      ~/.fleetbox/images
    key: fleetbox-${{ runner.os }}

arm64 hosted Linux runners do not have KVM ("not supported for this sku"); use an x86-64 runner for VM-boot CI. This is the "develop on a Mac, run in cheap x86-64 hosted Linux CI" story.

Roadmap

Roughly in priority order:

• Programmatic file copy — a library-side copy in/out for cases a fixture doesn't fit (the CLI already has cp over scp).
• Preserve host permissions in fixtures (they currently arrive world-readable, uid 0).

Done recently: read-only host→guest fixtures (WithFixture / --fixture, an ext4 payload, identical on macOS and Linux); VM-to-VM networking over a real network (vmnet SharedMode); and CLI clustering (fleetbox up node -n 3) — boot an actual cluster (kubeadm, etcd, a Raft group) on real interconnected nodes, not mocks or a single-host simulation.

License

MIT.

Documentation

Overview

Package fleetbox provides Linux VMs as Go test fixtures.

On macOS (Apple Silicon) fleetbox boots stock Linux cloud images via Apple Virtualization.framework; on Linux it boots them via cloud-hypervisor. Either way it configures the guest once with cloud-init and provides SSH access for testing, through the same backend-neutral Go API.

On macOS all Virtualization.framework work runs in a separately distributed, ad-hoc-signed fleetbox-helper subprocess that the library downloads at first use and drives over a unix socket, so the importable package — and therefore the user's test binary — is pure Go and needs neither cgo nor codesign (ADR-0017). On Linux the orchestration runs in-process. The public API below is identical on both.

Basic usage:

vm, err := fleetbox.Start(ctx, "myvm")
if err != nil {
	log.Fatal(err)
}
defer vm.Stop(ctx)

out, err := vm.SSH(ctx, "uname -a")

Index

• Constants
• Variables
• func NestedVirtSupported() bool
• func Prune() error
• type Cluster
  • func NewCluster(options ...Option) (*Cluster, error)
  • func StartCluster(ctx context.Context, names []string, options ...Option) (*Cluster, error)
  • func (c *Cluster) Add(ctx context.Context, name string) (*VM, error)
  • func (c *Cluster) Close() error
  • func (c *Cluster) VMs() []*VM
• type Fixture
• type Option
  • func WithCPUs(n int) Option
  • func WithDiskGB(n int) Option
  • func WithFixture(hostDir, guestPath string) Option
  • func WithImage(img string) Option
  • func WithMemoryGB(n int) Option
• type Options
• type VM
  • func Start(ctx context.Context, name string, options ...Option) (*VM, error)
  • func StartN(ctx context.Context, prefix string, n int, options ...Option) ([]*VM, error)
  • func (v *VM) Destroy(ctx context.Context) error
  • func (v *VM) IP() net.IP
  • func (v *VM) Name() string
  • func (v *VM) SSH(ctx context.Context, cmd string) (string, error)
  • func (v *VM) State() string
  • func (v *VM) Stop(ctx context.Context) error

Constants

const (
	Debian12   = "debian-12"
	Ubuntu2404 = "ubuntu-24.04"
)

Image aliases for common distributions.

Variables

var ErrClustersUnsupported = errors.New("fleetbox: clusters require macOS 26+")

ErrClustersUnsupported is returned when a second cluster member is requested on a backend that cannot interconnect VMs — macOS older than 26, where VZ NAT isolates VMs from one another (ADR-0008, ADR-0012). A single VM still works.

Functions

func NestedVirtSupported

func NestedVirtSupported() bool

NestedVirtSupported returns true if nested virtualization is available — what consumers that run KVM inside guests need. On macOS it requires Apple Silicon M3+ and macOS 15+; on Linux it requires /dev/kvm and the KVM nested parameter enabled. On macOS it is a pure-Go host heuristic (so deciding to skip a test never downloads the helper); the helper performs the authoritative VZ check at boot (ADR-0017, R7).

func Prune

func Prune() error

Prune reclaims the inert host resources a fleetbox holder leaves behind if it dies without running its teardown — on Linux, orphaned bridges, taps, and firewall rules, plus restoring ip_forward once nothing of ours remains. It runs automatically on the CLI's down, so cleanup is never the user's job; this exported form is for library callers that want to sweep explicitly. On macOS it is a no-op: vmnet owns its own state and the in-process VMs of a dead helper die with it (ADR-0013, ADR-0017).

Types

type Cluster

type Cluster struct {
	// contains filtered or unexported fields
}

Cluster is a set of VMs sharing one network, so every member reaches the others by IP — a vmnet SharedMode network on macOS, a Linux bridge on Linux (ADR-0008, ADR-0011). The shared network is a runtime object tied to the Cluster's lifetime — never persisted, so a Cluster is a runtime handle, not on-disk state. Members can be added after creation, which is what lets a CLI holder process grow a live cluster without recreating its network.

func NewCluster

func NewCluster(options ...Option) (*Cluster, error)

NewCluster creates a cluster's shared network but boots no VMs. Use Cluster.Add to bring members up on it. Shared setup (store, SSH key, image, backend) runs once here and is reused for every Add.

func StartCluster

func StartCluster(ctx context.Context, names []string, options ...Option) (*Cluster, error)

StartCluster boots the named VMs on one shared network and returns the Cluster. On any member's failure it destroys the members already started and closes the cluster, then returns the error (all-or-nothing, like StartN).

func (*Cluster) Add

func (c *Cluster) Add(ctx context.Context, name string) (*VM, error)

Add boots an additional VM on the cluster's shared network and registers it as a member. The new VM reaches every existing member by IP. Adding a second member on a backend that cannot interconnect VMs (macOS < 26) is rejected with ErrClustersUnsupported before any boot work — the same guard StartCluster applies up front, here also covering a node re-joining a live cluster (ADR-0012). The first member is a lone VM and is always allowed.

func (*Cluster) Close

func (c *Cluster) Close() error

Close releases the cluster's resources. On Linux it tears down the shared bridge and egress rules; on macOS it stops every remaining member and waits for the helper to exit. Call it once every member has been stopped or destroyed. It is idempotent.

func (*Cluster) VMs

func (c *Cluster) VMs() []*VM

VMs returns a snapshot of the cluster's current members in the order they were added.

type Fixture

type Fixture = opts.Fixture

Fixture is a read-only host directory packed into the guest at boot. At first creation HostPath is snapshotted into an ext4 image, attached to the VM as a read-only block device, and mounted by the stock guest at GuestPath via cloud-init. Fixtures are a property the VM is born with: the set is frozen at first creation and persisted, but the content is rebuilt from HostPath on every boot (so the guest sees the directory as it is at that boot, never live within a boot). Files arrive world-readable (0444), directories traversable (0555), owned by root; host permission and executable bits are not preserved. It works identically on macOS and Linux (ADR-0015).

type Option

type Option = opts.Option

Option is a functional option for configuring a VM.

func WithCPUs

func WithCPUs(n int) Option

WithCPUs sets the number of CPUs.

func WithDiskGB

func WithDiskGB(n int) Option

WithDiskGB sets the disk size in gigabytes.

func WithFixture

func WithFixture(hostDir, guestPath string) Option

WithFixture packs the host directory hostDir into the guest at guestPath as a read-only fixture: at boot the directory is snapshotted into an ext4 image, attached as a read-only block device, and mounted by the stock guest at guestPath. Call it more than once to add several fixtures. guestPath must be absolute; hostDir must exist and be a directory, and is resolved to an absolute path. The fixture set is frozen when the VM is first created and ignored when passed to an already-existing VM (exactly as cpu/memory/disk options are), but the content is rebuilt from hostDir on every boot. Files arrive world-readable, owned by root. In a StartN or StartCluster every member receives the same fixtures (ADR-0015).

func WithImage

func WithImage(img string) Option

WithImage sets the image to use (alias or URL).

func WithMemoryGB

func WithMemoryGB(n int) Option

WithMemoryGB sets the memory in gigabytes.

type Options

type Options = opts.Options

Options configures VM creation.

type VM

type VM struct {
	// contains filtered or unexported fields
}

VM represents a running virtual machine.

func Start

func Start(ctx context.Context, name string, options ...Option) (*VM, error)

Start creates and boots a new VM with the given name. If the VM already exists, it boots the existing VM. On Linux the orchestration runs in-process (cloud-hypervisor, pure Go).

func StartN

func StartN(ctx context.Context, prefix string, n int, options ...Option) ([]*VM, error)

StartN creates and boots N VMs with the given prefix (prefix-1, prefix-2, ...). All N VMs share ONE network, so they reach each other by IP — the cluster is interconnected (ADR-0008 macOS, ADR-0011 Linux). It is a thin wrapper over StartCluster with generated names.

func (*VM) Destroy

func (v *VM) Destroy(ctx context.Context) error

Destroy stops the VM and removes all its files.

func (*VM) IP

func (v *VM) IP() net.IP

IP returns the VM's IP address.

func (*VM) Name

func (v *VM) Name() string

Name returns the VM name.

func (*VM) SSH

func (v *VM) SSH(ctx context.Context, cmd string) (string, error)

SSH executes a command on the VM via SSH and returns the output.

func (*VM) State

func (v *VM) State() string

State returns the current VM state.

func (*VM) Stop

func (v *VM) Stop(ctx context.Context) error

Stop gracefully shuts down the VM. The disk is preserved.