baremetalvmm

module
v0.8.1 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 9, 2026 License: MIT

README

VMM - Bare Metal MicroVM Manager

WARNING This is a vibe-coded piece of software allow for creation of microVMs conveniently. It's still heavily in development, I do not recommend anyone apart from me use it :) !

The goal of the project is to allow for small development VMs to be spun up based on firecracker, so they're lightweight. It can build VM images from a Docker image, allowing for custom VMs.

The goal of the project is to be useful in cases where you want something like Docker, but want some more isolation that Docker provides, or you want to do lower level tasks in the VM that don't suit Docker well. N.B we're not there yet!

Pretty much all of the coding has been done with Claude code.

Requirements

  • Ubuntu 24.04 (or compatible Linux distribution). All testing has been done on Ubuntu 24.04, so it's likely only to work with that distro.
  • KVM support (/dev/kvm must be accessible)
  • Root access (for networking setup)
  • Go 1.21+ (only if building from source)

Quick Start

Installation
# Clone the repository
git clone https://github.com/raesene/baremetalvmm.git
cd baremetalvmm

# Install (requires root)
sudo ./scripts/install.sh

The install script will:

  • Download the pre-built vmm and vmm-web binaries from GitHub releases (amd64/arm64)
  • Fall back to building from source if download fails
  • Install the binaries to /usr/local/bin
  • Download Firecracker v1.11.0
  • Download a pre-built Linux 6.1 kernel from GitHub releases
  • Download a pre-built Kubernetes-compatible 6.6 kernel from GitHub releases (for vmm cluster)
  • Download a pre-built Ubuntu 24.04 rootfs from GitHub releases (falls back to Firecracker S3 URL)
  • Create data directories in /var/lib/vmm
  • Install build-kernel.sh and build-rootfs.sh to /usr/local/share/vmm
Uninstallation

To completely remove VMM and all associated resources:

sudo ./scripts/uninstall.sh

The uninstall script will:

  • Stop all running VMs (Firecracker processes)
  • Remove network resources (bridge, TAP devices, iptables rules)
  • Remove all VM data (/var/lib/vmm)
  • Remove user configuration (~/.config/vmm)
  • Remove the systemd service (if installed)
  • Remove binaries (vmm, vmm-web, firecracker, build-kernel.sh, build-rootfs.sh)

Use --yes or -y to skip the confirmation prompt:

sudo ./scripts/uninstall.sh --yes

Note: The script is idempotent and safe to run multiple times.

One time Setup

First up (one time only) run the init command

vmm config init

Next up we need to pull the default kernel and root image. The kernel is a pre-built Linux 6.1 kernel from our GitHub releases, and the rootfs is a pre-built Ubuntu 24.04 image also from our GitHub releases. We can change the rootfs with more commands and also use custom kernels (see Custom Kernels section). Again this is one-time, they should be present for future runs

sudo vmm image pull
Shell Completion

VMM supports shell completion for bash, zsh, and fish. Completions include command names, VM names, cluster names, kernel names, and image names.

# Bash (add to ~/.bashrc for persistence)
source <(vmm completion bash)

# Zsh (add to ~/.zshrc for persistence)
source <(vmm completion zsh)

# Fish
vmm completion fish | source
Basic usage

First up we create a VM. Key elements we can configure here are number of CPUs, amount of memory, amount of disk space and importantly an SSH key to use to connect to the VM once it's started. there also also other options for things like custom images (see later in README) and custom DNS servers.

sudo vmm create myvm --cpus 2 --memory 1024 --ssh-key ~/.ssh/id_ed25519.pub

Once the VM is created, we can start it up

sudo vmm start myvm

Then once it's started we should be able to SSH in to it. That can be done by name using the vmm command as shown below, or you can just use standard ssh with a username of root and the IP address of the VM. By default it's only reachable from the local machine, but you can use the port-forward command to expose the VM to the wider world (using an iptables command under the covers)

vmm ssh myvm

To stop the VM but leave it in place

sudo vmm stop myvm

and then to clean it up

sudo vmm delete myvm

Available Kernels and Root Filesystems

VMM ships with two kernels and two root filesystems, each designed for a specific use case. The vmm image list command shows all available options with descriptions:

$ vmm image list
Kernels:
  - k8s-kernel             32.4 MB  Kubernetes cluster kernel (Linux 6.6 LTS, Cilium/BPF)
  - security-kernel        85.2 MB  Security testing kernel (Linux 6.12 LTS, broad module coverage)
  - vmlinux.bin            72.7 MB  General-purpose VM kernel (Linux 6.1 LTS) (default)

Root filesystems:
  - k8s-1.36.0           2048.0 MB  Kubernetes image (kubeadm/containerd pre-installed)
  - rootfs                512.0 MB  Ubuntu 24.04 base image for general-purpose VMs (default)
Which to use
Use case Kernel Rootfs Command
Standalone VMs vmlinux.bin (default) rootfs (default) sudo vmm create myvm
Kubernetes clusters k8s-kernel k8s-<version> (auto-detected) sudo vmm cluster create mycluster
Security/vuln testing security-kernel rootfs (default) or k8s-<version> sudo vmm create testvm --kernel security-kernel
Naming convention

Kernels and rootfs images follow a prefix-based naming convention so that vmm image list and the web UI can automatically show descriptions:

Prefix Kernel meaning Rootfs meaning
(default) General-purpose (Linux 6.1 LTS, all built-in) Ubuntu 24.04 base (systemd, SSH, networking)
k8s- Kubernetes/Cilium (Linux 6.6 LTS, BPF JIT, VXLAN, modules) Kubernetes (kubeadm/containerd pre-installed)
security- Security testing (Linux 6.12 LTS, broad module coverage) (not yet used)
debug- Debug kernel (extra logging and debug options) (not yet used)
minimal- Minimal kernel (reduced feature set) Minimal image (reduced package set)

When adding new kernel or rootfs variants, use an appropriate prefix so that the description auto-populates. Custom user-imported images without a recognized prefix show as "Custom kernel" or "Custom image".

Custom Rootfs and Kernel

By default, vmm image pull downloads a pre-built Linux 6.1 kernel and an Ubuntu 24.04 rootfs from our GitHub releases (both built automatically via CI). The default rootfs includes systemd, OpenSSH server, and basic networking tools. If you want to run more complex use-cases it makes sense to get a custom rootfs.

Custom rootfs

The way this works is that vmm can get a docker image (needs docker installed) and turn it into a vmm base image, by injecting the necessary files for openssh server and the init system. So far this is all ubuntu based, so you want to stick with that for now.

The vmm image import command will handle that it, you give it the image to pull in docker format and a name to call it

sudo vmm image import ubuntu:24.04 --name ubuntu-24.04
Custom kernel

If you want a different kernel version, you can build one from source. Be aware it's going to download and compile a Linux kernel, so it'll take a while if you're running on a not very powerful machine and it needs disk space.

This command should give you a relatively modern 6.1 based kernel.

sudo vmm kernel build --version 6.1 --name kernel-6.1

Security Testing with Vulnerable Kernels

VMM includes a security testing kernel designed for vulnerability research and PoC exploit testing. This kernel is built from the 6.12 LTS series with broad subsystem coverage, so most kernel exploits that work on Ubuntu will work in VMM VMs.

The security kernel

The security kernel (--config-profile security in the build script) enables many kernel subsystems beyond what the default and k8s kernels provide:

Subsystem Config options Used by
IPsec/xfrm INET_ESP, INET6_ESP, XFRM dirtyfrag
AF_RXRPC AF_RXRPC dirtyfrag
AF_ALG crypto CRYPTO_USER_API_AEAD, CRYPTO_AUTHENC CVE-2026-31431 (copy-fail)
io_uring IO_URING Various io_uring CVEs
SCTP/DCCP/TIPC IP_SCTP, IP_DCCP, TIPC Network protocol CVEs
userfaultfd USERFAULTFD Race condition exploits
Tunneling GRE, IPIP, IPV6_SIT, L2TP Network stack exploits
FUSE/Btrfs/XFS FUSE_FS, BTRFS_FS, XFS_FS Filesystem CVEs
Traffic control NET_SCHED, NET_SCH_*, NET_CLS_* TC/qdisc exploits
LSMs SECURITY_APPARMOR, SECURITY_SELINUX Security testing
Tracing FTRACE, KPROBES, UPROBE_EVENTS Exploit development

The security kernel is available as a pre-built download from GitHub releases (tagged security-kernel-*), or you can build it locally.

Getting the security kernel

Option 1: Download from GitHub releases

# Download the latest security kernel release
wget https://github.com/raesene/baremetalvmm/releases/download/security-kernel-<version>/security-vmlinux.bin
sudo vmm kernel import security-vmlinux.bin --name security-kernel

Option 2: Build locally

sudo vmm kernel build --version 6.8 --name security-kernel
# Note: the CLI build command uses the default profile. To use the security profile,
# run the script directly:
sudo bash scripts/build-kernel.sh --version 6.12 --name security-kernel --config-profile security
Using the security kernel
# Standalone VM
sudo vmm create vuln-test --cpus 2 --memory 2048 --kernel security-kernel --ssh-key ~/.ssh/id_ed25519.pub
sudo vmm start vuln-test

# Kubernetes cluster (for container escape PoCs)
sudo vmm cluster create vuln-cluster --workers 1 --cpus 2 --memory 4096 \
    --kernel security-kernel --ssh-key ~/.ssh/id_ed25519.pub
Verifying kernel capabilities

Check that the required subsystems are available inside the VM:

# Check kernel version
sudo vmm ssh vuln-test -- "uname -r"

# Check specific config options
sudo vmm ssh vuln-test -- "zcat /proc/config.gz | grep -E 'INET_ESP|AF_RXRPC|IO_URING|CRYPTO_USER_API_AEAD'"

# Test AF_ALG (for copy-fail / CVE-2026-31431)
sudo vmm ssh vuln-test -- "python3 -c \"import socket; s = socket.socket(socket.AF_ALG, socket.SOCK_SEQPACKET, 0); s.bind(('aead', 'gcm(aes)')); print('AF_ALG AEAD: available'); s.close()\""
Default vs security kernel

The default kernel (6.1 LTS) and k8s kernel (6.6 LTS) include only the subsystems needed for running VMs and Kubernetes. They have a smaller attack surface and are what you'd use for normal development work.

The security kernel (6.12 LTS) deliberately enables a broad set of subsystems to match what's available on a stock Ubuntu 24.04 installation, making it suitable for reproducing PoC exploits that target those subsystems.

Cleanup
# Standalone
sudo vmm stop vuln-test && sudo vmm delete vuln-test

# Cluster
sudo vmm cluster delete vuln-cluster -f

Commands

VM Lifecycle
Command Description
vmm create <name> Create a new VM configuration (VM is not running yet)
vmm start <name> Start a VM - assigns IP address, sets up networking, boots VM (requires root)
vmm stop <name> Stop a running VM (requires root)
vmm delete <name> Delete a VM and its resources
vmm list List all VMs

Note: VMs must be explicitly started after creation. IP addresses are assigned at start time, not at creation time.

Create Options
vmm create <name> [flags]

Flags:
  --cpus int         Number of vCPUs (default 1)
  --memory int       Memory in MB (default 512)
  --disk int         Disk size in MB (default 1024)
  --ssh-key string   Path to SSH public key file for root access
  --dns string       Custom DNS servers (can be specified multiple times)
  --image string     Name of rootfs image to use (from 'vmm image import')
  --kernel string    Name of kernel to use (from 'vmm kernel import' or 'vmm kernel build')
  --mount string     Mount host directory in VM (format: /host/path:tag[:ro|rw], can be repeated)

Example with all options:

sudo vmm create myvm --cpus 2 --memory 2048 --disk 10000 \
  --ssh-key ~/.ssh/id_ed25519.pub \
  --dns 9.9.9.9 --dns 1.1.1.1 \
  --image ubuntu-base \
  --kernel my-kernel \
  --mount /home/user/code:code:ro
Access
Command Description
vmm ssh <name> SSH into a VM as root
vmm ssh <name> -u <user> SSH as specific user

Note: SSH access requires an SSH public key to be configured when creating the VM using the --ssh-key flag. The key is injected into the VM's rootfs at startup.

Tip: You can use sudo vmm ssh <name> if you prefer consistency with other commands. When run with sudo, VMM automatically detects the original user and uses their SSH keys from their home directory.

Networking
Command Description
vmm port-forward add <name> <host>:<guest> Forward port from host to VM
vmm port-forward list <name> List port forwards for a VM
vmm port-forward remove <name> <host>:<guest> Remove a port forward

Example:

# Forward host port 8080 to VM port 80 (needs sudo for iptables)
sudo vmm port-forward add myvm 8080:80

# List port forwards
vmm port-forward list myvm

# Remove a port forward
sudo vmm port-forward remove myvm 8080:80
Mounts
Command Description
vmm mount list <name> List mounts configured for a VM
vmm mount sync <name> <tag> Sync mount image from host directory (VM must be stopped)

Example:

# List mounts for a VM
vmm mount list myvm

# Sync mount contents after making changes on host
sudo vmm mount sync myvm code
Images
Command Description
vmm image list List available images with descriptions
vmm image pull Download default images
vmm image import <docker-image> --name <name> Import a Docker image as rootfs
vmm image snapshot <vm> --name <name> Snapshot a stopped VM's rootfs as a reusable base image
vmm image delete <name> Delete an imported image
Kernels
Command Description
vmm kernel list List available kernels
vmm kernel import <path> --name <name> Import a custom kernel binary
vmm kernel build --version <ver> --name <name> Build a kernel from source
vmm kernel delete <name> Delete a custom kernel
Configuration
Command Description
vmm config show Show current configuration
vmm config init Initialize directories and config

Configurable VM Defaults

You can set default values for vmm create parameters in your config file (~/.config/vmm/config.json). This is useful if you typically use the same settings for most VMs.

Available Default Settings
Field Type Default Description
cpus int 1 Number of vCPUs
memory_mb int 512 Memory in MB
disk_size_mb int 1024 Disk size in MB
image string (default rootfs) Rootfs image name
kernel string (default kernel) Kernel name
ssh_key_path string (none) Path to SSH public key
dns_servers []string [8.8.8.8, 8.8.4.4, 1.1.1.1] DNS servers
Example Configuration

Edit ~/.config/vmm/config.json to add a vm_defaults section:

{
  "data_dir": "/var/lib/vmm",
  "bridge_name": "vmm-br0",
  "subnet": "172.16.0.0/16",
  "gateway": "172.16.0.1",
  "host_interface": "eth0",
  "vm_defaults": {
    "cpus": 2,
    "memory_mb": 1024,
    "disk_size_mb": 4096,
    "ssh_key_path": "~/.ssh/id_ed25519.pub",
    "kernel": "kernel-6.1",
    "dns_servers": ["9.9.9.9", "1.1.1.1"]
  }
}
How Defaults Work

When you run vmm create, values are resolved in this order:

  1. CLI flag - If you specify a flag (e.g., --cpus 4), it takes priority
  2. Config default - If no flag is given, uses the value from vm_defaults
  3. Built-in default - If neither is set, uses the built-in default
Usage Examples
# With the example config above, this creates a VM with:
# - 2 CPUs, 1024 MB memory, 4096 MB disk (from config)
# - SSH key from ~/.ssh/id_ed25519.pub (from config)
# - kernel-6.1 kernel (from config)
sudo vmm create myvm

# Override specific defaults with CLI flags:
# - 4 CPUs (from flag), 1024 MB memory (from config)
sudo vmm create myvm --cpus 4

# Override multiple defaults:
sudo vmm create myvm --cpus 4 --memory 2048 --kernel kernel-5.10
Viewing Current Defaults

Use vmm config show to see current defaults and their source:

vmm config show
# Output shows each setting and whether it comes from config or built-in default

Note: The vm_defaults section is optional. Existing configs without it will continue to work unchanged, using the built-in defaults.

Kubernetes Clusters

VMM can create Kubernetes clusters from multiple Firecracker VMs, similar to kind but with VM-level isolation for each node. Clusters are bootstrapped with kubeadm and use Cilium as the CNI plugin.

Prerequisites

A Kubernetes-compatible kernel (k8s-kernel) is downloaded automatically during installation. This is a 6.6 LTS kernel with BPF JIT, VXLAN, and cgroups v2 bandwidth control enabled for Cilium CNI support. If you don't have it, you can build one manually:

sudo vmm kernel build --version 6.6 --name k8s-kernel

You also need an SSH key for VM access:

ssh-keygen -t ed25519  # if you don't already have one

If your SSH key has a passphrase, make sure it's loaded in ssh-agent and use sudo -E to preserve the agent socket:

eval "$(ssh-agent -s)"
ssh-add ~/.ssh/id_ed25519
sudo -E vmm cluster create ...
Creating a Cluster
# Single-node cluster (control plane only, like 'kind create cluster')
sudo vmm cluster create mycluster --ssh-key ~/.ssh/id_ed25519.pub --kernel k8s-kernel

# Multi-node cluster with 2 workers (3 VMs total)
sudo vmm cluster create mycluster --workers 2 --ssh-key ~/.ssh/id_ed25519.pub --kernel k8s-kernel

# With custom resources
sudo vmm cluster create mycluster --workers 2 \
  --cpus 4 --memory 8192 --disk 20480 \
  --ssh-key ~/.ssh/id_ed25519.pub \
  --kernel k8s-kernel \
  --k8s-version 1.30.0

The create command:

  1. Creates Firecracker VMs ({name}-control-plane, {name}-worker-1, etc.)
  2. Installs containerd, kubeadm, kubelet, and kubectl via SSH
  3. Runs kubeadm init on the control plane
  4. Installs Cilium CNI (with kube-proxy replacement)
  5. Joins worker nodes to the cluster
  6. Merges kubeconfig into ~/.kube/config as context vmm-{name}
Using a Cluster

Once created, the cluster is immediately usable via kubectl:

# Use the cluster context
kubectl --context vmm-mycluster get nodes
kubectl --context vmm-mycluster get pods -n kube-system

# Or set it as the default context
kubectl config use-context vmm-mycluster
kubectl get nodes

You can also SSH into individual nodes for debugging:

vmm ssh mycluster-control-plane
vmm ssh mycluster-worker-1
Cluster Commands
Command Description
vmm cluster create <name> Create a Kubernetes cluster
vmm cluster delete <name> Delete a cluster and all its VMs
vmm cluster list List all clusters
vmm cluster kubeconfig <name> Re-extract and merge kubeconfig
Create Options
vmm cluster create <name> [flags]

Flags:
  --workers int        Number of worker nodes (default 0, control-plane only)
  --cpus int           vCPUs per node (default 2)
  --memory int         Memory per node in MB (default 4096)
  --disk int           Disk per node in MB (default 10240)
  --k8s-version string Kubernetes version (default "1.36.0")
  --ssh-key string     Path to SSH public key (required)
  --kernel string      Kernel name (k8s-kernel recommended)
  --image string       Rootfs image name
Deleting a Cluster
# Delete with confirmation
sudo vmm cluster delete mycluster

# Force delete without confirmation
sudo vmm cluster delete mycluster -f

This stops and deletes all VMs in the cluster and removes the kubeconfig context.

Cluster Defaults
Setting Default Notes
Workers 0 Control plane only (single-node cluster)
CPUs 2 Minimum 2 required for kubeadm
Memory 4096 MB Minimum 2048 MB required
Disk 10240 MB (10 GB) Needs space for container images
Kubernetes 1.36.0 Any version available from pkgs.k8s.io
CNI Cilium With kube-proxy replacement enabled
Pod CIDR 10.244.0.0/16 Doesn't conflict with VM bridge network
Service CIDR 10.96.0.0/12 Standard Kubernetes default
What Gets Installed in Each VM
  • containerd (from Ubuntu repos) with SystemdCgroup enabled
  • kubeadm, kubelet, kubectl (from pkgs.k8s.io)
  • Cilium CLI (on control plane only)
  • Kernel modules and sysctl settings for networking and cgroups
  • BPF filesystem mount for Cilium
  • Shared mount propagation for Kubernetes volumes

Web UI

VMM includes an optional web-based dashboard (vmm-web) for managing and monitoring VMs from a browser. It's a separate binary that reuses the same internal libraries as the CLI, so all operations are consistent between both interfaces.

Starting the Web UI

The web UI requires a password set via the VMM_WEB_PASSWORD environment variable:

# Listen on localhost only (default)
VMM_WEB_PASSWORD=mysecretpassword sudo -E vmm-web

# Listen on all interfaces for remote access
VMM_WEB_PASSWORD=mysecretpassword sudo -E vmm-web --listen 0.0.0.0:8080

Then open http://<host>:8080 in a browser and log in with username admin and the password you set.

Features
  • Dashboard - Overview of all VMs and clusters with resource usage stats
  • VM Management - Create, start, stop, and delete VMs from the browser
  • Web Terminal - Browser-based SSH terminal for running VMs (xterm.js + WebSocket)
  • Cluster Management - Create and delete Kubernetes clusters
  • Live Status - VM status updates via Server-Sent Events (no page refresh needed)
  • JSON API - REST API at /api/v1/ for scripting and automation
  • Authentication - Session-based login with rate-limited password attempts
Web Terminal

Running VMs have a Terminal button on their detail page that opens a full-screen browser terminal. The terminal connects via WebSocket to an SSH session on the VM, giving you interactive shell access without needing a local SSH client.

  • Uses the host's SSH private key (auto-detected from ~/.ssh/, same as vmm ssh)
  • Supports terminal resize, scrollback, and clickable links
  • Requires the VM to be in "running" state with an SSH key configured
JSON API

The web UI also exposes a JSON API for scripting. Authenticate by logging in via the browser to get a session token, then use it as a Bearer token:

# List VMs
curl -H "Authorization: Bearer <session-token>" http://localhost:8080/api/v1/vms

# Create a VM
curl -X POST -H "Authorization: Bearer <session-token>" \
  -H "Content-Type: application/json" \
  -d '{"name":"myvm","cpus":2,"memory_mb":1024}' \
  http://localhost:8080/api/v1/vms

# Start a VM
curl -X POST -H "Authorization: Bearer <session-token>" \
  http://localhost:8080/api/v1/vms/myvm/start

# Health check (no auth required)
curl http://localhost:8080/api/v1/health
API Endpoints
Method Path Description
GET /api/v1/health Health check (no auth)
GET /api/v1/vms List all VMs
POST /api/v1/vms Create a VM
GET /api/v1/vms/{name} Get VM details
POST /api/v1/vms/{name}/start Start a VM
POST /api/v1/vms/{name}/stop Stop a VM
DELETE /api/v1/vms/{name} Delete a VM
GET /api/v1/clusters List clusters
POST /api/v1/clusters Create a cluster
DELETE /api/v1/clusters/{name} Delete a cluster
Security
  • Default bind address is 127.0.0.1:8080 (localhost only). You must explicitly pass --listen 0.0.0.0:8080 to allow remote access.
  • Login rate limiting - 5 attempts per minute per IP address.
  • Session cookies are HttpOnly and SameSite=Strict.
  • CSRF protection on all state-changing requests.
  • Security headers - CSP, X-Frame-Options DENY, X-Content-Type-Options nosniff.
  • The web server runs as root (required for Firecracker operations). For production use, consider putting it behind a reverse proxy with TLS (e.g., nginx, Caddy).

Architecture

┌───────────────────────────┐  ┌───────────────────────────┐
│         vmm CLI           │  │     vmm-web (HTTP)        │
├───────────────────────────┤  ├───────────────────────────┤
│  create | start | stop    │  │  Dashboard | VM mgmt      │
│  delete | list | ssh ...  │  │  Cluster mgmt | REST API  │
└─────────────┬─────────────┘  └─────────────┬─────────────┘
              │                               │
              └───────────┬───────────────────┘
                          ▼
┌─────────────────────────────────────────────────────────┐
│                  Internal Components                     │
├──────────────┬──────────────┬──────────────┬────────────┤
│   Config     │   Network    │    Image     │ Firecracker│
│   Store      │   Manager    │   Manager    │   Client   │
└──────────────┴──────────────┴──────────────┴────────────┘
                          │
                          ▼
┌─────────────────────────────────────────────────────────┐
│                  Firecracker VMM                         │
│              (One process per microVM)                   │
└─────────────────────────────────────────────────────────┘
Networking

VMs are connected via a bridge network with automatic IP configuration:

Host Network (eth0)
       │
       ▼
┌──────────────┐
│   iptables   │  ← NAT/MASQUERADE
│   DNAT/SNAT  │  ← Port forwarding
└──────────────┘
       │
       ▼
┌──────────────┐
│   vmm-br0    │  ← Bridge (172.16.0.1/16)
└──────────────┘
    │  │  │
    ▼  ▼  ▼
  tap0 tap1 tap2  ← One TAP per VM
    │  │  │
    ▼  ▼  ▼
  VM1 VM2 VM3     ← 172.16.0.2, 172.16.0.3, ...

IP addresses are allocated from 172.16.0.2 upward when a VM is started (not when created). Each start scans existing VMs and picks the next unused address, so multiple clusters and standalone VMs can coexist without IP collisions. The IP is configured via kernel command line parameters, so VMs get network connectivity immediately on boot.

Directory Structure

/var/lib/vmm/
├── config/           # Global configuration
├── vms/              # VM configurations and rootfs
├── clusters/         # Cluster configurations (JSON)
├── images/
│   ├── kernels/      # Linux kernel images
│   └── rootfs/       # Root filesystem images
├── mounts/           # Mount images (ext4 images from host directories)
├── sockets/          # Firecracker API sockets
├── logs/             # VM logs
└── state/            # Runtime state

Auto-Start on Boot

To enable VMs to automatically start after a host reboot, first install the systemd service:

# Install the systemd service (if not already installed)
sudo ./scripts/install-service.sh

# Enable the service
sudo systemctl enable vmm

# Check status
sudo systemctl status vmm

VMs with auto_start: true (the default) will be started automatically.

SSH Key Injection

VMM automatically injects SSH public keys into VMs at startup. When you create a VM with the --ssh-key flag, the specified public key is stored in the VM configuration. When the VM starts, VMM:

  1. Mounts the VM's rootfs image
  2. Creates /root/.ssh/ directory if needed
  3. Writes the public key to /root/.ssh/authorized_keys
  4. Sets correct permissions (700 for directory, 600 for file)
  5. Unmounts and boots the VM

This allows passwordless SSH access as root using your existing SSH key pair.

Note: SSH key injection requires root privileges (for mounting the rootfs image).

DNS Configuration

VMM automatically configures DNS in VMs at startup. By default, VMs use public DNS servers:

  • 8.8.8.8 (Google)
  • 8.8.4.4 (Google)
  • 1.1.1.1 (Cloudflare)

To use custom DNS servers, specify them when creating the VM:

# Use Quad9 and Cloudflare DNS
sudo vmm create myvm --dns 9.9.9.9 --dns 1.0.0.1

# Use corporate DNS
sudo vmm create myvm --dns 10.0.0.53 --dns 10.0.0.54

DNS configuration is written to /etc/resolv.conf in the VM's rootfs each time the VM starts.

Host Directory Mounting

VMM can mount host directories inside VMs, making them accessible as block devices. This is useful for sharing code, data, or configuration between the host and VMs.

How It Works

Since Firecracker doesn't support virtio-fs, VMM uses a block device approach:

  1. At VM start, an ext4 image is created from each host directory
  2. The image is attached as an additional block device (/dev/vdb, /dev/vdc, etc.)
  3. Fstab entries are injected into the VM rootfs for auto-mounting
  4. The VM boots with mounts available at /mnt/<tag>
Creating a VM with Mounts
# Single mount (read-write by default)
sudo vmm create myvm --mount /home/user/code:code --ssh-key ~/.ssh/id_ed25519.pub

# Multiple mounts with different modes
sudo vmm create myvm \
  --mount /home/user/code:code:ro \
  --mount /home/user/output:output:rw \
  --ssh-key ~/.ssh/id_ed25519.pub

# Start the VM
sudo vmm start myvm

The mount format is: /host/path:tag[:ro|rw]

  • /host/path - Absolute path to the directory on the host
  • tag - Name for the mount (alphanumeric, dashes, underscores only)
  • ro|rw - Optional mode, defaults to rw (read-write)
Accessing Mounts in the VM

After the VM starts, mounts are available at /mnt/<tag>:

# SSH into the VM
vmm ssh myvm

# Inside the VM:
ls /mnt/code      # Your mounted directory
cat /mnt/code/README.md
Syncing Mount Contents

If you make changes to the host directory while the VM is stopped, the changes will be included when you start the VM (the mount image is recreated from the host directory at each start).

To explicitly sync a mount image:

# Stop the VM first
sudo vmm stop myvm

# Sync the mount
sudo vmm mount sync myvm code

# Start the VM
sudo vmm start myvm
Listing Mounts
vmm mount list myvm
# Output:
# Mounts for VM 'myvm':
#   code: /home/user/code -> /mnt/code (ro) [/dev/vdb]
#   output: /home/user/output -> /mnt/output (rw) [/dev/vdc]
Limitations
  • Mount images are snapshots - changes inside the VM are not reflected back to the host
  • The VM must be stopped to sync mount contents from the host
  • Mount tags must be unique within a VM

Custom Docker Images

VMM can import Docker images as VM root filesystems. This allows you to use your existing Docker images as the base for VMs.

Importing an Image
# Import Ubuntu 22.04 as a base image
sudo vmm image import ubuntu:22.04 --name ubuntu-base

# Import with a larger size (default is 2GB)
sudo vmm image import ubuntu:22.04 --name ubuntu-large --size 4096

# Import a custom image from a registry
sudo vmm image import myregistry/myapp:latest --name myapp

The import process:

  1. Exports the Docker container filesystem
  2. Installs systemd, openssh-server, and networking tools
  3. Configures the image for Firecracker (serial console, SSH, networking)
  4. Creates an ext4 filesystem image
Using Custom Images
# Create a VM using the imported image
sudo vmm create myvm --image ubuntu-base --ssh-key ~/.ssh/id_ed25519.pub

# Start the VM
sudo vmm start myvm
Requirements
  • Docker must be installed and accessible
  • Only Debian/Ubuntu-based images are currently supported
  • The import process requires root privileges
Managing Images
# List all available images
vmm image list

# Delete an imported image
sudo vmm image delete ubuntu-base

VM Rootfs Snapshots

You can snapshot a VM's root filesystem and save it as a reusable base image. This is useful for installing tools and configuring a VM once, then creating multiple VMs from that template.

Creating a Snapshot

The VM must be stopped before snapshotting. The snapshot is automatically shrunk to minimum size to save disk space.

# Set up a template VM
sudo vmm create template --cpus 2 --memory 1024 --ssh-key ~/.ssh/id_ed25519.pub
sudo vmm start template

# SSH in and install your tools
vmm ssh template
# root@template:~# apt-get install -y python3 nodejs git
# root@template:~# exit

# Stop the VM and snapshot it
sudo vmm stop template
sudo vmm image snapshot template --name dev-tools
Using a Snapshot
# Create new VMs from the snapshot
sudo vmm create dev1 --image dev-tools --ssh-key ~/.ssh/id_ed25519.pub
sudo vmm create dev2 --image dev-tools --ssh-key ~/.ssh/id_ed25519.pub

# Each VM gets its own copy of the rootfs, resized to the configured disk size
sudo vmm start dev1
sudo vmm start dev2
How It Works
  1. Copies the VM's rootfs (ext4 image) to the shared images directory
  2. Runs e2fsck to verify filesystem consistency
  3. Runs resize2fs -M to shrink the filesystem to minimum size
  4. Truncates the file to match (e.g., a 1024 MB rootfs might shrink to ~160 MB)
  5. When a new VM is created from the snapshot, the image is copied and resized back to the VM's --disk size
Notes
  • Only the rootfs is captured — mounts and kernel selection are not included
  • The snapshot is a point-in-time copy; changes to the original VM after snapshotting are not reflected
  • SSH keys and DNS config are re-injected at VM start time, so the new VM gets its own configuration

Custom Kernels

VMM supports custom Linux kernels, allowing you to run newer kernel versions or kernels with specific configurations in your VMs.

Listing Available Kernels
vmm kernel list
# Output:
# Available kernels:
#   - k8s-kernel             32.4 MB  Kubernetes cluster kernel (Linux 6.6 LTS, Cilium/BPF)
#   - vmlinux.bin            72.7 MB  General-purpose VM kernel (Linux 6.1 LTS) (default)
Importing a Pre-built Kernel

If you have a pre-built vmlinux binary (uncompressed kernel), you can import it directly:

# Import a kernel binary
sudo vmm kernel import /path/to/vmlinux --name my-kernel

# Force overwrite an existing kernel
sudo vmm kernel import /path/to/vmlinux --name my-kernel --force

The kernel must be:

  • An uncompressed vmlinux ELF binary (not bzImage or zImage)
  • Built for the same architecture as the host (x86_64 or aarch64)
  • Configured with Firecracker-compatible options (virtio, serial console, etc.)
Building a Kernel from Source

VMM includes a build script that compiles Firecracker-compatible kernels from source:

# Build a 6.1 LTS kernel (default profile)
sudo vmm kernel build --version 6.1 --name kernel-6.1

# Supported versions: 5.10, 6.1, 6.6, 6.12

For the security testing profile with broad subsystem coverage, use the build script directly:

sudo bash scripts/build-kernel.sh --version 6.12 --name security-kernel --config-profile security
Build Requirements

The build script requires these packages:

sudo apt-get install build-essential flex bison bc libelf-dev libssl-dev wget
What the Build Script Does
  1. Downloads kernel source from kernel.org
  2. Downloads Firecracker's recommended kernel configuration
  3. Builds an uncompressed vmlinux binary with all drivers built-in
  4. Installs the kernel to /var/lib/vmm/images/kernels/<name>

Build time is typically 5-15 minutes depending on your system.

Using a Custom Kernel
# Create a VM with a custom kernel
sudo vmm create myvm --kernel kernel-6.1 --ssh-key ~/.ssh/id_ed25519.pub

# Start the VM
sudo vmm start myvm

# Verify the kernel version
vmm ssh myvm -- uname -r
# Output: 6.1.119
Deleting a Kernel
# Delete a custom kernel
sudo vmm kernel delete kernel-6.1

Note: You cannot delete the default kernel (vmlinux.bin). If VMs are configured to use a kernel you're deleting, they will fail to start until reconfigured.

When to Use Custom Kernels
  • Newer kernel features: Run a different kernel version than the default (6.1 LTS)
  • Security patches: Use a specific LTS kernel with security fixes
  • Custom configurations: Build kernels with specific options enabled
  • Testing: Test your application against different kernel versions

Troubleshooting

KVM not available
Error: /dev/kvm not found

Ensure:

  1. Your CPU supports virtualization (Intel VT-x or AMD-V)
  2. Virtualization is enabled in BIOS
  3. KVM modules are loaded: sudo modprobe kvm_intel or sudo modprobe kvm_amd
Permission denied on /dev/kvm
# Add your user to the kvm group
sudo usermod -aG kvm $USER
# Log out and back in
Network not working in VM

Ensure IP forwarding is enabled:

sudo sysctl -w net.ipv4.ip_forward=1

Check iptables rules:

sudo iptables -t nat -L -n

Test connectivity from host:

ping 172.16.0.2
VM can't reach the internet

Verify the host_interface in your config matches your actual network interface:

# Find your network interface
ip route | grep default
# Example output: default via 192.168.1.1 dev wlp3s0

# Check your config
cat ~/.config/vmm/config.json

# Update host_interface if needed (e.g., change "eth0" to "wlp3s0")

After updating the config, restart your VM for the NAT rules to be recreated with the correct interface.

VM won't start

Check the VM log:

cat /var/lib/vmm/logs/<vmname>.log

Check Firecracker socket:

ls -la /var/lib/vmm/sockets/
VM shows as stopped when running

Ensure you're checking with vmm list (no sudo required). The tool correctly detects running VMs even when run as non-root.

Development

Building from Source
# Install Go 1.21+
# Clone the repo
git clone https://github.com/raesene/baremetalvmm.git
cd baremetalvmm

# Build both binaries
make build-all

# Or build individually
go build -o vmm ./cmd/vmm/
go build -o vmm-web ./cmd/vmm-web/

# Run tests
go test ./...
Project Structure
├── cmd/
│   ├── vmm/main.go           # CLI entry point
│   └── vmm-web/main.go       # Web UI entry point
├── internal/
│   ├── config/               # Configuration management
│   ├── vm/                   # VM struct and persistence
│   ├── cluster/              # Kubernetes cluster management
│   ├── firecracker/          # Firecracker SDK wrapper
│   ├── network/              # TAP/bridge networking
│   ├── image/                # Kernel/rootfs management
│   ├── mount/                # Host directory mount management
│   └── web/                  # Web UI server, handlers, auth
├── web/
│   ├── embed.go              # Go embed directive for assets
│   ├── templates/            # HTML templates (HTMX + Tailwind)
│   └── static/               # JS/CSS assets (htmx, sse, styles)
├── scripts/
│   ├── install.sh            # Installation script
│   ├── uninstall.sh          # Uninstallation script
│   ├── install-service.sh    # Systemd service installation (optional)
│   ├── build-kernel.sh       # Custom kernel build script
│   ├── build-rootfs.sh       # Custom rootfs build script
│   └── vmm.service           # Systemd service unit file
└── go.mod                    # Go modules

AI Agent Skill

The skills/vmm-usage/ directory contains a Claude Code skill that teaches AI agents how to use vmm. It covers:

  • VM lifecycle (create, start, stop, delete)
  • SSH access and running commands in VMs
  • Available kernels and rootfs images
  • Creating reusable images via snapshots
  • Kubernetes cluster creation and management

To use the skill, add it to your Claude Code configuration or reference it directly. The skill assumes vmm is already installed on the target host.

Known Limitations

  1. Linux only - Firecracker only runs on Linux with KVM
  2. Root required - VM start/stop and networking require root privileges
  3. No GPU passthrough - Firecracker limitation
  4. No live migration - VMs must be stopped to move

License

MIT License - see LICENSE file for details.

Acknowledgments

Directories

Path Synopsis
cmd
vmm command
vmm-web command
internal
cluster
config
firecracker
image
mount
network
vm
web

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.