gitlab-runner-virt-plugin

README

gitlab-runner-virt-plugin

gitlab-runner-virt-plugin is a Fleeting plugin for GitLab-Runner docker-autoscaler setups backed by libvirt. When configured for single-use instances, it gives GitLab Runner a fresh sandbox VM per job, similar to the disposable job environments used by managed CI and build services such as GitHub-hosted runners, GitLab.com hosted runners, CircleCI Cloud, Docker Hub Automated Builds, Azure Pipelines Microsoft-hosted agents, Google Cloud Build, AWS CodeBuild, and Bitbucket Pipelines.

It does three main things:

1. Clones a Flatcar base image from a libvirt storage pool.
2. Generates an Ignition config from the Runner SSH connector settings.
3. Boots a libvirt VM and reports its SSH address back to Runner.

The generated Ignition config:

• creates or updates the SSH user from connector_config.username
• hashes connector_config.password into Ignition passwordHash
• derives and installs an SSH public key when connector_config.key is set
• copies additional PEM-encoded CA root certificates from plugin_config.ca_certificates_path when configured
• refreshes Flatcar's system trust store before docker.service starts when custom CAs are provided
• enables docker.service
• sets /etc/hostname to the VM name

For Runner SSH connectivity, use connector_config.key or key_path. connector_config.password is written into Ignition, but password-based SSH authentication does not work with the Runner/Fleeting SSH connector.

Requirements

• A working local libvirt daemon, typically reachable as qemu:///system
• A dedicated GitLab Runner libvirt/QEMU server. This plugin assumes it is the only service creating or running QEMU guests on the server; do not share the host with unrelated QEMU/libvirt workloads.
• A qcow2 Flatcar QEMU image already imported into a libvirt storage pool The documented Flatcar image is https://stable.release.flatcar-linux.net/amd64-usr/current/flatcar_production_qemu_image.img Copy the image onto the libvirt host, typically into /var/lib/libvirt/images/, before importing it into the storage pool. The plugin validates this base volume format during startup and rejects non-qcow2 images.
• A libvirt network that hands out DHCP leases
• A state_dir path that is readable by the QEMU/libvirt host process

Notes:

• This is implemented for a dedicated GitLab Runner server where the runner plugin owns the QEMU/libvirt workload lifecycle.
• The plugin injects Ignition through QEMU fw_cfg using Flatcar's libvirt path opt/org.flatcar-linux/config.
• Address discovery defaults to libvirt lease lookup, then falls back to guest-agent and ARP based lookup.
• The implementation assumes the Runner can reach the guest IP that libvirt reports. With the default NAT network this usually means the Runner is running on the same host as libvirt.

Plugin Config

Required fields:

• max_size
• pool_name
• base_volume_name or base_volume_path

Common fields:

{
  "uri": "qemu:///system",
  "pool_name": "default",
  "base_volume_name": "flatcar_production_qemu_image.img",
  "network_name": "default",
  "state_dir": "/var/lib/libvirt/gitlab-runner-virt-plugin",
  "ca_certificates_path": "/etc/gitlab-runner/ca-roots",
  "max_size": 10,
  "vcpu_count": 2,
  "memory_mib": 4096,
  "disk_size_gib": 40,
  "disk_bus": "virtio",
  "network_model": "virtio",
  "address_source": "auto"
}

Supported fields:

• uri: libvirt connection URI, default qemu:///system
• pool_name: target storage pool for cloned runner disks
• base_volume_name: source Flatcar volume inside pool_name
• base_volume_path: source Flatcar volume path, alternative to base_volume_name
• network_name: libvirt network name, default default
• state_dir: where generated Ignition files are written, default /var/lib/libvirt/gitlab-runner-virt-plugin
• ca_certificates_path: optional host path to a PEM certificate file or directory of PEM certificate files; the plugin writes them into /etc/ssl/certs inside the VM and runs update-ca-certificates
• max_size: maximum number of VMs Runner may request
• vcpu_count: vCPU count per VM, default 2, minimum 1
• memory_mib: memory per VM in MiB, default 4096, minimum 256
• disk_size_gib: optional disk resize target; if larger than the base image, the cloned disk is expanded
• domain_type: libvirt domain type, default kvm
• machine_type: optional machine type passed into the libvirt XML os/type element
• disk_bus: disk bus for the runner root disk, default virtio; set ide only for legacy guest compatibility
• network_model: libvirt NIC model, default virtio
• address_source: auto, lease, agent, or arp

Security Model

For the strongest isolation, use:

• capacity_per_instance = 1
• max_use_count = 1

That makes Runner create one VM for one job and then discard it. The job container runs against the Docker daemon inside that VM, not the libvirt host's Docker daemon. In practice, this is the GitLab Runner equivalent of the disposable hosted CI job pattern: a fresh environment per job with teardown after the job completes. In this plugin, that fresh environment is a VM you operate.

These comparisons are about lifecycle and isolation, not provider equivalence. Hosted services may use VMs, containers, or managed workers, and the provider manages the worker fleet, image pipeline, and platform hardening. With this plugin, you manage the libvirt host, base image contents, patching, network boundaries, and access to secrets.

If you increase capacity_per_instance or max_use_count, you trade some isolation for better density and faster warm reuse.

Runner Example

Example full config.toml for a GitLab Runner using docker-autoscaler with this plugin:

concurrent = 4
check_interval = 0
connection_max_age = "15m0s"
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "myrunner"
  url = "https://gitlab.com"
  id = 52258673
  token = "<private token>"
  executor = "docker-autoscaler"
  shell = "sh"

  [runners.docker]
    image = "alpine:3.20"
    pull_policy = "if-not-present"
    # privileged mode and the Docker socket mount apply inside the guest VM.
    # With capacity_per_instance = 1 and max_use_count = 1, each job gets a
    # fresh sandbox VM that is deleted after the job completes.
    privileged = true
    volumes = ["/var/run/docker.sock:/var/run/docker.sock"]

  [runners.autoscaler]
    plugin = "fleeting-plugin-libvirt"

    capacity_per_instance = 1
    max_use_count = 1
    max_instances = 10
    delete_instances_on_shutdown = true
    log_internal_ip = true
    log_external_ip = true

    [runners.autoscaler.plugin_config]
      uri = "qemu:///system"
      pool_name = "default"
      base_volume_name = "flatcar_production_qemu_image.img"
      network_name = "default"
      state_dir = "/var/lib/libvirt/gitlab-runner-virt-plugin"
      ca_certificates_path = "/etc/gitlab-runner/ca-roots"
      max_size = 10
      vcpu_count = 1
      memory_mib = 512
      disk_size_gib = 17
      disk_bus = "virtio"
      network_model = "virtio"
      address_source = "lease"

    [runners.autoscaler.connector_config]
      os = "linux"
      arch = "amd64"
      protocol = "ssh"
      protocol_port = 22
      username = "core"
      key_path = "/etc/gitlab-runner/libvirt-runner"
      use_static_credentials = true
      timeout = "10m"

    [[runners.autoscaler.policy]]
      idle_count = 3
      idle_time = "20m0s"
      preemptive_mode = true

Reuse behavior:

• max_use_count = 1 means each VM is used for exactly one job and then deleted.
• max_use_count = 5 means a VM can be reused for up to five jobs before Runner schedules it for removal.
• capacity_per_instance = 1 keeps the security boundary at one job per VM.
• capacity_per_instance = 1 plus max_use_count = 1 is the closest match to the disposable hosted CI job isolation model.
• concurrent should typically be max_instances * capacity_per_instance.

Use SSH keys for Runner connectivity. Set key_path. The plugin will derive the matching public key and install it via Ignition. connector_config.password does not work for Runner SSH connectivity here.

To create the key referenced by key_path, run:

sudo install -d -m 700 /etc/gitlab-runner
sudo ssh-keygen -t ed25519 -N '' -f /etc/gitlab-runner/libvirt-runner
sudo chmod 600 /etc/gitlab-runner/libvirt-runner

Build

Build the plugin under the name Runner expects (plugin = "fleeting-plugin-libvirt" in config.toml):

go build -o fleeting-plugin-libvirt ./cmd/fleeting-plugin-libvirt

Install it onto the libvirt host:

install -m 0755 fleeting-plugin-libvirt /usr/local/bin/fleeting-plugin-libvirt

Integration Test

The Fleeting provisioning integration test builds the plugin binary, starts it through Fleeting, provisions one libvirt VM, connects over SSH, and deletes the instance again. Run it only on a dedicated libvirt test host. In addition to the storage-pool import described in Requirements, the test harness expects a copy of the Flatcar qcow2 image at the repo root (see internal/libvirtplugin/fleeting_integration_test.go).

make integration-test

Operational Notes

• Run this plugin on a dedicated GitLab Runner libvirt/QEMU host. It is not designed to coexist with other QEMU workload managers on the same server.
• Imported Flatcar images should be the official QEMU/libvirt-ready image format.
• state_dir must exist on the hypervisor host filesystem because libvirt passes the Ignition file to QEMU by host path.
• ca_certificates_path is read on the host running the plugin. It may point to a single PEM file or a directory of PEM files.
• Managed instances use the fixed gitlab-runner domain-name prefix. VM IDs use the format gitlab-runner-<plugin_boot_time>-<vm_create_time>-<suffix>, for example gitlab-runner-20260424-120000-20260424-121530-a1b2c3.
• The first timestamp identifies the plugin process boot generation. During startup, the plugin ignores non-matching names and removes generated VM IDs from older plugin boot generations.
• The final six-character hex suffix is also used to derive the VM MAC address, for example suffix a1b2c3 maps to 52:54:00:a1:b2:c3.
• virsh list can be used to see active VMs.
• The plugin deletes the libvirt domain definition, the cloned storage volume, and the generated Ignition file when Runner scales an instance down.