karpenter-provider-exoscale

README

karpenter-provider-exoscale

A Karpenter cloud provider for Exoscale that lets the cluster operator own node bootstrap.

It implements sigs.k8s.io/karpenter's CloudProvider interface against the Exoscale Compute API, so Karpenter can launch, consolidate and terminate Exoscale instances in response to pending pods — on any Kubernetes distribution: k3s, RKE2, kubeadm, or a custom image that configures itself.

Status: alpha. The API group is karpenter.k8s.exoscale/v1alpha1 and will change. This is a community project and is not affiliated with Exoscale.

The idea

Karpenter's job is to decide what node to launch. Something still has to decide how that node becomes a Kubernetes node — install a kubelet, configure networking, join the cluster.

Most providers answer that by generating the bootstrap themselves. That works when the provider knows your distribution's join mechanism, and stops working when it does not. k3s, RKE2 and similar distributions are the awkward case: their agent binary supplies its own kubelet, CNI and kube-proxy, and joins with a distribution-specific token rather than by writing a kubelet config and a bootstrap kubeconfig.

This provider answers it the other way round. ExoscaleNodeClass.userData is a free-form string that reaches the instance verbatim — nothing is merged into it, no field of it is reserved, and no bootstrap is generated. You write how nodes join; the provider does everything else.

Two things are asked of your script in return, and both are documented in docs/bootstrap.md with a worked k3s example:

1. Register with provider ID exoscale://<instance-uuid>, read at boot from the metadata service.
2. Register with the labels and taints Karpenter resolved — which the provider renders for you.

Relationship to Exoscale's official provider

Exoscale publishes exoscale/karpenter-provider-exoscale. It targets a different setup, and if that setup is yours, prefer it — it is maintained by Exoscale.

	exoscale/karpenter-provider-exoscale	This provider
Target clusters	Clusters built from Exoscale SKS node images	Any self-managed cluster
Node images	SKS node images	Any Exoscale template, public or private
Node configuration	sks-node-agent, driven by TOML user-data	Your script, whatever it is
userData	TOML, deep-merged into the provider's own bootstrap, with provider fields taking precedence	Free-form string, passed through verbatim
Join mechanism	kubeadm-style TLS bootstrap token, standard kubelet	Whatever your script does
Distributions	kubeadm-style, via the SKS node agent	k3s, RKE2, kubeadm, custom images
Helm chart	Not shipped	charts/
Maintained by	Exoscale	Community

The practical difference is the userData row. Because the official provider deep-merges your TOML into its own bootstrap configuration and its fields win, there is no way to substitute a different installer — which is what running a non-kubeadm distribution requires. Both designs are reasonable; they just serve different clusters.

Use the official provider if your nodes are SKS node images and join with a standard kubelet. Use this one if you run k3s, RKE2, a custom image, or otherwise need the bootstrap to be yours.

Supported Exoscale features

Feature	Support
Compute instances	Created individually, so a NodePool can scale to zero
Instance types	Discovered from the API, filtered to those authorized for the account
Zones	All; a NodeClass may span them
Templates	By UUID, or resolved by name per zone — newest build wins
Root disk	Sized per NodeClass, 10–51200 GiB
Security groups	Resolved by UUID or name
Private networks	Resolved by UUID, name or Exoscale label; attached after launch
Anti-affinity groups	Resolved by UUID or name (see the caveat below)
SSH keys	By name
Public IPv4	Optional per NodeClass
Instance labels	Reconciled in place, never forcing node replacement
GPU instance types	nvidia.com/gpu capacity advertised; priced above CPU types
Drift detection	NodeClass spec changes, and zone removal
Garbage collection	Orphaned instances reaped by label, with a grace period

Not supported: Instance Pools (they cannot be scaled to zero, which defeats the purpose), Elastic IPs, block storage volumes, spot or reserved capacity (Exoscale sells neither — every offering is on-demand).

Anti-affinity groups, a caveat. Exoscale places every member of an anti-affinity group on a distinct hypervisor. Instance creation therefore starts failing once a group's membership reaches the number of available hosts — a hard ceiling on how far a NodePool can scale, surfacing as a launch failure rather than a quota error. Prefer topology spread constraints across zones, and keep anti-affinity groups for small fixed sets of nodes.

Install

Karpenter core must already be able to run in your cluster: the CRDs it owns (NodePool, NodeClaim, NodeOverlay) plus this provider's ExoscaleNodeClass.

1. Credentials

Create a restricted IAM role and an API key for it — see docs/iam.md for the exact set of operations the controller needs and why. Then:

kubectl create namespace karpenter
kubectl create secret generic exoscale-credentials -n karpenter \
  --from-literal=EXOSCALE_API_KEY=EXO... \
  --from-literal=EXOSCALE_API_SECRET=...

2. CRDs and controller

Charts and the controller image are published per release. Pick a version from Releases and use it for both — the CRD schema has to match the API types the controller was built from, so they are cut from one commit and must be installed at the same version.

VERSION=0.1.0

helm upgrade --install karpenter-crd-exoscale \
  oci://ghcr.io/kubekanvas/karpenter-provider-exoscale/charts/karpenter-crd-exoscale \
  --version "$VERSION" \
  --namespace karpenter --create-namespace

helm upgrade --install karpenter-exoscale \
  oci://ghcr.io/kubekanvas/karpenter-provider-exoscale/charts/karpenter-exoscale \
  --version "$VERSION" \
  --namespace karpenter \
  --set settings.clusterName=my-cluster \
  --set settings.zone=de-fra-1 \
  --set credentials.existingSecret=exoscale-credentials

Every release also attaches the same charts as tarballs, which is the path to use if the registry is unavailable to you — a ghcr package is created private even for a public repository and has to be made public once, whereas release assets inherit the repository's visibility:

VERSION=0.1.0
BASE=https://github.com/kubekanvas/karpenter-provider-exoscale/releases/download/v$VERSION

helm upgrade --install karpenter-crd-exoscale "$BASE/karpenter-crd-exoscale-$VERSION.tgz" \
  --namespace karpenter --create-namespace
helm upgrade --install karpenter-exoscale "$BASE/karpenter-exoscale-$VERSION.tgz" \
  --namespace karpenter \
  --set settings.clusterName=my-cluster \
  --set settings.zone=de-fra-1 \
  --set credentials.existingSecret=exoscale-credentials

Controller image: ghcr.io/kubekanvas/karpenter-provider-exoscale/controller:<version>, built multi-arch for linux/amd64 and linux/arm64.

A worked example

An ExoscaleNodeClass describing what a node looks like, and a NodePool describing when to launch one. The userData here is abridged — the full version, including the private-network interface sequencing it depends on, is in docs/bootstrap.md.

apiVersion: karpenter.k8s.exoscale/v1alpha1
kind: ExoscaleNodeClass
metadata:
  name: default
spec:
  # Zones nodes may be launched into. Every resource selected below must exist in each.
  zones:
    - ch-gva-2
    - de-fra-1

  # Resolved by name per zone, because template UUIDs differ between zones. Pinning
  # spec.templateID instead ties the NodeClass to a single zone.
  templateSelectorTerms:
    - name: Linux Ubuntu 24.04 LTS 64-bit

  # Root disk in GiB. Exoscale bundles no disk with an instance type, so this is required.
  # It also sets the ephemeral-storage capacity Karpenter schedules against.
  diskSize: 50

  # Optional. Leaving this empty lets Karpenter consolidate onto any authorized instance
  # type, which is usually what you want; set it to hard-exclude families such as GPUs.
  instanceTypes:
    - standard.medium
    - standard.large
    - standard.extra-large

  securityGroupSelectorTerms:
    - name: k8s-nodes

  privateNetworkSelectorTerms:
    - name: k8s-internal

  sshKey: ops-team

  # Where the node joins the cluster. Passed to the instance verbatim.
  userData: |
    #!/bin/bash
    set -euo pipefail

    # Exoscale is CloudStack-derived: the AWS-style /latest/meta-data/ layout, not
    # /metadata/v1/. The UUID cannot be templated in — the API assigns it as the
    # instance is created, and this user data is part of that same request.
    INSTANCE_ID="$(curl -sf --retry 10 --retry-delay 2 \
      http://169.254.169.254/latest/meta-data/instance-id)"

    # See docs/bootstrap.md: the private NIC is attached asynchronously and has no
    # address until configured. Anything binding to a private address must wait for
    # both the interface and its DHCP lease.
    # ... wait_for_private_nic; configure_netplan; wait_for_lease ...

    curl -sfL https://get.k3s.io | INSTALL_K3S_EXEC="agent" sh -s - \
      --server https://10.0.0.10:6443 \
      --token "$(cat /etc/k3s-token)" \
      --kubelet-arg="provider-id=exoscale://${INSTANCE_ID}" \
      --node-label "{{ .NodeLabelsCSV }}" \
      --node-taint "{{ .NodeTaintsCSV }}"
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata:
  name: general
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.k8s.exoscale
        kind: ExoscaleNodeClass
        name: default
      requirements:
        - key: kubernetes.io/arch
          operator: In
          values: ["amd64"]
        - key: karpenter.k8s.exoscale/instance-family
          operator: In
          values: ["standard"]
        # Narrow by CPU rather than by naming instance types, so Karpenter can
        # consolidate onto a cheaper type it was never explicitly told about.
        - key: karpenter.k8s.exoscale/instance-cpu
          operator: In
          values: ["2", "4", "8"]
      expireAfter: 720h
  disruption:
    consolidationPolicy: WhenEmptyOrUnderutilized
    consolidateAfter: 1m
  limits:
    cpu: "100"
    memory: 400Gi

Apply both, watch the NodeClass become ready, and scale a workload:

kubectl get exoscalenodeclass       # Ready should become True
kubectl describe exoscalenodeclass default   # each selector has its own condition
kubectl get nodeclaims -w

ExoscaleNodeClass reference

Field	Required	Description
zones	no	Zones to launch into. Defaults to the controller's clusterZone.
templateID	one of	Template UUID. Zone-scoped, so this pins the NodeClass to one zone.
templateSelectorTerms	one of	Resolve the template by name (and optionally visibility) per zone.
diskSize	yes	Root disk in GiB, 10–51200. Must be at least the template's own size.
instanceTypes	no	Allow-list of <family>.<size> names. Empty means every authorized type.
userData	no	Free-form cloud-init user data. See docs/bootstrap.md.
sshKey	no	Name of an Exoscale SSH key.
securityGroupSelectorTerms	no	Select security groups by id or name.
privateNetworkSelectorTerms	no	Select private networks by id, name or labels.
antiAffinityGroupSelectorTerms	no	Select anti-affinity groups by id or name. See the caveat above.
publicIPAssignment	no	inet4 (default) or none.
labels	no	Extra Exoscale instance labels. Reconciled in place.
namePrefix	no	Prepended to the generated instance name.
kubelet	no	Kubelet settings used to compute allocatable capacity. See below.

Selector terms are ORed with each other; the fields within one term are ANDed. A term that matches nothing fails the NodeClass rather than being silently dropped — a node launched without the security group it was meant to have is a firewall hole, not a degraded launch.

spec.kubelet does not configure the kubelet — this provider writes no kubelet configuration. Karpenter uses it to compute how much of a node is allocatable. Your bootstrap script must apply the same values, or Karpenter will schedule against capacity the node does not have.

Node labels

Alongside the well known Kubernetes labels, nodes carry:

Label	Example
karpenter.k8s.exoscale/instance-cpu	4
karpenter.k8s.exoscale/instance-memory	8192 (MiB)
karpenter.k8s.exoscale/instance-family	standard
karpenter.k8s.exoscale/instance-size	large
karpenter.k8s.exoscale/instance-gpu-count	0

topology.kubernetes.io/zone and topology.kubernetes.io/region are both set to the Exoscale zone. Exoscale exposes no grouping above a zone, and its cloud-controller-manager labels nodes the same way.

On pricing

Exoscale publishes no price list through its API, and Karpenter needs a price for every offering in order to pick the cheapest type that fits and to decide whether consolidation is an improvement.

Both of those are comparisons — Karpenter never sums prices or reports a bill — so this provider derives a synthetic cost from each instance type's CPU, memory and GPU count rather than shipping a table of euros per hour that would go stale silently and be wrong for any account with negotiated rates. The ordering is what matters, and the derived model reproduces it.

If your account's rates are not proportional to resources, or you want spend-accurate consolidation, individual instance types can be overridden.

Notes on the implementation

A few Exoscale behaviours shape the design, and are worth knowing if you are reading the code or filing a bug:

• The API is addressed per zone by endpoint. A client built for ch-gva-2 cannot see an instance in de-fra-1, so clients are built and cached per zone and anything enumerating instances fans out across them. It is also why a provider ID — which carries no zone — may cost a lookup in several zones after a controller restart.
• Private networks are attached after creation. The create request has no field for them. That is the root of the boot-ordering problem documented in docs/bootstrap.md.
• Most mutations are asynchronous. They return an operation, not the finished resource, so a create is not complete — and the instance's UUID is not known — until that operation succeeds.
• There is no cloud-controller-manager in a plain k3s or kubeadm cluster. Nothing else reaps a Node object whose instance is gone, so Karpenter's own termination path deletes it.
• Nodes are discovered by an Exoscale instance label carrying the cluster name. Exoscale label keys cannot contain /, so the managed keys are flattened forms of the Kubernetes ones — karpenter-sh_managed-by rather than karpenter.sh/managed-by.

Development

make build        # compile
make test         # unit tests
make fmt vet      # format and vet
make generate     # regenerate CRDs and deepcopy functions

make generate runs controller-gen from go.tools.mod via go tool, so it needs no globally installed tool. Regenerate and commit whenever pkg/apis/ changes — CI checks the tree is clean.

Contributing

Issues and pull requests are welcome. Two things help a lot:

• If you are reporting a node that never registers, work through the debugging checklist at the end of docs/bootstrap.md first and say which step failed.
• If you are changing behaviour that depends on an Exoscale API shape, say whether you verified it against the live API or inferred it — the code marks the difference with TODO comments, and it is worth keeping honest.

License

Apache-2.0. See LICENSE.