karpenter-provider-rackspace-spot

README

Karpenter Provider for Rackspace Spot

A Karpenter cloud provider for Rackspace Spot. Provisions SpotNodePool and OnDemandNodePool resources in a Cloudspace in response to unschedulable pods, with smart bid pricing driven by Rackspace's published percentile feed.

Status: Alpha — driven end-to-end on a real Cloudspace, but APIs and chart values may shift before v1.0.

Requirements

• A Rackspace Spot account and at least one Cloudspace (the target Kubernetes cluster).
• A Rackspace Spot OAuth refresh token (from your account profile in the Rackspace Spot console).
• helm 3.8+ (for OCI chart support).
• Kubernetes v1.31+ inside the Cloudspace.

Provider	Karpenter core	Kubernetes	Rackspace Spot SDK
v0.1.x	v1.12.x	v1.31+	spot-go-sdk v0.2+

Installation

Grab a refresh token from your Rackspace Spot account page, then install with one of:

# inline (token from env var or pasted directly)
helm install karpenter \
  oci://ghcr.io/kanya-approve/charts/karpenter-provider-rackspace-spot \
  --version 0.1.6 \
  --namespace karpenter --create-namespace \
  --set spot.cloudspaceName=my-cloudspace \
  --set spot.refreshToken=$SPOT_REFRESH_TOKEN

# pre-existing secret
kubectl create namespace karpenter
kubectl -n karpenter create secret generic karpenter-spot \
  --from-literal=refreshToken=$SPOT_REFRESH_TOKEN

helm install karpenter \
  oci://ghcr.io/kanya-approve/charts/karpenter-provider-rackspace-spot \
  --version 0.1.6 \
  --namespace karpenter \
  --set spot.cloudspaceName=my-cloudspace \
  --set spot.existingSecret=karpenter-spot

spot.cloudspaceName is required — one Helm install runs one operator for one Cloudspace.

The chart ships the provider's RackspaceSpotNodeClass CRD and the upstream karpenter.sh/v1 NodePool + NodeClaim CRDs.

For development against the source tree, swap oci://... for charts/karpenter and --set image.tag=latest.

Snapshot images (:latest, :sha-<commit>) ship on every push to latest; snapshot charts (<chart-version>-latest.<short-sha>) ship when charts/** changes. Tag releases (v*.*.*) publish plain-versioned images and charts.

Documentation

• Configuration
• Bidding
• Workload affinity
• Node repair (opt-in)
• Authentication
• Architecture notes
• Development

Configuration

A minimal NodeClass + NodePool pair that provisions spot capacity (the Cloudspace itself is configured at install time on the operator, not per NodeClass):

apiVersion: karpenter.rackspace.com/v1
kind: RackspaceSpotNodeClass
metadata: {name: default}
spec:
  bidPercentile: P80              # P20 | P50 | P80 (default P80). See "Bidding".
---
apiVersion: karpenter.sh/v1
kind: NodePool
metadata: {name: default}
spec:
  template:
    spec:
      nodeClassRef:
        group: karpenter.rackspace.com
        kind: RackspaceSpotNodeClass
        name: default
      requirements:
        - {key: karpenter.sh/capacity-type, operator: In, values: ["spot"]}
        - {key: kubernetes.io/arch,         operator: In, values: ["amd64"]}
        - {key: kubernetes.io/os,           operator: In, values: ["linux"]}
      expireAfter: 720h
  limits: {cpu: "100", memory: 400Gi}
  disruption: {consolidationPolicy: WhenEmptyOrUnderutilized, consolidateAfter: 30s}

Working manifests live in config/samples/.

Node capacity

A ServerClass advertises the flavor as sold, which is more than the kubelet ends up offering: the guest kernel takes a cut of the advertised RAM, mkfs takes a cut of the disk, and the kubelet reserves a fixed slice on top. Published capacity has to stay at or below what the node reports, or the scheduler nominates pods onto nodes that cannot hold them and the provisioner relaunches the same too-small flavor indefinitely.

The reserved slice is read from the node kubelet config and is the same on every flavor: 500m CPU, 1024Mi memory plus a 100Mi eviction threshold, and 2Gi disk plus a 10% nodefs eviction threshold. On top of that, the first node of a given flavor is provisioned against a conservative estimate — the advertised size less 3% memory and 4% disk. Once a node of that flavor registers, the instancetype.capacity controller records what it actually reported and the estimate is replaced by the measurement, keeping the smallest value ever seen. The cache is in-memory and rebuilds from the running nodes on startup.

Env var	Default	Notes
VM_MEMORY_OVERHEAD_PERCENT	0.03	Shaved off advertised memory until a node of that flavor is measured.
RACKSPACE_KUBE_RESERVED_CPU	500m
RACKSPACE_KUBE_RESERVED_MEMORY	1024Mi

Set them via the chart's extraEnv if Rackspace retunes its node image.

Launch timeout

Rackspace accepts a bid within seconds but can take far longer to attach a server to the pool — 17 minutes, measured end to end, for mh.vs1.large-iad. Karpenter's registration timeout is a fixed 15-minute constant with no flag, and when it fires it deletes the NodeClaim and the pool, so Rackspace restarts from zero and the flavor can never provision.

The provider therefore holds the NodeClaim in Launched=Unknown with reason AwaitingServerAssignment until the Cloudspace reports a server attached to the pool, rather than reporting it launched as soon as the bid is won. That puts the wait under LaunchTimeout, which upstream exposes as a variable, defaulting here to 30 minutes. Pool creation is idempotent — the pool is named for the NodeClaim UID — so the retries re-use the in-flight pool instead of restarting it.

If no server is ever attached, the NodeClaim is deleted at the timeout, the pool is cleaned up, and the NodePool is marked NodeRegistrationHealthy=False so karpenter stops favouring that flavor.

Env var	Default	Notes
RACKSPACE_LAUNCH_TIMEOUT	30m	Any Go duration string.

Bidding

Every spot pool's bid is computed at Create time as max(current_market, percentile) × 1.05, clamped up to the ServerClass's minBidPrice floor and rounded to Rackspace's precision rules (≤3 decimal places, or multiples of $0.01 above $0.05). There is no user-set bidPrice field on the NodeClass — the only knob is which percentile of Rackspace's published 30-day distribution to bid at:

• P80 (default): bid clears ~80% of typical price ticks. Stable, more expensive on volatile SKUs.
• P50: bid clears ~50%. Cheaper, more eviction-prone.
• P20: cheapest, most eviction-prone.

Cost ceilings should be expressed via NodePool.limits or NodePool requirements on node.kubernetes.io/instance-type, not via a per-pool bid.

Pricing data comes from Rackspace's public S3 feed. The controller fetches it live (5-min cache) and falls back to an embedded snapshot at pkg/providers/pricing/initial-prices.json (refreshed weekly via .github/workflows/update-pricing.yaml).

Workload affinity

The Rackspace/OpenStack cloud-controller overwrites node.kubernetes.io/instance-type (with the underlying VM SKU, e.g. compute1-4) and topology.kubernetes.io/region (with the OpenStack region code, e.g. HKG) post-registration. Workloads using nodeSelector or affinity should target Karpenter-set labels instead:

• karpenter.sh/nodepool=<name> — which NodePool spawned the node
• karpenter.sh/capacity-type=spot|on-demand — what kind of capacity
• karpenter.rackspace.com/managed=true — Karpenter-provisioned (vs. seed/manual pools)
• karpenter.rackspace.com/rackspacespotnodeclass=<name> — which NodeClass

Don't filter on node.kubernetes.io/instance-type=<ServerClass> — the CCM clobbers it.

Node repair (opt-in)

RepairPolicies returns a non-empty list (Ready=False/Unknown or NetworkUnavailable=True for 30+ min triggers replacement), but Karpenter's node.health controller is gated behind the NodeRepair feature gate, off by default. A transient kubelet hiccup or control-plane blip can briefly flip many nodes to Ready=Unknown, and an over-eager repair controller would race to wipe them; the conservative default leaves that decision to the operator.

To opt in:

helm upgrade karpenter ... \
  --set 'extraEnv[0].name=FEATURE_GATES' \
  --set 'extraEnv[0].value=NodeRepair=true\,ReservedCapacity=true\,SpotToSpotConsolidation=false\,NodeOverlay=false\,StaticCapacity=false'

The full set has to be passed because FEATURE_GATES is parsed as a complete override, not a partial merge.

Authentication

The controller reads SPOT_REFRESH_TOKEN (and other SPOT_* env vars per spot-go-sdk Config). The chart wires this from spot.refreshToken or spot.existingSecret.

What's verified

Flow	Notes
Scale up	spot + on-demand, single + multi-replica, mixed shapes
Scale down	consolidation drains via eviction API, deletes the pool
Smart bidding	percentile knob + minBidPrice floor clamp + precision rounding
External pool delete recovery	registrationTimeout triggers Delete, NodeClaim replaced
Node repair	Behind NodeRepair feature gate — see above

What's not yet in

• #2 Preemption webhook receiver — Rackspace gives ~6 min notice via a push webhook; we don't run a receiver yet, so spot evictions are ungraceful (pods get hard-killed when the server vanishes).

Architecture notes

• 1 Rackspace pool per NodeClaim, desired_server_count=1. Rackspace's smallest provisioning unit is a pool with a replica count; Karpenter wants per-machine lifecycle. This mapping gives clean Karpenter semantics at the cost of higher pool count under churn.
• Cloudspace is single-region. Each RackspaceSpotNodeClass references one Cloudspace; the region is whatever the Cloudspace is in (cached in status.region, never user-set). Multi-Cloudspace deployments run one Karpenter install per Cloudspace.
• Pool name = NodeClaim UID. Rackspace's admission webhook requires lowercase-UUID names. Karpenter-managed pools are distinguished by the karpenter.rackspace.com/managed=true label rather than a name prefix.
• karpenter.sh/v1 core CRDs ship with the chart. The binary embeds sigs.k8s.io/karpenter and runs its core controllers (provisioning, disruption, nodeclaim, nodepool) alongside our RackspaceSpotNodeClass reconciler and the nodelink reconciler.
• nodelink reconciler bridges the providerID gap. Rackspace's CCM sets Node.spec.providerID = openstack:///<vm-uuid>; our Create() returns rackspacespot://<cs>/<kind>/<nodeclaim-uid> because we don't know the VM UUID at create time (Rackspace's auction decouples bid acceptance from server assignment). When a Node joins carrying our karpenter.rackspace.com/managed=true label, nodelink patches the NodeClaim's Status.ProviderID to match the CCM-set value so Karpenter's lifecycle controller can complete the binding.

Development

make generate       # regenerate CRDs + deepcopy; sync chart crds/
make build          # build the controller binary
make test           # unit tests (mocks via spot-go-sdk/api/v1/mocks)
make image          # ko build + push (set KO_DOCKER_REPO and TAG)
make chart-lint     # helm lint
make update-pricing # refresh the embedded pricing snapshot from the S3 feed

CI runs make build, make test, make chart-lint, make chart-template on every PR.

Community

• Karpenter discussion and troubleshooting: #karpenter in the Kubernetes Slack.
• Bug reports and feature requests: GitHub Issues.

Contributing

See CONTRIBUTING.md for the pull-request workflow and dev setup.

Code of Conduct

Participation in this project is governed by the Code of Conduct, adapted from Contributor Covenant 1.4.

References

• Karpenter — the upstream autoscaler this provider plugs into.
• Rackspace Spot — the platform we provision against.
• spot-go-sdk — official Go SDK for Rackspace Spot.
• Layout and conventions modeled after oracle/karpenter-provider-oci, aws/karpenter-provider-aws, and sergelogvinov/karpenter-provider-proxmox.

License

Licensed under the Apache License, Version 2.0. See LICENSE.