README
¶
cluster-api-provider-nico
Kubernetes Cluster API (CAPI) infrastructure provider to provision bare metal nodes in NVIDIA Infra Controller (NICo).
CAPNICo is a Cluster API infrastructure provider and does one job: give
Cluster API the machines it asks for. It does not install Kubernetes and it does
not join nodes — the kubeadm bootstrap and control-plane providers do that. You
declare a Cluster and a MachineDeployment as usual, and this provider turns
each requested machine into a NICo instance on real hardware.
NicoClusterholds the target site, and may optionally reference a per-cluster credentials Secret.NicoMachinerepresents one NICo instance managed by Cluster API, and carries the VPC.NicoMachineTemplatesupportsKubeadmControlPlaneandMachineDeployment.NicoIdentityreports the health of the credentials Secret it names. The manager checks every Identity in its watch scope.
Features
- Bare-metal machines through the Cluster API contract. A
MachineDeploymentorKubeadmControlPlaneprovisions real hardware with no separate workflow. - Two credential styles. A static bearer token, or OAuth2 client credentials, supplied through a Secret.
- Per-cluster credentials.
NicoClustermay reference its own Secret, so one management cluster can drive several sites. - External control-plane endpoints, including a kube-vip template and a developer flow for a single requested IP.
- Machine repair and reboot, both driven by annotations.
How this fits with other tools
This provider does one job and leaves the rest to the standard Cluster API components:
| Component | What it does | Relationship |
|---|---|---|
| Cluster API core | Decides which machines should exist | Asks this provider for them |
| Kubeadm bootstrap and control-plane providers | Install Kubernetes and join nodes | This provider does neither |
| NVIDIA Infra Controller (NICo) | Provisions the physical instances | This provider drives its API |
| Other infrastructure providers (CAPD, CAPA, and so on) | The same role on a different substrate | Interchangeable — swap the provider, keep the workflow |
If you already run Cluster API, your workflow does not change. You declare the same objects, and this provider satisfies them with NICo hardware.
Documentation
- docs/getting-started.md — start here. First cluster three ways: the in-repo fake, a local NICo, and a production site. What must exist on the NICo side first, the credentials Secret, what the OS image has to contain, and the traps.
- docs/architecture.md — how the pieces fit: the CRDs and which fields live on which, credential resolution and client caching, machine reconciliation and what the finalizer guards, provider ID and node matching, teardown order, and the annotation-driven repair and reboot contracts. Read this before changing controller behaviour.
- docs/api-reference.md: field-by-field reference for all five CRDs and every condition and reason
- docs/troubleshooting.md: symptom, cause, and fix for problems beyond getting-started.md's stall-reason table
- CONTRIBUTING.md — development setup and the pull-request flow
- RELEASE.md — versioning and the Cluster API contract
- SECURITY.md — vulnerability reporting
- NICo documentation — the infrastructure this provider drives
Compatibility
CAPNICo tracks two independent version axes: the Cluster API contract it
implements, and the sigs.k8s.io/cluster-api module version it is built
and tested against. A newer module version does not by itself mean a newer
contract.
| CAPNICo release series | CAPI contract | Built against sigs.k8s.io/cluster-api |
|---|---|---|
| 0.0.x | v1beta2 | v1.13.4 |
metadata.yaml records the contract per release series; see
RELEASE.md for when it changes.
Support level: Experimental.
Community and support
- Questions and discussion — ask in GitHub Discussions, where an answer can be marked as the accepted one.
- Bug reports and feature requests — open a GitHub issue.
- Code of Conduct — everyone taking part is expected to follow the Code of Conduct.
- Security — report vulnerabilities as described in SECURITY.md. Do not open a public issue for a security report.
- Cluster API itself — for questions about Cluster API rather than this
provider, the upstream project runs
#cluster-apion Kubernetes Slack, and the Cluster API book covers the core concepts.
How It Works
Architecture
Three reconcilers, five CRDs, and one outbound API. Everything runs in the management cluster; nothing is installed on the provisioned nodes.
Cluster API core management cluster
├── Cluster ──────────► NicoCluster (target site, credentials)
├── MachineDeployment ─────────► NicoMachineTemplate
└── Machine ──────────► NicoMachine (one instance, one VPC)
│
│ CAPNICo controllers
▼
NicoIdentity ── names ──► Secret ──► NICo REST API
(credential health) (credentials) │
▼
bare-metal instance
(iPXE boot, kubeadm cloud-init)
Cluster API owns the desired state. The cluster and machine controllers
reconcile each NicoMachine into one NICo instance and report the provider ID
back, which is how the node is matched. The kubeadm providers take over from
there. Separately, the NicoIdentity controller periodically checks the Secret
each NicoIdentity names against NICo and records the result in the
Identity's status, without affecting provisioning.
docs/architecture.md has the detail: field ownership, credential resolution and client caching, what the finalizer guards, teardown order, and the repair and reboot contracts.
Credentials and reconciliation
The provider uses a Secret for NICo connection details and authentication. The Secret supports either a static bearer token or OAuth2 client-credentials.
Each NicoCluster supplies the target site. The VPC is set per machine, on
NicoMachine.spec.vpcID.
Each NicoMachine becomes one NICo instance, and NicoMachineTemplate is intended for use from KubeadmControlPlane and MachineDeployment.
The controller discovers the current tenant through the NICo API and caches it in the client.
NICo clients themselves are cached per Secret resourceVersion, so external
rotations of the credentials Secret (for example by External Secrets Operator)
are picked up on the next reconcile without restarting the manager.
Machine reconciliation treats instance creation and instance readiness separately, and create conflicts are handled idempotently by looking up an existing instance by name.
The provider expects the NICo API to be reachable from the management cluster, and it assumes the target VPC and subnet or VPC prefix are already defined for the machines you want to provision.
The NicoMachine's iPXE script must boot an OS image that can consume kubeadm cloud-init user data.
Local development (no NICo access needed)
make tilt-up creates a local kind cluster, installs CAPNICo, and runs it
against the fake NICo endpoint shipped in this repository. It needs no
hardware and no access to a real NICo deployment.
make tilt-up # web UI on localhost:10352
export KUBECONFIG=~/.kube/capnico.kubeconfig # or: eval "$(make kubeconfig)"
kubectl apply -f examples/cluster-fake.yaml
See docs/development.md for the complete workflow, fake-NICo seeding, and test commands.
Failure domains
CapNICo implements the Cluster API failure domain contract so control-plane
machines can be spread across correlated-failure boundaries such as rack
groups. NicoCluster.status.failureDomains lists the domains NICo offers for
the cluster's site, and NicoCluster.spec.failureDomainLabelKey selects
which NICo Machine label carries the domain name. An unset key disables the
feature entirely.
See docs/architecture.md for the full mechanism, capability requirements, and placement-failure semantics.
Install
CAPNICo requires cert-manager in the management cluster for its validating
webhook. clusterctl init installs cert-manager when required.
Initialize the core Cluster API controllers and the kubeadm providers:
export KUBECONFIG=/path/to/management-cluster.kubeconfig
clusterctl init --bootstrap kubeadm --control-plane kubeadm
Build and push the controller image you want to run:
make docker-build IMG=ghcr.io/your-org/cluster-api-provider-nico:latest
Update config/manager/manager.yaml to use that image, then install the provider:
kubectl apply -k config/default
Alternatively, install the native Helm chart and supply the same controller image without editing the Kustomize source:
helm upgrade --install capi-provider-nico ./chart \
--namespace capnico-system \
--create-namespace \
--set manager.image.repository=ghcr.io/your-org/cluster-api-provider-nico \
--set manager.image.tag=latest \
--wait
See the chart documentation for published chart locations, configuration values, and compatibility notes for upgrading an existing Helm release.
Provider release artifacts
CAPNICo publishes Cluster API provider artifacts in the same shape consumed by
clusterctl: metadata.yaml and infrastructure-components.yaml.
Generate the local artifacts with the controller image you want to publish:
CONTROLLER_IMG=ghcr.io/dsx-ai-factory/cluster-api-provider-nico/controller:v0.0.43 \
make release-manifests
This writes the clusterctl artifacts to out/. Tagging a release attaches the
same two files to the GitHub Release.
Update metadata.yaml only when introducing a new major/minor release series or
changing the Cluster API contract supported by a release series. Patch releases
within the same series should keep the existing metadata entry.
To consume a released CAPNICo provider from clusterctl, add the release asset
URL to your clusterctl configuration. Create or update the default config file:
mkdir -p ~/.config/cluster-api
$EDITOR ~/.config/cluster-api/clusterctl.yaml
If the file already exists, merge the nico entry into the existing providers
list. The version in the URL is the default used when a clusterctl command
does not specify one explicitly:
providers:
- name: nico
type: InfrastructureProvider
url: https://github.com/dsx-ai-factory/cluster-api-provider-nico/releases/download/v0.0.43/infrastructure-components.yaml
Then initialize the provider:
clusterctl init --infrastructure nico:v0.0.43
Controller scope
The CAPNICo manager accepts the standard Cluster API provider scope flags:
--namespacelimits reconciliation to Cluster API objects in one namespace. The empty default watches all namespaces, which is the mode used byclusterctlinstallations.--watch-filterlimits reconciliation to objects labeledcluster.x-k8s.io/watch-filter=<value>. The empty default reconciles all objects.
Use both flags when running multiple CAPNICo manager instances in one management cluster.
This release does not publish workload cluster templates yet. Use
clusterctl generate cluster --from <template-file-or-url> with a local
template when generating clusters.
The generated provider manifest references the controller image you pass through
CONTROLLER_IMG. Management clusters must be able to pull that image. For local
or private deployments, either grant the cluster access to that registry or
regenerate the artifacts with an image mirrored to a registry the cluster can
reach.
NICo credentials Secret
The provider supports two sources for the NICo credentials Secret. Each is
optional on its own, but at least one must be configured for the provider to
reconcile a NicoCluster. They are not mutually exclusive — both may be
present at the same time:
- A provider-level Secret in the manager's own namespace, used by every
NicoClusterthat does not set its ownspec.identityRef. - A per-cluster Secret referenced by
NicoCluster.spec.identityRef, in the same namespace as theNicoCluster. When set, it overrides the provider-level Secret for thatNicoCluster.
The provider-level Secret's name and namespace are configurable via two manager flags:
--provider-credentials-namespace(default:$POD_NAMESPACE, falling back tocapnico-systemwhen unset, e.g. undermake run)--provider-credentials-secret-name(default:nico-credentials)
The same keys are used for both the provider-level Secret or per-cluster override(s).
Required keys:
endpoint: base URL for the NICo API, for examplehttps://nico.example.comorgID: NICo org identifier used in API paths
Choose one authentication mode.
Static bearer token:
token: bearer token used for API calls
OAuth2 client credentials:
tokenURL: OAuth2 token endpointclientID: OAuth2 client IDclientSecret: OAuth2 client secretscope: optional space-separated scopes, for examplecarbide
Optional keys:
ca.crt: PEM-encoded CA bundleinsecureSkipTLSVerify:trueorfalseapiName: NICo API name, NICo allows customers to optionally specify a custom API name for their deployment.
The published NICo SDK does not include a helper for token acquisition beyond accepting a bearer token in request context, so this provider performs the OAuth2 client-credentials exchange itself and refreshes access tokens automatically.
Example using OAuth2 client credentials:
apiVersion: v1
kind: Secret
metadata:
name: nico-credentials
namespace: default
type: Opaque
stringData:
endpoint: https://nico.example.com
orgID: your-org
tokenURL: https://issuer.example.com/token
clientID: <client-id>
clientSecret: <client-secret>
scope: carbide
ca.crt: |
-----BEGIN CERTIFICATE-----
...
-----END CERTIFICATE-----
Kubeadm examples
The kubeadm examples are local clusterctl templates. They assume the
provider-level credentials Secret already exists:
kubectl create secret generic -n capnico-system nico-credentials \
--from-literal=endpoint=https://nico.example.com \
--from-literal=orgID=your-org \
--from-literal=token=<bearer-token>
# Or use OAuth2 client credentials instead of a static token:
kubectl create secret generic -n capnico-system nico-credentials \
--from-literal=endpoint=https://nico.example.com \
--from-literal=orgID=your-org \
--from-literal=tokenURL=<token-url> \
--from-literal=clientID=<client-id> \
--from-literal=clientSecret=<client-secret> \
--from-literal=scope=<scope>
The kubeadm templates patch kubelet with providerID: nico://<instance-id> by reading
the NICo metadata service at 169.254.169.254:7777. This lets Cluster API match
workload-cluster Nodes back to their Machine objects.
CAPNICo also backfills an empty Node.spec.providerID after confirming the NICo
instance identity, using the workload-cluster kubeconfig Secret. It never
overwrites a different provider ID. Clusters where an external cloud provider
owns this field can opt out by setting the
nico.nvidia.com/skip-node-provider-id-reconciliation: "true" annotation on the
Cluster API Cluster.
The NICo architecture documentation describes The Metadata Service (FMDS) as the local HTTP metadata API provided by the DPU agent: Overview and Components.
List the variables required by a template with:
clusterctl generate cluster demo \
--from examples/kubeadm/cluster.yaml \
--list-variables
External control-plane endpoint
Use examples/kubeadm/cluster.yaml when a stable Kubernetes API endpoint
already exists, for example through DNS, an external load balancer, or kube-vip
managed outside this template.
NICO_NETWORK_METHOD selects the NICo network attachment field and defaults to
vpcPrefixID, which is preferred for new clusters. Set
NICO_NETWORK_METHOD=subnetID for legacy subnet networking. NICO_NETWORK_ID
is the corresponding VPC prefix or subnet ID.
Set CONTROL_PLANE_ENDPOINT_HOST to the stable API endpoint IP or DNS name.
kubectl create namespace demo
NICO_SITE_ID=your-site \
NICO_VPC_ID=your-vpc \
NICO_CONTROL_PLANE_INSTANCE_TYPE_ID=your-control-plane-instance-type \
NICO_WORKER_INSTANCE_TYPE_ID=your-worker-instance-type \
NICO_NETWORK_METHOD=vpcPrefixID \
NICO_NETWORK_ID=your-vpc-prefix \
NICO_CONTROL_PLANE_IPXE_SCRIPT='chain https://boot.example.com/ipxe/control-plane.ipxe' \
NICO_WORKER_IPXE_SCRIPT='chain https://boot.example.com/ipxe/worker.ipxe' \
CONTROL_PLANE_ENDPOINT_HOST=10.0.0.100 \
clusterctl generate cluster demo \
--from examples/kubeadm/cluster.yaml \
--target-namespace demo \
--kubernetes-version v1.36.0 \
--control-plane-machine-count 1 \
--worker-machine-count 1 \
| kubectl apply -f -
Kube-vip convenience template
Use examples/kubeadm/cluster-kube-vip.yaml when a stable API endpoint does not
already exist and you want the template to bootstrap kube-vip as a static pod.
The network variables are the same as cluster.yaml.
Set KUBE_VIP_ADDRESS to the stable API endpoint IP. Set
CONTROL_PLANE_ENDPOINT_HOST to that IP or to a DNS name that resolves to it.
Joining nodes require this endpoint to be reachable after kube-vip starts. By
default, KUBE_VIP_BGP_PEER_AS=auto reads the peer ASN from the NICo instance
metadata service at /latest/meta-data/asn.
kubectl create namespace demo
NICO_SITE_ID=your-site \
NICO_VPC_ID=your-vpc \
NICO_CONTROL_PLANE_INSTANCE_TYPE_ID=your-control-plane-instance-type \
NICO_WORKER_INSTANCE_TYPE_ID=your-worker-instance-type \
NICO_NETWORK_METHOD=vpcPrefixID \
NICO_NETWORK_ID=your-vpc-prefix \
NICO_CONTROL_PLANE_IPXE_SCRIPT='chain https://boot.example.com/ipxe/control-plane.ipxe' \
NICO_WORKER_IPXE_SCRIPT='chain https://boot.example.com/ipxe/worker.ipxe' \
KUBE_VIP_ADDRESS=10.0.0.100 \
CONTROL_PLANE_ENDPOINT_HOST=10.0.0.100 \
clusterctl generate cluster demo \
--from examples/kubeadm/cluster-kube-vip.yaml \
--target-namespace demo \
--kubernetes-version v1.36.0 \
--control-plane-machine-count 1 \
--worker-machine-count 1 \
| kubectl apply -f -
Developer: single control-plane requested IP
examples/kubeadm/cluster-single-control-plane-static-ip.yaml exists for
developer bring-up and requested-IP testing. It skips kube-vip and requests
CONTROL_PLANE_ENDPOINT_IP directly as the control-plane NICo interface IP,
then uses the same address as the Kubernetes API server endpoint.
CONTROL_PLANE_ENDPOINT_IP must be an available IP in the VPC prefix and must
have its least-significant host bit set to 1, which is required by NICo's
VPC-prefix linknet allocation. Do not use this template for normal clusters or
HA control planes; use cluster-kube-vip.yaml instead. This is only intended
for development with minimal hardware requirements.
kubectl create namespace demo
NICO_SITE_ID=your-site \
NICO_VPC_ID=your-vpc \
NICO_CONTROL_PLANE_INSTANCE_TYPE_ID=your-control-plane-instance-type \
NICO_WORKER_INSTANCE_TYPE_ID=your-worker-instance-type \
NICO_VPC_PREFIX_ID=your-vpc-prefix \
NICO_CONTROL_PLANE_IPXE_SCRIPT='chain https://boot.example.com/ipxe/control-plane.ipxe' \
NICO_WORKER_IPXE_SCRIPT='chain https://boot.example.com/ipxe/worker.ipxe' \
CONTROL_PLANE_ENDPOINT_IP=10.0.0.11 \
clusterctl generate cluster demo \
--from examples/kubeadm/cluster-single-control-plane-static-ip.yaml \
--target-namespace demo \
--kubernetes-version v1.36.0 \
--worker-machine-count 1 \
| kubectl apply -f -
Deletion lifecycle
The supported teardown path is to delete the CAPI Cluster and wait for the
Cluster deletion to complete before deleting the namespace, if it is still
needed. This lets Cluster API delete machines before the shared infrastructure
object and gives each NicoMachine a chance to delete its backing NICo instance
while credentials are still available.
Do not use namespace deletion as the normal teardown mechanism while NICo
instances may still exist. A namespace delete can remove the per-cluster
credentials Secret before NicoMachine finalizers finish, which can leave
machines stuck in deletion and require manual NICo cleanup.
Machine Repair
CAPNICo exposes repair as an annotation-driven contract on the owning CAPI
Machine. A consumer requests that a NICo instance be flagged for repair before
deletion by setting the configured repair annotation on the Machine. CAPNICo
treats a non-empty annotation value as the repair request and forwards the
health issue to the NICo delete request as machine health context for the repair
workflow.
The value is parsed as JSON first:
{"category": "Thermal", "summary": "over temperature", "details": "optional"}
A value that does not parse as JSON is treated as a legacy plain-string summary
and assigned the category Other.
The feature can be disabled by setting the flag to an empty string.
The default annotation key is:
nico.nvidia.com/machine-health-issue
The key is configurable with a manager flag:
--repair-annotation
Machine Reboot
Set one of these annotations to a non-empty value on the owning CAPI Machine.
CAPNICo treats the value as consumer-owned metadata and does not interpret it.
Use a new value for each request with the same annotation key. Reapplying the
latest completed request's value is ignored, even after CAPNICo removes the
annotation. To replace an active request, set the same annotation to a new
non-empty value. A removal or replacement can race with dispatch if the
controller has not observed the change yet. Do not set more than one reboot
annotation at a time.
| Annotation | Behavior |
|---|---|
nico.nvidia.com/reboot |
Request NICo GracefulRestart, then use the existing hard instance reboot if no changed workload Node boot ID is observed within 30 minutes. Configure this key with --reboot-annotation. |
nico.nvidia.com/reboot-soft |
Request GracefulRestart without an automatic hard fallback. |
nico.nvidia.com/reboot-hard |
Request the existing hard instance reboot immediately. |
The graceful action uses NICo's Machine power API. That API requires a provider
organization and a credential with the PROVIDER_ADMIN role. Set
NicoCluster.spec.powerControlIdentityRef.name to a Secret in the same namespace
when the regular NICo credential does not have provider access. The Secret uses
the same keys as spec.identityRef, including its own orgID. When the power
control reference is unset, CAPNICo tries the regular credential. If NICo
rejects the graceful request, the default annotation falls back to hard reboot.
CAPNICo confirms graceful recovery when the workload Node's boot ID changes and the Node becomes Ready. It waits 30 minutes to allow the host, DPU, and network to return. If it cannot observe a changed boot ID, the default annotation uses the hard fallback. This can cause a second reboot if the graceful reboot succeeded but the Node remained unreachable. The graceful-only annotation avoids that fallback.
NicoMachine.status.reboot records the mode, phase, outcome, and timestamps.
Kubernetes events report accepted requests and fallback. CAPNICo saves the
reboot status at the end of reconciliation. If the controller stops or the
status patch fails after NICo receives a reboot request, the next reconcile
can send it again. CAPNICo removes the request annotation after it finishes
handling the request if its value still matches the accepted request. A
replacement value remains set for the next request.
Development
CAPNICo uses Kubebuilder project metadata and Makefile conventions. Start with the Makefile help output when looking for common development tasks:
make help
Common local checks:
make generate
make manifests
make test
make build
CAPNICo keeps controllers in the top-level controllers package to stay close
to CAPA and CAPG. Kubebuilder go/v4 scaffolds controllers under
internal/controller by default, so this repository includes a small external
Kubebuilder plugin that adapts generated controller files into the CAPNICo
layout.
Use the plugin Makefile when initializing the project scaffold or adding a new API/controller:
make -C hack/kubebuilder/plugins/capnico-layout/v1 init-project
make -C hack/kubebuilder/plugins/capnico-layout/v1 create-api KIND=NicoCluster
For template resources that do not need reconcilers:
make -C hack/kubebuilder/plugins/capnico-layout/v1 create-api KIND=NicoClusterTemplate CONTROLLER=false
The plugin's own README describes what it does and how to install it.
Contributing
- Start here: CONTRIBUTING.md
- Code of Conduct: CODE_OF_CONDUCT.md
- Coding agents: AGENTS.md
See Community and support above for where to ask questions and how to report a vulnerability.
License
Apache License 2.0. See LICENSE.
Directories
¶
| Path | Synopsis |
|---|---|
|
api
|
|
|
v1alpha1
Package v1alpha1 contains API Schema definitions for the Nico infrastructure provider.
|
Package v1alpha1 contains API Schema definitions for the Nico infrastructure provider. |
|
fake
command
Command fake runs the self-contained fake NICo endpoint.
|
Command fake runs the self-contained fake NICo endpoint. |
|
internal
|
|
|
fake
Package fake implements a self-contained stand-in for the NICo API.
|
Package fake implements a self-contained stand-in for the NICo API. |
|
pkg
|
|
|
test/fixture
Package fixture registers file-backed envtest cases as Ginkgo specs.
|
Package fixture registers file-backed envtest cases as Ginkgo specs. |
|
test
|
|
|
tools
|
|