admissioncontroller

README

Kubernetes Nvidia Runtime Admission Controller

Based on the boilerplate admission controller by douglasmakey.

This controller has one simple purpose: set runtimeClassName: nvidia for any pod that is scheduled with Nvidia GPU resources.

This is especially useful for when systems like Kubeflow is deployed to heterogenous clusters where the nvidia runtime cannot be the default across all nodes, or for installations like k3s where nvidia runtime is registered as a separate runtime class.

Deployment

The project currently supports deployment to existing kubeflow projects that use cert-manager, by using kustomize.

kustomize build kustomize/overlays/cert-manager | kubectl apply -f is the easiest way to install the admission controller to the default namespace.

Verify it works

In a cluster that contains a runtime class for nvidia (such as k3s installed on systems with GPU, or other custom installations, probably not using gpu-operator that already does this), deploy a pod asking for nvidia gpus:

$ cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: gpu-pod
spec:
  restartPolicy: Never
  containers:
    - name: cuda-container
      image: nvcr.io/nvidia/k8s/cuda-sample:vectoradd-cuda10.2
      resources:
        limits:
          nvidia.com/gpu: 1 # requesting 1 GPU
  tolerations:
  - key: nvidia.com/gpu
    operator: Exists
    effect: NoSchedule
EOF

As a fun side-effect, by letting this webhook set the runtime without the user specifying it, the warning in https://github.com/NVIDIA/k8s-device-plugin regarding all GPUs being exposed if you don't explicitly request resources will not be a problem as containers won't even use the nvidia runtime unless they do request resources.

Documentation

Index

• type AdmitFunc
• type Hook
  • func (h *Hook) Execute(r *admission.AdmissionRequest) (*Result, error)
• type PatchOperation
  • func AddPatchOperation(path string, value interface{}) PatchOperation
  • func CopyPatchOperation(from, path string) PatchOperation
  • func MovePatchOperation(from, path string) PatchOperation
  • func RemovePatchOperation(path string) PatchOperation
  • func ReplacePatchOperation(path string, value interface{}) PatchOperation
• type Result

Types

type AdmitFunc

type AdmitFunc func(request *admission.AdmissionRequest) (*Result, error)

AdmitFunc defines how to process an admission request

type Hook

type Hook struct {
	Create  AdmitFunc
	Delete  AdmitFunc
	Update  AdmitFunc
	Connect AdmitFunc
}

Hook represents the set of functions for each operation in an admission webhook.

func (*Hook) Execute

func (h *Hook) Execute(r *admission.AdmissionRequest) (*Result, error)

Execute evaluates the request and try to execute the function for operation specified in the request.

type PatchOperation

type PatchOperation struct {
	Op    string      `json:"op"`
	Path  string      `json:"path"`
	From  string      `json:"from"`
	Value interface{} `json:"value,omitempty"`
}

PatchOperation is an operation of a JSON patch https://tools.ietf.org/html/rfc6902.

func AddPatchOperation

func AddPatchOperation(path string, value interface{}) PatchOperation

AddPatchOperation returns an add JSON patch operation.

func CopyPatchOperation

func CopyPatchOperation(from, path string) PatchOperation

CopyPatchOperation returns a copy JSON patch operation.

func MovePatchOperation

func MovePatchOperation(from, path string) PatchOperation

MovePatchOperation returns a move JSON patch operation.

func RemovePatchOperation

func RemovePatchOperation(path string) PatchOperation

RemovePatchOperation returns a remove JSON patch operation.

func ReplacePatchOperation

func ReplacePatchOperation(path string, value interface{}) PatchOperation

ReplacePatchOperation returns a replace JSON patch operation.

type Result

type Result struct {
	Allowed  bool
	Msg      string
	PatchOps []PatchOperation
}

Result contains the result of an admission request