envd-server-pod-webhook

README

envd-server-pod-webhook

Installation

This project can fully run locally and includes automation to deploy a local Kubernetes cluster (using Kind).

Requirements

• Docker
• kubectl
• Kind
• Go >=1.16 (optional)

Usage

Create Cluster

First, we need to create a Kubernetes cluster:

❯ make cluster

🔧 Creating Kubernetes cluster...
kind create cluster --config dev/manifests/kind/kind.cluster.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.21.1) 🖼
 ✓ Preparing nodes 📦
 ✓ Writing configuration 📜
 ✓ Starting control-plane 🕹️
 ✓ Installing CNI 🔌
 ✓ Installing StorageClass 💾
Set kubectl context to "kind-kind"
You can now use your cluster with:

kubectl cluster-info --context kind-kind

Have a nice day! 👋

Make sure that the Kubernetes node is ready:

❯ kubectl get nodes
NAME                 STATUS   ROLES                  AGE     VERSION
kind-control-plane   Ready    control-plane,master   3m25s   v1.21.1

And that system pods are running happily:

❯ kubectl -n kube-system get pods
NAME                                         READY   STATUS    RESTARTS   AGE
coredns-558bd4d5db-thwvj                     1/1     Running   0          3m39s
coredns-558bd4d5db-w85ks                     1/1     Running   0          3m39s
etcd-kind-control-plane                      1/1     Running   0          3m56s
kindnet-84slq                                1/1     Running   0          3m40s
kube-apiserver-kind-control-plane            1/1     Running   0          3m54s
kube-controller-manager-kind-control-plane   1/1     Running   0          3m56s
kube-proxy-4h6sj                             1/1     Running   0          3m40s
kube-scheduler-kind-control-plane            1/1     Running   0          3m54s

Deploy Admission Webhook

To configure the cluster to use the admission webhook and to deploy said webhook, simply run:

❯ make deploy

📦 Building envd-server-pod-webhook Docker image...
docker build -t envd-server-pod-webhook:latest .
[+] Building 14.3s (13/13) FINISHED
...

📦 Pushing admission-webhook image into Kind's Docker daemon...
kind load docker-image envd-server-pod-webhook:latest
Image: "envd-server-pod-webhook:latest" with ID "sha256:46b8603bcc11a8fa1825190d3ed99c099096395b22a709e13ec6e7ae2f54014d" not yet present on node "kind-control-plane", loading...

⚙️  Applying cluster config...
kubectl apply -f dev/manifests/cluster-config/
namespace/apps created
mutatingwebhookconfiguration.admissionregistration.k8s.io/envd-server.tensorchord.ai created
validatingwebhookconfiguration.admissionregistration.k8s.io/envd-server.tensorchord.ai created

🚀 Deploying envd-server-pod-webhook...
kubectl apply -f dev/manifests/webhook/
deployment.apps/envd-server-pod-webhook created
service/envd-server-pod-webhook created
secret/envd-server-pod-webhook-tls created

Then, make sure the admission webhook pod is running (in the default namespace):

❯ kubectl get pods
NAME                                        READY   STATUS    RESTARTS   AGE
envd-server-pod-webhook-77444566b7-wzwmx   1/1     Running   0          2m21s

You can stream logs from it:

❯ make logs

🔍 Streaming envd-server-pod-webhook logs...
kubectl logs -l app=envd-server-pod-webhook -f
time="2021-09-03T04:59:10Z" level=info msg="Listening on port 443..."
time="2021-09-03T05:02:21Z" level=debug msg=healthy uri=/health

And hit it's health endpoint from your local machine:

❯ curl -k https://localhost:8443/health
OK

Deploying pods

Deploy a valid test pod that gets succesfully created:

❯ make pod

🚀 Deploying test pod...
kubectl apply -f dev/manifests/pods/lifespan-seven.pod.yaml
pod/lifespan-seven created

You should see in the admission webhook logs that the pod got mutated and validated.

Deploy a non valid pod that gets rejected:

❯ make bad-pod

🚀 Deploying "bad" pod...
kubectl apply -f dev/manifests/pods/bad-name.pod.yaml
Error from server: error when creating "dev/manifests/pods/bad-name.pod.yaml": admission webhook "envd-server.tensorchord.ai" denied the request: pod name contains "offensive"

You should see in the admission webhook logs that the pod validation failed. It's possible you will also see that the pod was mutated, as webhook configurations are not ordered.

Testing

Unit tests can be run with the following command:

$ make test
go test ./...
?   	github.com/tensorchord/envd-server-pod-webhook	[no test files]
ok  	github.com/tensorchord/envd-server-pod-webhook/pkg/admission	0.611s
ok  	github.com/tensorchord/envd-server-pod-webhook/pkg/mutation	1.064s
ok  	github.com/tensorchord/envd-server-pod-webhook/pkg/validation	0.749s

Admission Logic

A set of validations and mutations are implemented in an extensible framework. Those happen on the fly when a pod is deployed and no further resources are tracked and updated (ie. no controller logic).

Validating Webhooks

Implemented

• name validation: validates that a pod name doesn't contain any offensive string

How to add a new pod validation

To add a new pod mutation, create a file pkg/validation/MUTATION_NAME.go, then create a new struct implementing the validation.podValidator interface.

Mutating Webhooks

Implemented

• inject env: inject environment variables into the pod such as KUBE: true
• minimum pod lifespan: inject a set of tolerations used to match pods to nodes of a certain age, the tolerations injected are controlled via the acme.com/lifespan-requested pod label.

How to add a new pod mutation

To add a new pod mutation, create a file pkg/mutation/MUTATION_NAME.go, then create a new struct implementing the mutation.podMutator interface.

Acknowledgements

• slackhq/simple-kubernetes-webhook