extendeddaemonset

README

ExtendedDaemonSet

!!! Alpha version !!!

ExtendedDaemonSet aims to provide a new implementation of the Kubernetes DaemonSet resource with key features:

• Canary Deployment: Deploy a new DaemonSet version with only a few nodes.
• Custom Rolling Update: Improve the default rolling update logic available in Kubernetes batch/v1 Daemonset.

How to use it

Deployment

To use the ExtendedDaemonSet controller in your Kubernetes cluster, deploy its resources definitions in the /deploy folder.

First, deploy the Custom Resources Definitions:

$ kubectl apply -f ./deploy/crds

If you want to deploy a specific controller version, the version can be set in the deploy/operator.yaml file. By default, the latest docker image tag is used.

$ kubectl apply -f ./deploy/

By default, the controller only watches the ExtendedDaemonSet resources that are present in its own namespace. If you want to deploy the controller cluster wide, modify the deploy/operator.yaml to remove or set an empty value for the WATCH_NAMESPACE environment variable:

            env:  
            - name: WATCH_NAMESPACE
              value: ""

Demo application

If you want to test and compare the advantages of the ExtendedDaemonSet over the the standard DaemonSet, you can use the demo application available in the /example folder. Follow the below scenario:

First, you need a Kubernetes cluster with several nodes; we recommend using three nodes. If you want, you can use kind.sigs.k8s.io to create a three node cluster with the following command: kind create cluster --config examples/kind-cluster-configuration.yaml.

This creates a three node cluster with one master and two worker nodes:

$ kind create cluster --config examples/kind-cluster-configuration.yaml
Creating cluster "kind" ...
 ✓ Ensuring node image (kindest/node:v1.15.3) 🖼
 ✓ Preparing nodes 📦📦📦 
 ✓ Creating kubeadm config 📜 
 ✓ Starting control-plane 🕹️ 
 ✓ Installing CNI 🔌 
 ✓ Installing StorageClass 💾 
 ✓ Joining worker nodes 🚜 
Cluster creation complete. You can now use the cluster with:

$ export KUBECONFIG="$(kind get kubeconfig-path --name="kind")"

ExtendedDaemonSet controller deployment

# deploy the controller needed crds
$ kubectl apply -f ./deploy/crds
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsets.datadoghq.com created
customresourcedefinition.apiextensions.k8s.io/extendeddaemonsetreplicasets.datadoghq.com created

# deploy the controller pod
$ kubectl apply -f ./deploy/
clusterrole.rbac.authorization.k8s.io/extendeddaemonset created
clusterrolebinding.rbac.authorization.k8s.io/extendeddaemonset created
deployment.apps/extendeddaemonset created
role.rbac.authorization.k8s.io/extendeddaemonset created
rolebinding.rbac.authorization.k8s.io/extendeddaemonset created
serviceaccount/extendeddaemonset created

# you should see the extendeddaemonset controller pod running
$ kubectl get pods
NAME                                 READY   STATUS    RESTARTS   AGE
extendeddaemonset-855cd7c679-gpmql   1/1     Running   0          2m11s

foo ExtendedDaemonSet deployment

Create the foo app with the ExtendedDaemonSet. For demo purposes, we'll use the k8s.gcr.io/pause Docker image, which is only awaiting a terminating signal. You can look at the foo application definition in the file examples/foo-eds_v1.yaml.

$ kubectl apply -f examples/foo-eds_v1.yaml
extendeddaemonset.datadoghq.com/foo created

You can see the state of the ExtendedDaemonSet foo with:

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-8z7lr               44s

# Also the `extendeddaemonsetreplicaset` resource generated by the controller from the `foo` EDS instance:
$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   3         3         3       3                           61s

foo ExtendedDaemonSet deployment update with canary strategy

Now we can try to update the ExtendedDaemonSet foo. The only difference between the two versions is the Docker image used in the pod template.

$ diff examples/foo-eds_v1.yaml examples/foo-eds_v2.yaml
17c17
<         image: k8s.gcr.io/pause:3.0
---
>         image: k8s.gcr.io/pause:3.1

$ kubectl apply -f examples/foo-eds_v2.yaml
extendeddaemonset.datadoghq.com/foo configured

As you can see with the following command, a canary ReplicaSet is now configured for the foo ExtendedDaemonSet. Additionally, a new ExtendedReplicaSet has been created to handle the new foo pod template version.

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS   ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Canary   foo-8z7lr   foo-xdj4b   85s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-8z7lr   active   2         2         2       2                           2m
foo-xdj4b   canary   1         1         1       1                           40s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME                                 READY   STATUS    RESTARTS   AGE
foo-8z7lr-bp9w8                      1/1     Running   0          108s
foo-8z7lr-jlvrq                      1/1     Running   0          88s
foo-xdj4b-zvss2                      1/1     Running   0          8s

Only one pod is running with the ExtendedReplicaSet foo-xdj4b pod template version. This corresponds to the setting spec.canary.replicas in the ExtendedDaemonSet foo.

Rolling update after the canary deployment validation period ended

After 5 minutes, which corresponds to spec.canary.duration, the controller will set as valid and activate the foo-xdj4b ExtendedReplicaSet. It will trigger the full foo-xdj4b ExtendedReplicaSet deployment.

$ kubectl get eds
NAME   DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   STATUS    ACTIVE RS   CANARY RS   AGE
foo    3         3         3       3            3           Running   foo-xdj4b               9m21s

$ kubectl get ers
NAME        STATUS   DESIRED   CURRENT   READY   AVAILABLE   NODE SELECTOR   AGE
foo-xdj4b   active   3         3         3       3                           8m21s

$ kubectl get pod -l extendeddaemonset.datadoghq.com/name=foo
NAME              READY   STATUS    RESTARTS   AGE
foo-xdj4b-hh6d8   1/1     Running   0          5m11s
foo-xdj4b-rgtk9   1/1     Running   0          5m31s
foo-xdj4b-zvss2   1/1     Running   0          10m

Overwrite container's Pod resources for a specific Node

The ExtendedDaemonset controller allows to overwrite the container's pod managed by an ExtendedDaemonset for a specific Node, thanks to an annotation that you can set on the Node: resources.extendeddaemonset.datadoghq.com/<eds-namespace>.<eds-name>.<container-name>={...}. the value corresponds to the Resources definition in JSON.

For example, for the ExtendedDaemonset named foo in the bar namespace. The container myapp resources specification can be overwriten by adding the following annotation on a Node:

$ kubectl annotate node <node-name> `resources.extendeddaemonset.datadoghq.com/bar.foo.myapp={"requests":{"cpu":"2.0","memory":"2G"}}`
node/<node-name> annotated

Overwrite container's Pod resources for a set of Nodes with ExtendedDaemonsetSettings

In some cases (for example with different nodes type), it can be useful to have different resource configurations for a Daemonset to handle the Node's workload specificity.

To do so you can create an instance of ExtendedDaemonsetSetting resource that aims to overwrite the resources definition of the container(s) present in ExtendedDaemonset Pods.

the information needed is:

• spec.nodeSelector: a NodeLabels selector that matches with the nodes where it must trigger the usage of this resource.
• spec.reference: contains enough information to let you identify the referred resource.
• spec.containers: contains a list of Container spec overwrites.

apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonsetSetting
metadata:
  name: foo-xxl-node
spec:
  nodeSelector:
    matchLabels:
      node-type: xxl
  reference:
    kind: ExtendedDaemonset
    name: foo
  containers:
   - name: daemon
    resources:
      requests:
        cpu: "0.5"
        memory: "300m"

Remove a pod on a given node using nodeAffinity

In some cases, it could be useful to remove a daemon pod on a given node. This can be done using the podTemplate.spec.affinity.nodeAffinity field.

First set a new requiredDuringSchedulingIgnoredDuringExecution.nodeSelectorTerms field

apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonSet
metadata:
  name: foo
spec:
  template:
    spec:
      //...
      affinity:
        nodeAffinity:
          requiredDuringSchedulingIgnoredDuringExecution:
            nodeSelectorTerms:
            - matchExpressions:
              - key: extendeddaemonset.datadoghq.com/exclude
                operator: NotIn
                values:
                - foo

Then add the label extendeddaemonset.datadoghq.com/exclude=foo to the node in question

kubectl label nodes <your-node-name> extendeddaemonset.datadoghq.com/exclude=foo

Kubectl plugin

To build the the kubectl ExtendedDaemonSet plugin, you can run the command: make build-plugin. This will create the kubectl-eds Go binary, corresponding to your local OS and architecture. Then, add or move this binary to the PATH and run the command kubectl eds:

$ kubectl eds
Usage:
  ExtendedDaemonset [command]

Available Commands:
  get         get ExtendedDaemonSet deployment(s)
  get-ers     get-ers ExtendedDaemonSetReplicaset deployment(s)
  help        Help about any command
  validate    validate canary replicaset

How to migrate from a DaemonSet

If you already have an application running in your cluster with a DaemonSet, it is possible to migrate to an ExtendedDaemonSet with a smooth migration path.

• Update your DaemonSet specification to set a toleration that does not correspond to your node's taints. As a result, the DaemonSet pods that are already running will not be deleted, and the DaemonSet controller will not take any new actions on it.

• In the ExtendedDaemonSet definition, add a specific annotation to inform the extendeddaemonset-controller which DaemonSet needs to be migrated. The controller will recognize the pods from the "old" DaemonSet as a previous version and will do a proper rolling update.

apiVersion: datadoghq.com/v1alpha1
kind: ExtendedDaemonSet
metadata:
  name: foo
  annotations:
    extendeddaemonset.datadoghq.com/old-daemonset: foo
spec:
    # ...

Developers section

How to build it

This project uses go module. Ensure you have it activated: export GO111MODULE=on.

Run make install-tools to install mandatory tooling, like the operator-fwk command line and the golangci linter.

$ make build
CGO_ENABLED=0 go build -i -installsuffix cgo -ldflags '-w' -o controller ./cmd/manager/main.go

How to test it

Unit tests

$ make test
./go.test.sh
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonset   1.107s  coverage: 77.0% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset 1.098s  coverage: 63.9% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy        1.036s  coverage: 5.3% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/extendeddaemonsetreplicaset/strategy/limits 1.016s  coverage: 83.3% of statements
ok      github.com/datadog/extendeddaemonset/pkg/controller/utils       1.015s  coverage: 100.0% of statements

End to end tests

For end to end tests, consult /test/README.md

Linter validation

To use the linter, run:

$ make validate
./bin/golangci-lint run ./...

How to release it

• Checkout the repository on the correct branch (master)
• Run locally the command make VERSION=x.y.z pre-release
• Commit all the changes generated from the previous command:

  $ git add .
  $ git commit -s -m "release vX.Y.Z"
  

• Add the release tag:

  $ git tag vX.Y.Z
  

• Push it:

  $ git push origin vX.Y.Z