pupernetes

README

pupernetes - p8s

Run a managed Kubernetes setup.

This project purpose is to provide a simple Kubernetes setup to validate any software on top of it.

You can use it to validate a software dependence on Kubernetes itself or just to run some classic app workflows with argo

Our main use case is the end to end testing pipeline of the datadog-agent

Provides:

• etcd v3
• kubectl
• kubelet
• kube-apiserver
• kube-scheduler
• kube-controller-manager
• kube-proxy
• coredns

The default setup is secured with:

• Valid x509 certificates provided by an embedded vault PKI
  • Able to use the Kubernetes CSR and the service account root-ca
• HTTPS webhook to provide token lookups for the kubelet API
• RBAC

Table of Contents

• Requirements
  • Runtime
  • Development
    • Create Ubuntu VM
    • Install Docker
• Build it
• Run it
• Use it
  • Command line
  • Quick run
  • Quick systemd-run
  • Systemd as job type
• Current limitations

Requirements

Runtime

Executables in PATH:

• tar
• unzip
• systemctl
• systemd-resolve (or a non-systemd managed /etc/resolv.conf)
• openssl
• mount

Any implicit requirements for the kubelet like the container runtime and more

A systemd environment.

Development

Setup a linux environment for running pupernetes. This is only a suggested environment for running pupernetes. You could also create a VM using Vagrant (not yet documented here).

curl -LOf https://github.com/DataDog/pupernetes/releases/download/v${VERSION}/pupernetes
chmod +x ./pupernetes

Ubuntu VM

pupernetes must be run on linux (or linux VM).

Example: Download the latest version of Ubuntu Desktop and create the Ubuntu VM with your preferred virtualization software.

Install Docker

Follow the instructions here to install docker.

Note:

If you are seeing the following error after running sudo apt-get install docker-ce to install docker-ce.

E: Invalid operation docker-ce

Try running the following command to setup the stable repository that instead specifies an older Ubuntu distribution like xenial instead of using lsb_release -cs (using bionic doesn't seem to always works).

$ sudo add-apt-repository \
  "deb [arch=amd64] https://download.docker.com/linux/ubuntu \
  xenial \
 stable"

Now try running $ sudo apt-get install docker-ce again.

To manage docker as a non-root user (so you don't have to keep using sudo) follow the instructions here. You must log out and log back in (or just restart your VM) so that your group membership is re-evaluated

Build it

• go
• make

Run it

sudo ./pupernetes daemon run sandbox/

Use it

Note:

kubectl is automatically installed by pupernetes.

You may need to run the following command to add kubectl to the $PATH:

sudo ./pupernetes run sandbox/ --kubectl-link /usr/local/bin/kubectl

kubectl get svc,ds,deploy,job,po --all-namespaces

NAMESPACE     NAME         TYPE        CLUSTER-IP      EXTERNAL-IP   PORT(S)         AGE
default       kubernetes   ClusterIP   192.168.254.1   <none>        443/TCP         3m
kube-system   coredns      ClusterIP   192.168.254.2   <none>        53/UDP,53/TCP   3m

NAMESPACE     NAME             DESIRED   CURRENT   READY     UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-system   kube-proxy       1         1         1         1            1           <none>          3m
kube-system   kube-scheduler   1         1         1         1            1           <none>          3m

NAMESPACE     NAME      DESIRED   CURRENT   UP-TO-DATE   AVAILABLE   AGE
kube-system   coredns   1         1         1            1           3m

NAMESPACE     NAME                       READY     STATUS    RESTARTS   AGE
kube-system   coredns-747dbcf5df-p2lhq   1/1       Running   0          3m
kube-system   kube-controller-manager    1/1       Running   0          3m
kube-system   kube-proxy-wggdn           1/1       Running   0          3m
kube-system   kube-scheduler-92zrj       1/1       Running   0          3m

Hyperkube version:

Example: --hyperkube-version=1.9.3

• 1.11
• 1.10
• 1.9
• 1.8
• 1.7
• 1.6
• 1.5
• 1.4
• 1.3

Command line

The full documentation is available here.

Quick run

make
sudo ./pupernetes daemon run sandbox/

Graceful stop it with:

• SIGINT
• SIGTERM
• --timeout
• curl -XPOST 127.0.0.1:8989/stop

Quick systemd-run

sudo systemd-run ./pupernetes daemon run ${PWD}/sandbox

Graceful stop it with:

• systemctl stop run-r${UNIT_ID}.service
• --timeout
• curl -XPOST 127.0.0.1:8989/stop

Find any systemd-run unit with:

sudo systemctl list-units run-r*.service

The DESCRIPTION field should match the initial {COMMAND} [ARGS...]

Systemd as job type

It's possible to run pupernetes as a systemd service directly with the command line. In this case, pupernetes asks to be started with the given arguments. See more info about it in the run command.

Graceful stop it with:

• systemctl stop pupernetes.service
• --timeout
• curl -XPOST 127.0.0.1:8989/stop

Current limitations

• Container runtime
  • You need docker already up and running
  • You cannot use cri-containerd / crio without changing manually the systemd unit /run/systemd/system/p8s-kubelet.service
• Systemd
  • Currently working with systemd only
  • Could be containerized with extensive mounts
• Networking
  • The CNI bridge cannot be used yet
  • Kubernetes cluster IP range is statically set
• Secrets
  • IP SAN
    • Statically configured with the given Kubernetes cluster IP range
• Versions
  • You just can minimally change the version of the downloaded binaries in the state directory during the run phase but the compatibility isn't granted