pupernetes - p8s
Run a managed Kubernetes setup.
This project purpose is to provide a simple Kubernetes setup to validate any software on top of it.
You can use it to validate a software dependence on Kubernetes itself or just to run some classic app workflows with argo
Our main use case is the end to end testing pipeline of the datadog-agent
Provides:
- etcd v3
- kubectl
- kubelet
- kube-apiserver
- kube-scheduler
- kube-controller-manager
- kube-proxy
- coredns
The default setup is secured with:
- Valid x509 certificates provided by an embedded vault PKI
- Able to use the Kubernetes CSR and the service account root-ca
- HTTPS webhook to provide token lookups for the kubelet API
- RBAC
Table of Contents
Requirements
Runtime
Executables in PATH:
- tar
- unzip
- systemctl
- systemd-resolve (or a non-systemd managed
/etc/resolv.conf
)
- openssl
- mount
Any implicit requirements for the kubelet like the container runtime and more
A systemd environment.
Development
Setup a linux environment for running pupernetes
. This is only a suggested environment for running pupernetes. You could also create a VM using Vagrant (not yet documented here).
curl -LOf https://github.com/DataDog/pupernetes/releases/download/v${VERSION}/pupernetes
chmod +x ./pupernetes
Ubuntu VM
pupernetes
must be run on linux (or linux VM).
Example:
Download the latest version of Ubuntu Desktop and create the Ubuntu VM with your preferred virtualization software.
Install Docker
Follow the instructions here to install docker.
Note:
If you are seeing the following error after running sudo apt-get install docker-ce
to install docker-ce
.
E: Invalid operation docker-ce
Try running the following command to setup the stable repository that instead specifies an older Ubuntu distribution like xenial
instead of using lsb_release -cs
(using bionic
doesn't seem to always works).
$ sudo add-apt-repository \
"deb [arch=amd64] https://download.docker.com/linux/ubuntu \
xenial \
stable"
Now try running $ sudo apt-get install docker-ce
again.
To manage docker as a non-root user (so you don't have to keep using sudo
) follow the instructions here. You must log out and log back in (or just restart your VM) so that your group membership is re-evaluated
Build it
Run it
sudo ./pupernetes daemon run sandbox/
Use it
Note:
kubectl
is automatically installed by pupernetes
.
You may need to run the following command to add kubectl
to the $PATH
:
sudo ./pupernetes run sandbox/ --kubectl-link /usr/local/bin/kubectl
kubectl get svc,ds,deploy,job,po --all-namespaces
NAMESPACE NAME TYPE CLUSTER-IP EXTERNAL-IP PORT(S) AGE
default kubernetes ClusterIP 192.168.254.1 <none> 443/TCP 3m
kube-system coredns ClusterIP 192.168.254.2 <none> 53/UDP,53/TCP 3m
NAMESPACE NAME DESIRED CURRENT READY UP-TO-DATE AVAILABLE NODE SELECTOR AGE
kube-system kube-proxy 1 1 1 1 1 <none> 3m
kube-system kube-scheduler 1 1 1 1 1 <none> 3m
NAMESPACE NAME DESIRED CURRENT UP-TO-DATE AVAILABLE AGE
kube-system coredns 1 1 1 1 3m
NAMESPACE NAME READY STATUS RESTARTS AGE
kube-system coredns-747dbcf5df-p2lhq 1/1 Running 0 3m
kube-system kube-controller-manager 1/1 Running 0 3m
kube-system kube-proxy-wggdn 1/1 Running 0 3m
kube-system kube-scheduler-92zrj 1/1 Running 0 3m
Hyperkube version:
Example: --hyperkube-version=1.9.3
- 1.11
- 1.10
- 1.9
- 1.8
- 1.7
- 1.6
- 1.5
- 1.4
- 1.3
Command line
The full documentation is available here.
Quick run
make
sudo ./pupernetes daemon run sandbox/
Graceful stop it with:
- SIGINT
- SIGTERM
--timeout
curl -XPOST 127.0.0.1:8989/stop
Quick systemd-run
sudo systemd-run ./pupernetes daemon run ${PWD}/sandbox
Graceful stop it with:
systemctl stop run-r${UNIT_ID}.service
--timeout
curl -XPOST 127.0.0.1:8989/stop
Find any systemd-run unit with:
sudo systemctl list-units run-r*.service
The DESCRIPTION
field should match the initial {COMMAND} [ARGS...]
Systemd as job type
It's possible to run pupernetes as a systemd service directly with the command line.
In this case, pupernetes asks to be started with the given arguments.
See more info about it in the run command.
Graceful stop it with:
systemctl stop pupernetes.service
--timeout
curl -XPOST 127.0.0.1:8989/stop
Current limitations
- Container runtime
- You need docker already up and running
- You cannot use cri-containerd / crio without changing manually the systemd unit
/run/systemd/system/p8s-kubelet.service
- Systemd
- Currently working with systemd only
- Could be containerized with extensive mounts
- Networking
- The CNI bridge cannot be used yet
- Kubernetes cluster IP range is statically set
- Secrets
- IP SAN
- Statically configured with the given Kubernetes cluster IP range
- Versions
- You just can minimally change the version of the downloaded binaries in the state directory during the
run
phase but the compatibility isn't granted