README
¶
scuttle 
scuttle handles SIGTERM or spot termination notices by optionally draining or deleting the current Kubernetes node. It's best run as a systemd unit designed to gracefully stop on shutdown (shown below).
- Uncordon the node on start (reboot case)
- Handle SIGTERM (unit stop or shutdown)
- Monitor instance metadata for termination notices (AWS, Azure planned)
- Drain and/or delete (de-register) a Kubernetes node
- Evict Pods left by Kubelet's GracefulNodeShutdown
scuttle compliments Kubelet's GracefulNodeShutdown feature, which only handles a part of gracefully stopping the Kubelet. See the blog post to learn more.
Usage
scuttle must be run in a systemd unit that is designed to stop before shutdown. Systemd shutdown can be subtle. Read systemd Shutdown Units and Kubelet Graceful Shutdown for background.
In effect, scuttle is just a Go implementation of the bash scripts shown in the posts.
[Unit]
Description=Scuttle Kubelet before Shutdown
After=multi-user.target
[Service]
Type=simple
ExecStartPre=-/usr/bin/podman rm scuttle
ExecStart=/usr/bin/podman run \
--name scuttle \
--network host \
--log-driver=k8s-file \
--env KUBECONFIG=/var/lib/kubelet/kubeconfig \
-v /var/lib/kubelet:/var/lib/kubelet:ro,z \
--stop-timeout=60 \
quay.io/poseidon/scuttle:v0.1.0 \
-platform=aws
ExecStop=/usr/bin/podman stop scuttle
TimeoutStopSec=180
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
Users of Fedora CoreOS or Flatcar Linux should use the Butane Config:
variant: fcos
version: 1.4.0
systemd:
units:
- name: scuttle.service
contents: |
[Unit]
Description=Scuttle Kubelet before Shutdown
After=multi-user.target
[Service]
Type=simple
ExecStartPre=-/usr/bin/podman rm scuttle
ExecStart=/usr/bin/podman run \
--name scuttle \
--network host \
--log-driver=k8s-file \
--env KUBECONFIG=/var/lib/kubelet/kubeconfig \
-v /var/lib/kubelet:/var/lib/kubelet:ro,z \
--stop-timeout=60 \
quay.io/poseidon/scuttle:261fc0f \
-platform=aws
ExecStop=/usr/bin/podman stop scuttle
TimeoutStopSec=180
SuccessExitStatus=143
[Install]
WantedBy=multi-user.target
- name: scuttle.path
enabled: true
contents: |
[Unit]
Description=Watch for Kubelet kubeconfig
[Path]
PathExists=/var/lib/kubelet/kubeconfig
[Install]
WantedBy=multi-user.target
Spot Termination Notices
AWS and Azure provide warnings via instance metadata (2 min) before spot terminations. scuttle can monitor platform specific instance metadata endpoints to trigger drain or delete actions before shutdown.
Directories