pod-restarter: restarts Pods in a failed state using client-go kubernetes packages
Performs the following steps:
- Looks for latest Pod Events that matches an Event Reason and Message
- If there are matching Pods, these Pods will go through a sequence of steps before they get deleted:
- verify Pod exists
- verify Pod has owner/controller
- verify Pod has not been scheduled to be deleted
- verify Pod is in a Failing State (Pending/Failed or Running with failing containers)
- If all above checks pass, Pod will be deleted
These steps are repeated in a loop on a polling interval basis.
Configuring pod-restarter
pod-restarter is configurable through cli parameters.
--polling-interval
- Delete Pods that have matching Events with default Reason and Message every poll interval (seconds).
- Default value: 30 (seconds)
./pod-restarter --polling-interval 10
--dry-run
- Logs pod-restarter actions but don't actually delete any pods.
- Default value: disabled
./pod-restarter --dry-run
--reason
and --error-message
- These parameters work together because every Event has a Reason and a related Message.
- These parameters are used for identifying failing Pods that match Event Reason and Message.
- Default values:
- "FailedCreatePodSandBox" (Event Reason)
- "container veth name provided (eth0) already exists" (Event Message)
# delete Pods that have Events with default Reason "FailedCreatePodSandBox" and default Message "container veth name provided (eth0) already exists"
./pod-restarter
# delete Pods that have Events with Reason "BackOff" and Message "Back-off pulling image"
./pod-restarter --reason "BackOff" --error-message "Back-off pulling image"
--namespace
- The kubernetes namespavce where pod-restarter should look for Failing Pods.
- Default value: "" (look for all namespaces)
# delete Pods in namespace default
./pod-restarter --namespace default
--kubeconfig
- When run locally (outside of cluster), specifies the kubeconfig config.
- Default value: ~/.kube/config
Run and test on local machine/laptop
# install dependencies
go mod init
go mod tidy
# compile source code into executable binary
go build -o pod-restarter
# run in dry mode
./pod-restarter --dry-run --polling-interval 10
# run unit tests
go test -v ./...
Deploy to k8s with helm
# build container image
task build
# deploy helm chart
task install
# verify helm release
helm list -A
# uninstall helm chart
task uninstall
Simulate Failing Pods and test
# generate failing pods due to pulling wrong images
cd infra/tests
bash generate_pending_pods.sh
# check app logs
kubectl logs -l app=pod-restarter -f