README
ΒΆ
K8s Rightsizer
A robust Kubernetes automation tool designed to apply resource recommendations (CPU/Memory) to Deployments and StatefulSets with an integrated automatic rollback mechanism.
The tool reads a list of recommendations from an Excel file, applies them, and monitors the rollout. If a Pod fails to start (OOMKilled, CrashLoopBackOff, Unschedulable, etc.), it immediately restores the previous stable configuration.
π Key Features
- Bulk Updates: Process multiple resource changes via recommendation file (supported format .xlsx, .xsl).
- Safety First: Automatic rollback if the new resources cause deployment failures.
- Smart Monitoring: Detects
OOMKilled,CrashLoopBackOff, andInsufficient Resourcesin real-time. - Cross-Controller Support: Works seamlessly with both Deployments and StatefulSets.
- Helm Powered: Easy distribution and configuration for Local and Remote environments.
π Prerequisites
- Kubernetes Cluster (v1.34+)
- Helm (v4.1+)
- Go (v1.25+) - Only for local development
- Make
- Podman or Docker
π» Local Environment (Minikube)
To test the Rightsizer engine locally, you need to sync your container image and recommendation data with a Minikube node. We provide an automated script to spin up a pre-configured environment.
1. Setup the local environment
The setup script initializes a single-node Minikube cluster using the Podman driver, enables necessary addons (YAKD Dashboard, Metrics Server), and mounts your local data folder. This script will automatically detect the driver to set up the cluster (docker, podman) but you can override this selection by setting the variable DRIVER.
# Navigate to the test directory
cd ./test-env/local
# Run the setup script by passing the local folder containing your recommendation data
# Usage: ./setup-rightsizer-env.sh <absolute_or_relative_path>
./setup-rightsizer-env.sh ~/my-project-data
# Force using podman driver
# DRIVER=podman ./setup-rightsizer-env.sh ~/my-project-data
Note: The script mounts your local folder to /mnt/data inside the Minikube node. Ensure your Kubernetes PersistentVolume manifests point to this path.
2. Build and Load the Image
Minikube needs the image in its internal registry. When using Podman, the most reliable way is via a tarball:
# 1. Build the image with a local tag
make image-build REGISTRY_USER=localhost VERSION=local
# 2. Load image into Minikube and deploy via helm
make deploy
3. Cleanup
make undeploy
βοΈ Remote Environment
1. Build and push to registry
#1. set env variables
export REGISTRY_USER=<registry_user>
export VERSION=<image_ver>
#2. build and push the image to your image registry
make image-build
make image-push
2. Deploy
# 3. Deploy
make deploy ENV=dev
3. Cleanup
make undeploy
βοΈ Execution Parameters
Below is a list of all the parameters of the k8s-rightsizer. You can use them by setting env variables as shown above.
| Parameter | Value | Default | Description |
|---|---|---|---|
| REGISTRY_USER | localhost | The Container Registry | |
| VERSION | local | Version of the container image | |
| IMG | $(REGISTRY_USER)/k8s-rightsizer:$(VERSION) | Container image name and tag | |
| ENV | local | dev | local | The tool execution environment |
| RESIZE_ON_RECREATE | true | false | false | Whether to resize workload with update strategy Recreate |
| DRY_RUN | true | false | false | Plan the execution without resizing containers |
| WORKERS | 1 | Number of concurrent resizing workers* |
*Using concurrent workers can be helpful for speeding up work. Considerations:
- 3-5 workers are a safe and prudent threshold
- 10 workers for robust clusters with small microservices
- 20+ workers is not recommended as it may cause node pressure
π‘οΈ Rollback Logic Specification
The K8s Rightsizer is built with a "Safety-First" approach. Instead of simply applying changes, it treats every resource update as a monitored transaction.
π The Lifecycle of an Update
The tool follows a strict state machine for each entry in the Excel file:
1. Pre-checks
Before applying any recommendation, the engine performs a series of safety checks to ensure that resizing won't cause service disruptions or violate cluster policies.
A resize operation is automatically skipped if any of the following conditions are met:
-
Paused State: The workload (Deployment/StatefulSet) is currently paused by the user.
-
PDB Restrictions: A PodDisruptionBudget is active and too restrictive (e.g., maxUnavailable: 0 or current available replicas at the limit), making any pod restart unsafe.
-
Unsupported Update Strategies: Only RollingUpdate is currently supported to ensure zero-downtime transitions.
- OnDelete: Skipped because the update wouldn't trigger automatically.
- Recreate: Skipped by default to avoid the full downtime typical of this strategy. You can change this behavior by setting the -resize-on-procreate parameter to true.
-
Degraded Health: The workload is not healthy. We don't resize unstable systems.
-
Ongoing Rollout: A deployment is already in progress. We wait for the system to reach a stable state.
-
Critical Pod Errors: Critical issues are detected in the existing pods (e.g., CrashLoopBackOff, ImagePullBackOff). The resizer won't interfere with workloads that are already failing.
2. Snapshot Phase (Pre-Check)
Before any modification, the tool fetches the current resource configuration of the target (Deployment or StatefulSet).
- Action: Saves
cpuandmemorylimits/requests into an in-memory backup. - Metadata: Records the current
generationof the resource.
3. Application Phase
The tool applies the new values using a Strategic Merge Patch.
- Trigger: Updates the container spec with values from the Excel file.
- Wait: Triggers a new Rollout in Kubernetes.
4. Monitoring Phase (The "Watch" Loop)
This is the core of the rollback logic. The tool monitors the new Pods for a configurable timeout (default: 3 minutes).
The system identifies a failure if any of the following conditions are met:
- CrashLoopBackOff: The application crashes immediately after start.
- OOMKilled: The new memory limit is too low for the application's heap.
- ImagePullBackOff: Issues with the container registry.
- Unschedulable: The requested resources are too high for the available nodes (Insufficient CPU/Memory).
- Timeout: The Pods do not reach the
Readystate within the time limit.
5. Rollback Phase (Recovery)
If a failure is detected, the tool immediately aborts the monitoring and initiates recovery.
- Action: Re-applies the Snapshot taken in Phase 1.
- Verification: Ensures the resource returns to its original
Readystate. - Reporting: Logs the specific error (e.g., "OOMKilled detected") and marks the Excel row as
FAILED - ROLLED BACK.
π Logic Flowchart
- START β Read row from Excel.
- RETRIEVE β Retrieve current
resourcee. - PRECHECK β Check current
resourcesconditions - BACKUP β Create current
resourcesbackup. - PATCH β Apply new
resources. - MONITOR β Watch Pod status.
- β IF READY within 3m β COMMIT (Next row).
- β IF ERROR (OOM/Crash/Timeout) β ROLLBACK.
- RESTORE β Re-apply backup β LOG ERROR.
βοΈ Failure Detection Parameters
| Condition | Detection Method | System Response |
|---|---|---|
| Out of Memory | Container status OOMKilled |
Immediate Rollback |
| Startup Crash | Container status CrashLoopBackOff |
Immediate Rollback |
| Resource Starvation | Event FailedScheduling |
Immediate Rollback |
| Liveness Failure | Container Unhealthy events |
Rollback after 3 retries |
Configuration
π Excel File Structure
Recommendation file (.xslx, .xsl) must contain the following columns (order is important).
| Column Name | Description | Example Value |
|---|---|---|
| Environment | The stage environment. | production |
| Namespace | The K8s namespace where the resource resides. | prod-app |
| Kind | The type of resource (Deployment or StatefulSet). |
Deployment |
| Workload Name | The name of the resource. | api-gateway |
| Container | The name of the container in this workload. | api-gateway |
| Replicas | The number of replicas. | 2 |
| CPU Request | The current CPU request value. | 250m |
| CPU Limit | The current CPU limit value. | 500m |
| CPU Request Recommendation | The new CPU request value recommended. | 150m |
| CPU Limit Recommendation | The new CPU limit value recommended. | 300m |
| Mem Request | The current Memory request value. | 256Mi |
| Mem Limit | The current Memory limit value. | 512Mi |
| Mem Request Recommendation | The new Memory request value recommended. | 256Mi |
| Mem Limit Recommendation | The new Memory limit value recommended. | 512Mi |
Note Empty values for recommended columns are not allowed. Therefore set
- CPU Limit Recommended to 0m and
- Memory Limit Recommended to 0Mi