predictive-cluster-sizer

README

Predictive Cluster Sizer

An experiment about predicting cluster size using historical data.

To build, use go 1.20+, then type go build in the root of project.

Demo:

1. Apply the predictive_node_scaling_demo.yaml manifest
2. It will generate a sample data series in the init container and dump it into the the /data volume
   • The generated data currently sine-wave oscillates over an hour so it's fast enough that you can see it predict scale your machineset while you're watching it
3. Then the predictive-cluster-sizer "controller" will feed the generated data into the prediction server in autoscaler-predictor-model to fit the model to the data
4. Then the predictive-cluster-sizer "controller" will start asking the autoscaler-predictor-model for predictions
5. The predictive-cluster-sizer will use those predictions to scale the machineset in your cluster (for now, the first machineset it sees)

[jkyros@jkyros-thinkpadp1gen5 predictive-cluster-sizer]$ oc logs -f shiftweek-cluster-sizer predictive-cluster-sizer
1119 08:42:27.090035       1 main.go:110] "Checking for total nodes in cluster" nodes=15
I1119 08:42:27.090078       1 main.go:129] "Filtering out compute nodes from total" nodes=12
I1119 08:42:27.090086       1 main.go:131] "CPU capacity for compute nodes" value="48"
I1119 08:42:27.090094       1 main.go:136] "Memory resource capacity for compute nodes" value="193115464Ki"
I1119 08:42:27.092881       1 main.go:171] "Each node we have has these resources" cpu="4" memory="16Gi"
I1119 08:42:27.096029       1 main.go:444] REQUESTING: http://localhost:5001/predict?timestamp=2023-11-19T09%3A02%3A27&type=resource
I1119 08:42:27.096097       1 main.go:182] Prediction: CPU: 39702 MEM: 158808
I1119 08:42:27.096148       1 main.go:185] MEMPRC: 0% CPUPRC: 0%	
I1119 08:42:27.096172       1 main.go:191] CPU Delta is -8298
I1119 08:42:27.096192       1 main.go:192] Memory Delta is -29781
I1119 08:42:27.096204       1 main.go:194] Each new node will have: CPU: 4000 MEM: 16384
I1119 08:42:27.096213       1 main.go:200] CPU thinks it needs -2 more nodes 
I1119 08:42:27.096221       1 main.go:201] Memory thinks it needs -1 more nodes
I1119 08:42:27.096228       1 main.go:213] I want to scale machineset jkyros-weekend-keda-s-8mjsf-worker-us-east-1f to 11 desired nodes (currently 13)
I1119 08:42:27.119017       1 main.go:235] Collecting usage ip-10-0-1-62.ec2.internal 4185804Ki 215m
I1119 08:42:27.119036       1 main.go:235] Collecting usage ip-10-0-12-244.ec2.internal 1523636Ki 41m
I1119 08:42:27.119052       1 main.go:235] Collecting usage ip-10-0-18-246.ec2.internal 1633220Ki 48m
I1119 08:42:27.119062       1 main.go:235] Collecting usage ip-10-0-21-208.ec2.internal 1481604Ki 46m
I1119 08:42:27.119073       1 main.go:235] Collecting usage ip-10-0-25-0.ec2.internal 5695232Ki 295m
I1119 08:42:27.119083       1 main.go:235] Collecting usage ip-10-0-30-199.ec2.internal 1547552Ki 39m
I1119 08:42:27.119095       1 main.go:235] Collecting usage ip-10-0-33-17.ec2.internal 1517460Ki 40m
I1119 08:42:27.119110       1 main.go:235] Collecting usage ip-10-0-38-39.ec2.internal 1436012Ki 40m
I1119 08:42:27.119124       1 main.go:235] Collecting usage ip-10-0-46-87.ec2.internal 1662680Ki 55m
I1119 08:42:27.119183       1 main.go:235] Collecting usage ip-10-0-48-177.ec2.internal 1510116Ki 40m
I1119 08:42:27.119200       1 main.go:235] Collecting usage ip-10-0-5-242.ec2.internal 1515616Ki 51m
I1119 08:42:27.119214       1 main.go:235] Collecting usage ip-10-0-51-174.ec2.internal 1583148Ki 45m
I1119 08:42:27.119225       1 main.go:251] Total usage CPU: 955 MEM: 24699
I1119 08:42:27.119239       1 main.go:281] Scaling machineset jkyros-weekend-keda-s-8mjsf-worker-us-east-1f to 11 desired nodes (currently 13 ready 12)

Documentation

Overview

Copyright 2023 Red Hat, Inc.

Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at

http://www.apache.org/licenses/LICENSE-2.0

Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License.