predictive-horizontal-pod-autoscaler

README

Burst-aware Predictive Horizontal Pod Autoscaler

Forked from Predictive Horizontal Pod Autoscaler, this is a Custom Pod Autoscaler; aiming to have identical functionality to the Horizontal Pod Autoscaler, however with added predictive elements using statistical models and ability to handle bursty-load.

This uses the Horizontal Pod Autoscaler Custom Pod Autoscaler extensively to provide most functionality for the Horizontal Pod Autoscaler parts.

Why would I use it?

This autoscaler lets you choose models and fine tune them in order to predict how many replicas a resource should have, preempting events such as regular, repeated high load.

However, statistical models are not perfect and can be easily overfit. To adapt to real world scenarios with bursty-load, ability to override replica count recommendation from predictive model is added on top of the original Predictive Horizontal Pod Autoscaler.

Features

• Functionally identical to Horizontal Pod Autoscaler for calculating replica counts without prediction.
• Choice of statistical models to apply over Horizontal Pod Autoscaler replica counting logic.
  • Holt-Winters Smoothing
  • Linear Regression
• Allows customisation of Kubernetes autoscaling options without master node access. Can therefore work on managed solutions such as EKS or GCP.
  • CPU Initialization Period.
  • Downscale Stabilization.
  • Sync Period.
  • Initial Readiness Delay.
• Runs in Kubernetes as a standard Pod.
• In work and experimental: Provide burst-awareness based on configured Service Level Objective

How does it work?

This project works by calculating the number of replicas a resource should have, then storing these values and using statistical models against them to produce predictions for the future. These predictions are compared and can be used instead of the raw replica count calculated by the Horizontal Pod Autoscaler logic. The evaluator will then query Prometheus based on a defined SLO (Service Level Objective) and if the SLO is not met, the evaluator will increase the replica count over the amount predicted by the model.

More information

See the wiki for more information, such as guides and references.

See the examples/ directory for working code samples.

Developing this project

Environment

Developing this project requires these dependencies:

• Go >= 1.16
• Golint == v0.0.0-20201208152925-83fdc39ff7b5
• Docker
• Python == 3.8.5

Burst-awareness functionality requires the following dependencies:

• A service mesh, currently supports Istio >= 1.11.0
• Prometheus >= 2.17.0

Any Python dependencies must be installed by running:

pip install -r requirements-dev.txt

To view docs locally you need some Python dependencies, run:

pip install -r docs/requirements.txt

It is recommended to test locally using a local Kubernetes managment system, such as k3d (allows running a small Kubernetes cluster locally using Docker).

Once you have a cluster available, you should install the Custom Pod Autoscaler Operator (CPAO) onto the cluster to let you install the PHPA.

With the CPAO installed you can install your development builds of the PHPA onto the cluster by building the image locally, and then pushing the image to the K8s cluster's registry (to do that with k3d you can use the k3d image import command).

Finally you can deploy a PHPA example (see the examples/ directory for choices) to test your changes.

Note that the examples generally use ImagePullPolicy: Always, you may need to change this to ImagePullPolicy: IfNotPresent to use your local build.

Commands

• go mod vendor - generates a vendor folder.
• make - builds the Predictive HPA binary.
• make docker - builds the Predictive HPA image.
• make lint - lints the code.
• make beautify - beautifies the code, must be run to pass the CI.
• make unittest - runs the unit tests.
• make doc - hosts the documentation locally, at 127.0.0.1:8000.
• make view_coverage - opens up any generated coverage reports in the browser.

Documentation

Overview

Predictive Horizontal Pod Autoscaler provides executable Predictive Horizontal Pod Autoscaler logic, which can be built into a Custom Pod Autoscaler. The Horizontal Pod Autoscaler has two modes, metric gathering and evaluation. Metric mode gathers metrics, taking in a resource to get the metrics for and outputting these metrics as serialised JSON. Evaluation mode makes decisions on how many replicas a resource should have, taking in metrics and outputting evaluation decisions as seralised JSON. The predictive element uses past evaluations to predict how to scale in the future.