cluster-health-monitor

module
v0.0.46 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Oct 29, 2026 License: MIT

README

Cluster Health Monitor

A Kubernetes cluster health monitoring tool that checks various aspects of cluster health, including DNS resolution and pod startup time.

Overview

Cluster Health Monitor runs as a Kubernetes deployment and exposes metrics about the health of your cluster through a Prometheus endpoint.

Deployment

Deploying Base Manifests

To deploy the Cluster Health Monitor using the base manifests:

kubectl apply -k manifests/base/
Removing Deployment

To remove the base deployment:

kubectl delete -k manifests/base/
Customizing Deployment

For custom deployments, create your own overlay in manifests/overlays/ and change the directory to the directory containing kustomization.yaml, e.g., manifests/overlays/test.

Testing

GPU checks run only when enabled and are skipped for upgrade-request checks owned by a HealthSignal. They are experimental, and their runtime can exceed AKS RP's upgrade health-check deadline. Other node health checks still run for these requests.

Running Unit Tests

To run unit tests:

make test-unit
Running End-to-End (E2E) Tests

To run E2E tests, ensure Ginkgo is installed and run:

# Full E2E test: set up cluster, run tests, clean up cluster.
make test-e2e

This command supports the following environment variables:

  • E2E_SKIP_CLUSTER_SETUP=true - Skip cluster setup and use existing Kind cluster.
  • E2E_SKIP_CLUSTER_CLEANUP=true - Skip cluster cleanup after tests.
  • KIND_CLUSTER_NAME=<name> - Customize the name of the Kind cluster to use or create.
Setting Up a Kind Cluster for E2E Testing

To set up a Kind cluster for E2E tests:

make kind-setup-e2e

This creates a Kind cluster and loads the necessary images without deploying the application. You can also set KIND_CLUSTER_NAME to use a custom cluster name.

See useful commands for other commands that can be used once the test environment is set up.

Local Testing with Kind

Kind (Kubernetes IN Docker) is used to create a local Kubernetes cluster for testing and development.

Prerequisites
Setting Up a Local Environment

To set up a local test environment with Kind:

make kind-test-local

This command:

  1. Creates a Kind cluster.
  2. Builds a Docker image for the cluster health monitor.
  3. Loads the image into the Kind cluster.
  4. Applies the Kubernetes manifests using the test overlay.
Useful Commands

Once your test environment is set up, you can use the following commands:

  • Set kubectl context:

    make kind-export-kubeconfig
    
  • Mock LocalDNS:

    • Enable:

      make kind-enable-local-dns-mock
      
    • Disable:

      make kind-disable-local-dns-mock
      
  • Redeploy after changes:

    make kind-redeploy
    

    This rebuilds the image, loads it into the Kind cluster, and redeploys the application.

  • Clean up:

    make kind-delete-cluster
    

    This deletes the Kind cluster when you're done testing.

Directories

Path Synopsis
apis
cmd
nodechecker command
pkg
checker/apiserver
Package apiserver provides a checker for the Kubernetes API server.
Package apiserver provides a checker for the Kubernetes API server.
checker/azurepolicy
Package azurepolicy provides a checker for Azure Policy webhook validations.
Package azurepolicy provides a checker for Azure Policy webhook validations.
checker/dnscheck
Package dnscheck provides a checker for DNS.
Package dnscheck provides a checker for DNS.
checker/metricsserver
Package metricsserver provides a checker for the Kubernetes metrics server.
Package metricsserver provides a checker for the Kubernetes metrics server.
nodecheckerrunner/checkers/gpu
Package gpu runs the intrusive GPU benchmarks as ordinary node checkers.
Package gpu runs the intrusive GPU benchmarks as ordinary node checkers.
utils
Package utils contains small helpers for inspecting Kubernetes Node objects that are shared across controllers.
Package utils contains small helpers for inspecting Kubernetes Node objects that are shared across controllers.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL