heartbeats

README

Heartbeats Operator

A Kubernetes operator that monitors the health of HTTP endpoints by periodically sending requests and tracking their responses.

Disclaimer

[!WARNING] AI-GENERATED CODE AHEAD! Every single line of code in this repository is generated.

This entire repo is the result of me going "Hey, I wonder if I can vibe-code a Kubernetes operator in a weekend?"

Me over the weekend:

Buddy-coded with Cursor Pro, 307 premium model requests.

[!WARNING] Use at your own risk. No guarantees, no warranties, just pure vibes and questionable life choices. 🤪

Overview

The Heartbeats Operator provides a custom resource Heartbeat that allows you to monitor the health of HTTP endpoints in your Kubernetes cluster. It periodically checks the endpoints and updates their status based on HTTP response codes with comprehensive error handling and reporting.

Features

• Robust Health Monitoring: Monitor HTTP endpoints with configurable intervals and retry logic
• Flexible Status Code Ranges: Define custom status code ranges for healthy/unhealthy states
• Secure Configuration: Store endpoint URLs securely in Kubernetes secrets
• Real-time Status Updates: Detailed health information with comprehensive error messages
• Configurable Timeouts: Adjustable timeout and retry settings for different network conditions
• Comprehensive Error Handling: Detailed error messages for troubleshooting
• Report Endpoints: Reporting to healthy/unhealthy endpoints for external monitoring systems
• Controller-runtime metrics: HTTPS /metrics on port 8443

Installation

Prerequisites

• Kubernetes cluster
• kubectl configured to access your cluster

make install
make deploy IMG=ghcr.io/siutsin/heartbeats:latest

make install applies the CRD. make deploy builds config/default with kustomize.

Usage

Basic Example

1. Create a secret containing your endpoint URLs:

   apiVersion: v1
   kind: Secret
   metadata:
     name: heartbeat-endpoints
   type: Opaque
   stringData:
     targetEndpoint: "https://api.example.com/health"
     healthyEndpoint: "https://httpbin.org/status/200"
     unhealthyEndpoint: "https://httpbin.org/status/500"
   

2. Create a Heartbeat resource:

   apiVersion: monitoring.siutsin.com/v1alpha1
   kind: Heartbeat
   metadata:
     name: api-health
   spec:
     endpointsSecret:
       name: heartbeat-endpoints
       targetEndpointKey: targetEndpoint
       healthyEndpointKey: healthyEndpoint
       unhealthyEndpointKey: unhealthyEndpoint
       healthyEndpointMethod: GET
       unhealthyEndpointMethod: GET
     expectedStatusCodeRanges:
       - min: 200
         max: 299
     interval: 30s
   

Advanced Example with Report Endpoints

apiVersion: monitoring.siutsin.com/v1alpha1
kind: Heartbeat
metadata:
  name: api-health-with-reporting
spec:
  endpointsSecret:
    name: heartbeat-endpoints
    targetEndpointKey: targetEndpoint
    healthyEndpointKey: healthyEndpoint
    unhealthyEndpointKey: unhealthyEndpoint
    healthyEndpointMethod: GET
    unhealthyEndpointMethod: GET
  expectedStatusCodeRanges:
    - min: 200
      max: 299
    - min: 404
      max: 404  # Accept 404 as healthy for certain endpoints
  interval: 30s

Configuration Options

Heartbeat Spec

Field	Type	Description	Required
endpointsSecret	object	Reference to the secret containing endpoint URLs	Yes
expectedStatusCodeRanges	array	Ranges of HTTP status codes considered healthy	Yes
interval	string	Time between health checks (e.g., "30s", "5m", "1h")	Yes

EndpointsSecret

Field	Type	Description	Required
name	string	Name of the secret	Yes
namespace	string	Namespace of the secret (defaults to Heartbeat's namespace)	No
targetEndpointKey	string	Key containing the target endpoint URL	Yes
healthyEndpointKey	string	Key containing the healthy endpoint URL for reporting	Yes
unhealthyEndpointKey	string	Key containing the unhealthy endpoint URL for reporting	Yes
healthyEndpointMethod	string	HTTP method for the healthy report (GET, POST, PUT, PATCH)	No
unhealthyEndpointMethod	string	HTTP method for the unhealthy report (GET, POST, PUT, PATCH)	No

StatusCodeRange

Field	Type	Description	Required
min	integer	Minimum status code in range (100-599)	Yes
max	integer	Maximum status code in range (100-599)	Yes

Status Information

The Heartbeat resource's status includes:

• healthy: Boolean indicating if the endpoint is healthy
• lastStatus: Last HTTP status code received
• message: Human-readable status message with detailed error information
• lastChecked: Timestamp of the last health check
• reportStatus: Status of reporting to external endpoints ("Success" or "Failure")

Error Messages

The operator provides detailed error messages for troubleshooting:

• missing required key: A required endpoint key is missing from the secret
• endpoint is not specified: An endpoint URL is empty or not provided
• failed to check endpoint health: Network or HTTP errors during health checks
• endpoint timed out: The endpoint took too long to respond
• invalid status code range: Status code range has invalid min/max values
• status code is not within expected ranges: Endpoint returned unexpected status code

Development

Development Prerequisites

• Go 1.24+
• Docker
• Kind (for local testing)
• kubectl

Building and Testing

# Run unit tests
make test

# Run unit tests with race detection (for CI)
make test-ci

# Run e2e tests locally
make test-e2e LOCAL=true

# Run e2e tests with race detection (for CI)
make test-e2e-ci LOCAL=true

# Build the operator
make build

# Build Docker image
make docker-build

Code Quality

# Format code
make fmt

# Run linter
make lint

# Run linter with fixes
make lint-fix

# Check markdown files
make lint-markdown

Logging

The operator logs JSON to stdout via slog.

Log Levels

• INFO: General operational information, health check results
• ERROR: Error conditions, failed health checks, configuration issues
• DEBUG/V(1): Detailed debugging information, HTTP request details

Monitoring

The operator exposes controller-runtime metrics at /metrics on port 8443.

Metrics Access

The metrics endpoint is secured and requires authentication. Access is controlled via RBAC:

# Get service account token
TOKEN=$(kubectl create token heartbeats-operator-controller-manager -n heartbeats-operator-system)

# Access metrics
curl -k -H "Authorization: Bearer $TOKEN" \
  https://heartbeats-operator-controller-manager-metrics-service.heartbeats-operator-system.svc.cluster.local:8443/metrics

Troubleshooting

Common Issues

1. Endpoint Timeout

   • Check if the endpoint is accessible from the cluster
   • Verify network policies allow the connection
   • Consider increasing the timeout duration
   • Check logs for detailed timeout information

2. Invalid Status Code

   • Ensure the endpoint returns expected status codes
   • Verify the status code ranges in the Heartbeat spec
   • Check the status message for specific error details

3. Secret Not Found

   • Confirm the secret exists in the correct namespace
   • Verify the secret keys match the Heartbeat configuration
   • Check for "missing required key" error messages

4. Report Endpoint Failures

   • Verify the report endpoints are accessible
   • Check the reportStatus field in the Heartbeat status
   • Review logs for report endpoint errors

Debugging

# Check Heartbeat status
kubectl get heartbeat <name> -o yaml

# View operator logs
kubectl logs -n heartbeats-operator-system deployment/heartbeats-operator-controller-manager

# Check metrics
kubectl port-forward -n heartbeats-operator-system svc/heartbeats-operator-controller-manager-metrics-service 8443:8443

Contributing

Contributions are welcome! Please see our Contributing Guide for details.

Development Guidelines

• Follow the existing code style and patterns
• Add comprehensive unit tests for new features
• Include e2e tests for integration scenarios
• Use structured logging with appropriate log levels
• Follow error handling patterns established in the codebase

License

This project is licensed under the Apache License 2.0 - see the LICENSE file for details.