backfiller

README

Backfiller Integration Tests

This directory contains minimal integration tests for the CloudZero Agent backfiller functionality. The tests verify that the backfiller can correctly discover Kubernetes namespace resources and send them to the collector in the proper Prometheus RemoteWrite format.

Test Components

1. Kind Cluster Setup (kind-config.yaml)

• Creates a minimal single-node Kind cluster named cloudzero-backfiller-test
• Includes volume mounts for test data exchange

2. Test Resources (testdata/test-namespaces.yaml)

• Creates 4 test namespaces with various labels and annotations:
  • production - with environment=production, team=backend, cost-center=engineering
  • staging - with environment=staging, team=frontend, cost-center=engineering
  • development - with environment=development, team=devops, cost-center=operations
  • test-exclude - with exclude-from-monitoring=true (should be filtered out)

3. Mock Collector (mock_collector.go)

• HTTP server that mimics the CloudZero collector API
• Captures Prometheus RemoteWrite requests (protobuf + snappy compression)
• Validates received metrics and saves them to files for inspection
• Provides methods to verify namespace-specific metrics

4. Integration Test (backfiller_integration_test.go)

• End-to-end test that:
  • Sets up Kind cluster
  • Applies test namespaces
  • Starts mock collector
  • Runs backfiller against the cluster
  • Validates results

Prerequisites

Install Required Tools

# Install Kind on macOS with Homebrew
brew install kind

# Or install Kind using Go
go install sigs.k8s.io/kind@latest

# Install kubectl
# Follow instructions at https://kubernetes.io/docs/tasks/tools/install-kubectl/

# Verify installation
kind version
kubectl version --client

Docker Desktop Setup

Kind (Kubernetes in Docker) uses your Docker Desktop to create Kubernetes clusters inside Docker containers. The kind-config.yaml file is just a configuration file that tells Kind how to set up the cluster.

Setup Steps:

1. Ensure Docker Desktop is Running

   • Make sure Docker Desktop is running on your machine
   • Kind will use Docker Desktop's Docker daemon

2. Verify Docker Desktop Setup

   # Check Docker is running
   docker version
   
   # Check if Docker Desktop has Kubernetes enabled (optional)
   docker info | grep -i kubernetes
   

3. Test Kind Installation

   # Create a test cluster
   kind create cluster --name test-cluster
   
   # Verify it's working
   kubectl cluster-info --context kind-test-cluster
   
   # Clean up
   kind delete cluster --name test-cluster
   

Note: You don't need to manually create a Kind cluster - the test handles that automatically!

Running the Tests

Quick Test

cd tests/backfiller
make test-backfiller

Debug Mode (keeps cluster running)

cd tests/backfiller
make test-backfiller-debug

Note: The first run may take 5-10 minutes to download the Kind node image. Subsequent runs will be much faster.

Check Test Status

cd tests/backfiller
make test-backfiller-status

Manual Cleanup

cd tests/backfiller
make test-backfiller-cleanup

Test Configuration

The test uses a configuration (testdata/test-config.yaml) that:

• Enables label filtering for: environment, team, cost-center
• Enables annotation filtering for: deployment.kubernetes.io/managed-by, description
• Focuses on namespaces only (pods, deployments, etc. are disabled)
• Uses localhost:8080 as the collector endpoint

Expected Results

The test validates:

1. Resource Discovery: Backfiller discovers all 4 test namespaces
2. Label Filtering: Only configured labels are captured
3. Annotation Filtering: Only configured annotations are captured
4. Protobuf Format: Metrics are sent in proper Prometheus RemoteWrite format
5. Collector Integration: Mock collector receives and parses the data correctly

Test Output

The test generates several outputs:

1. Console logs showing test progress and results
2. Metric files in /tmp/cloudzero-backfiller-test-* directories
3. Detailed validation of expected vs actual namespaces found

How the Test Works

The test creates everything automatically:

1. Creates the Kind cluster using the embedded configuration:

   kind create cluster --name cloudzero-backfiller-test --wait 60s
   

2. Applies the test namespaces to the cluster:

   kubectl apply -f testdata/test-namespaces.yaml
   

3. Runs the backfiller against the cluster

4. Validates the results

5. Cleans up (unless you use debug mode)

Manual Kind Cluster Operations

If you want to manually work with Kind clusters:

# Create a cluster manually
kind create cluster --name cloudzero-backfiller-test

# List clusters
kind get clusters

# Get kubeconfig
kind get kubeconfig --name cloudzero-backfiller-test

# Use kubectl with the cluster
kubectl cluster-info --context kind-cloudzero-backfiller-test

# Delete the cluster
kind delete cluster --name cloudzero-backfiller-test

Troubleshooting

Common Issues

1. Kind cluster creation fails:

   • Check if Docker Desktop is running
   • Ensure Kind is properly installed
   • Check if port 8080 is available

2. Test timeout:

   • Increase timeout in test configuration
   • Check if namespaces are created successfully: kubectl get namespaces

3. No metrics received:

   • Check mock collector logs
   • Verify backfiller configuration
   • Ensure Kind cluster is accessible

Debugging

1. Keep cluster running:

   make test-backfiller-debug
   

2. Check cluster resources:

   kubectl --kubeconfig /tmp/kubeconfig get namespaces
   kubectl --kubeconfig /tmp/kubeconfig describe namespace production
   

3. View mock collector output:

   find /tmp -name "cloudzero-backfiller-test-*" -type d
   ls -la /tmp/cloudzero-backfiller-test-*/received_metrics_*.json
   

What This Test Validates

✅ Namespace Discovery: Confirms backfiller finds all Kubernetes namespaces
✅ Label Filtering: Verifies only configured labels are captured
✅ Annotation Filtering: Verifies only configured annotations are captured
✅ Protobuf Format: Validates proper Prometheus RemoteWrite format
✅ End-to-End Pipeline: Tests complete flow from K8s API → Backfiller → Collector

Extending the Tests

To extend these tests:

1. Add more resource types: Modify testdata/test-namespaces.yaml to include pods, deployments, etc.
2. Test different filters: Update testdata/test-config.yaml with different label/annotation patterns
3. Add negative tests: Create resources that should be filtered out
4. Performance testing: Add many namespaces to test pagination and worker pools

Files Structure

tests/backfiller/
├── README.md                          # This file
├── Makefile                           # Test automation
├── testdata/
│   ├── test-namespaces.yaml           # Test Kubernetes resources
│   └── test-config.yaml               # Backfiller configuration
├── mock_collector.go                  # Mock collector implementation
├── backfiller_integration_test.go     # Main integration test
└── [generated files]                  # Test outputs and temporary files

Documentation

Overview

Package backfiller provides utilities for testing label backfiller integrations.

Index

• type MockCollector
  • func NewMockCollector(port int, outputDir string) *MockCollector
  • func (mc *MockCollector) Clear()
  • func (mc *MockCollector) GetNamespaceMetrics() []prompb.TimeSeries
  • func (mc *MockCollector) GetReceivedData() []prompb.WriteRequest
  • func (mc *MockCollector) SetExpectedLabels(labels map[string]string)
  • func (mc *MockCollector) Start() error
  • func (mc *MockCollector) Stop() error
  • func (mc *MockCollector) ValidateNamespaceMetrics(expectedNamespaces []string, expectedLabels []string) error
• type ReceivedMetrics

Types

type MockCollector

type MockCollector struct {
	// contains filtered or unexported fields
}

MockCollector captures and validates Prometheus RemoteWrite requests

func NewMockCollector

func NewMockCollector(port int, outputDir string) *MockCollector

NewMockCollector creates a new mock collector for testing

func (*MockCollector) Clear

func (mc *MockCollector) Clear()

Clear clears all received data

func (*MockCollector) GetNamespaceMetrics

func (mc *MockCollector) GetNamespaceMetrics() []prompb.TimeSeries

GetNamespaceMetrics returns only the namespace-related metrics

func (*MockCollector) GetReceivedData

func (mc *MockCollector) GetReceivedData() []prompb.WriteRequest

GetReceivedData returns all received WriteRequests

func (*MockCollector) SetExpectedLabels

func (mc *MockCollector) SetExpectedLabels(labels map[string]string)

SetExpectedLabels sets the labels we expect to see in the metrics

func (*MockCollector) Start

func (mc *MockCollector) Start() error

Start starts the mock collector HTTP server

func (*MockCollector) Stop

func (mc *MockCollector) Stop() error

Stop stops the mock collector server

func (*MockCollector) ValidateNamespaceMetrics

func (mc *MockCollector) ValidateNamespaceMetrics(expectedNamespaces []string, expectedLabels []string) error

ValidateNamespaceMetrics validates that expected namespace metrics were received

type ReceivedMetrics

type ReceivedMetrics struct {
	Timestamp time.Time
	Request   prompb.WriteRequest
}

ReceivedMetrics holds captured metrics data for verification