throttle-proxy

README

Prometheus Backpressure Proxy

Adaptive Protection for Your Backend Services

🛡️ Dynamically shield your services from traffic overload using smart, metrics-driven congestion control.

See Netflix concurrency-limits for the inspiration.

Why Use This Project?

Throttle Proxy is the best solution for ensuring the safety and stability of your distributed systems during load spikes. Here's why:

• Proven Algorithm: Utilizes the Additive Increase/Multiplicative Decrease (AIMD) algorithm, inspired by TCP congestion control, to dynamically adjust request concurrency.
• Real-Time Metrics: Leverages Prometheus metrics to make real-time decisions, ensuring your system adapts quickly to changing loads.
• Configurable and Flexible: Allows you to set custom thresholds and monitor multiple signals, providing fine-grained control over your traffic management.
• Prevents Failures: Helps prevent cascading failures and maintains system stability under unpredictable load conditions.

Key Features

• 📊 Adaptive Traffic Management: Automatically adjusts request concurrency based on real-time Prometheus metrics
• 🔀 Smart Scaling: Uses Additive Increase/Multiplicative Decrease (AIMD) algorithm
• 🚦 Configurable Limits: Set min and max concurrent request thresholds
• 🔍 Multi-Signal Monitoring: Track system health across multiple metrics simultaneously

Quick Example

config := proxymw.BackpressureConfig{
    EnableBackpressure: true,
    BackpressureQueries: []BackpressureQuery{
        {
            Query:              `sum(rate(http_server_errors_total[5m]))`,
            // Start to throttle when error rate reaches 50%
            WarningThreshold:   0.5,
            // Hard throttling up to 100 CongestionWindowMax when error rate is >80%
            EmergencyThreshold: 0.8,
        }
    },
    CongestionWindowMin: 10,
    CongestionWindowMax: 100,
}

How It Works

1. 🔭 Continuously monitor system metrics
2. 📈 Dynamically adjust request throughput
3. 🛑 Automatically throttle when system stress detected

When to Use

• Protecting microservices from sudden traffic spikes
• Preventing cascading failures
• Maintaining system stability under unpredictable load

Quick Start

1. Configure backpressure queries as Prometheus metrics
2. Define min/max request windows
3. Choose the server-side http proxy or client-side roundtripper
4. Import the starter Grafana dashboard
5. Let the proxy handle the rest!

Development

Installation

Build the docker-compose stack

make all
docker compose down
docker compose up --build

• Generate fake traffic with ./scripts/traffic_generator.py
• View metrics in the local Grafana instance

Lint and Test

make test
make lintfix

Upgrade dependencies

make deps

Contributing

CONTRIBUTING.md

Documentation

Overview

Package throttleproxy provides an adaptive backpressure proxy mechanism for dynamically managing traffic and protecting backend services using Prometheus metrics.

Usage Example:

	config := proxymw.BackpressureConfig{
	    EnableBackpressure: true,
	    BackpressureQueries: []BackpressureQuery{
	        {
	            Query:              `sum(rate(http_server_errors_total[5m]))`,
                Name:               "http_error_rate"
	            WarningThreshold:   0.5,
	            EmergencyThreshold: 0.8,
	        }
	    },
	    CongestionWindowMin: 10,
	    CongestionWindowMax: 100,
	}

Use Cases:

• Protecting microservices from traffic spikes
• Preventing cascading failures
• Maintaining system stability under unpredictable load

The package supports both server-side HTTP proxy and client-side RoundTripper implementations, providing flexible integration options.