shedder

README

kube-shedder

A minimal Go library for pod-level load shedding in Kubernetes services.

Overview

kube-shedder tracks in-flight HTTP requests and provides automatic load shedding when a pod becomes overloaded. When the number of concurrent requests exceeds a configurable limit, the pod's readiness probe returns 503, signaling Kubernetes to stop routing traffic until load drops.

Installation

go get github.com/sampath030/kube-shedder

Quick Start

package main

import (
    "net/http"

    "github.com/sampath030/kube-shedder"
)

func main() {
    // Create a shedder with a hard limit of 100 concurrent requests
    s := shedder.New(shedder.Config{
        HardLimit: 100,
    })

    // Add readiness endpoint for Kubernetes
    http.Handle("/ready", s.ReadyHandler())

    // Add health endpoint for liveness probe
    http.Handle("/health", shedder.HealthHandler())

    // Wrap your API handlers with the middleware
    http.Handle("/api/", s.Middleware(apiHandler))

    http.ListenAndServe(":8080", nil)
}

Features

Hard Limit

When in-flight requests exceed HardLimit:

• New requests receive 503 Service Unavailable
• Readiness endpoint returns 503
• Kubernetes removes the pod from load balancing

s := shedder.New(shedder.Config{
    HardLimit: 100,  // Required: max concurrent requests
})

Soft Limit (Optional)

Soft limit enables selective shedding of low-priority requests before reaching hard limit:

Using a callback function:

s := shedder.New(shedder.Config{
    HardLimit: 100,
    SoftLimit: 80,
    ShedDecider: func(r *http.Request) bool {
        // Return true to shed this request
        return r.Header.Get("X-Priority") == "low"
    },
})

Using header matching:

s := shedder.New(shedder.Config{
    HardLimit: 100,
    SoftLimit: 80,
    ShedHeader: &shedder.HeaderMatcher{
        Name:  "X-Priority",
        Value: "low",
    },
})

Shed Notifications

Get notified when requests are shed (useful for logging/metrics):

s := shedder.New(shedder.Config{
    HardLimit: 100,
    OnShed: func(r *http.Request, reason shedder.ShedReason) {
        log.Printf("Shed request: %s (reason: %s)", r.URL.Path, reason)
    },
})

API

Types

// Config holds shedder configuration
type Config struct {
    HardLimit   int64                        // Required: max in-flight requests
    SoftLimit   int64                        // Optional: threshold for selective shedding
    ShedDecider func(r *http.Request) bool   // Optional: callback to decide shedding
    ShedHeader  *HeaderMatcher               // Optional: header-based shedding
    OnShed      func(r *http.Request, ShedReason) // Optional: notification callback
}

// HeaderMatcher for header-based shedding
type HeaderMatcher struct {
    Name  string  // Header name (e.g., "X-Priority")
    Value string  // Value to match (e.g., "low")
}

// ShedReason indicates why a request was shed
type ShedReason int
const (
    ShedReasonHardLimit ShedReason = iota
    ShedReasonSoftLimit
)

Methods

// Create a new shedder
s := shedder.New(cfg Config) *Shedder

// Convenience constructor
s := shedder.NewWithLimits(hardLimit, softLimit int64) *Shedder

// HTTP middleware
handler := s.Middleware(next http.Handler) http.Handler

// Middleware function for chains
mw := s.MiddlewareFunc() func(http.Handler) http.Handler

// Readiness handler (200 OK or 503)
handler := s.ReadyHandler() http.Handler

// Health handler (always 200 OK)
handler := shedder.HealthHandler() http.Handler

// Status methods
inflight := s.Inflight() int64
overloaded := s.IsOverloaded() bool
softOverloaded := s.IsSoftOverloaded() bool

// Notes:
// - OnShed is invoked for both hard and soft shedding events.
// - If SoftLimit > 0 but neither ShedDecider nor ShedHeader is set, soft shedding is skipped.

Response Headers

Shed responses include:

• Retry-After: 1 - Suggests retry after 1 second
• X-Shed-Reason: hard_limit|soft_limit - Indicates why the request was shed

Kubernetes Integration

Configure your deployment with separate readiness and liveness probes:

Important: Do NOT use the same endpoint for both probes. The readiness probe (/ready) returns 503 when overloaded, which is intentional - it removes the pod from load balancing. The liveness probe (/health) should always return 200 as long as the process is running. Using /ready for liveness would cause Kubernetes to restart healthy-but-busy pods.

Set failureThreshold on the readiness probe low (typically 1) so the pod is marked NotReady as soon as load shedding starts. Liveness can keep the default higher threshold since it should only trip on true process failure.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: my-app
spec:
  template:
    spec:
      containers:
      - name: app
        ports:
        - containerPort: 8080
        livenessProbe:
          httpGet:
            path: /health
            port: 8080
          periodSeconds: 10
        readinessProbe:
          httpGet:
            path: /ready
            port: 8080
          periodSeconds: 5
          failureThreshold: 1

Framework Compatibility

kube-shedder uses standard net/http types and works with any Go HTTP framework:

Chi:

r := chi.NewRouter()
r.Use(s.MiddlewareFunc())

Gin:

r := gin.New()

// Wrap a standard http.Handler with kube-shedder, then register via gin.WrapH
api := s.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
    w.Write([]byte("ok"))
}))
r.Any("/api/*path", gin.WrapH(api))

Echo:

e := echo.New()
e.Use(echo.WrapMiddleware(s.MiddlewareFunc()))

License

MIT

Documentation

Overview

Package shedder provides pod-level load shedding for Kubernetes services.

The library tracks how many HTTP requests are currently in flight for a given pod. When the number exceeds a configurable HardLimit, the readiness endpoint returns 503, signaling Kubernetes to remove the pod from load balancing until load drops.

Basic Usage

// Create a shedder with a hard limit of 100 concurrent requests
s := shedder.New(shedder.Config{
    HardLimit: 100,
})

// Use the middleware with your HTTP server
http.Handle("/api/", s.Middleware(apiHandler))

// Add the readiness endpoint
http.Handle("/ready", s.ReadyHandler())

Soft Limit with Callback

An optional SoftLimit enables selective shedding of low-priority requests before reaching the HardLimit:

s := shedder.New(shedder.Config{
    HardLimit: 100,
    SoftLimit: 80,
    ShedDecider: func(r *http.Request) bool {
        // Shed requests with low priority header
        return r.Header.Get("X-Priority") == "low"
    },
})

Soft Limit with Header Matching

Alternatively, use HeaderMatcher for simple header-based shedding:

s := shedder.New(shedder.Config{
    HardLimit: 100,
    SoftLimit: 80,
    ShedHeader: &shedder.HeaderMatcher{
        Name:  "X-Priority",
        Value: "low",
    },
})

Integration with Kubernetes

Important: Use SEPARATE endpoints for liveness and readiness probes. The readiness probe should use ReadyHandler (returns 503 when overloaded). The liveness probe should use HealthHandler (always returns 200). Using the readiness endpoint for liveness would cause Kubernetes to restart healthy-but-busy pods.

The handlers are path-agnostic - register them at any path you prefer:

// Liveness - always 200 if process is running
http.Handle("/healthz", shedder.HealthHandler())

// Readiness - 503 when overloaded
http.Handle("/readyz", s.ReadyHandler())

Kubernetes deployment configuration (paths must match your registration):

livenessProbe:
  httpGet:
    path: /healthz
    port: 8080
readinessProbe:
  httpGet:
    path: /readyz
    port: 8080
  periodSeconds: 5
  failureThreshold: 1

Index

• func HealthHandler() http.Handler
• type Config
• type HeaderMatcher
• type ShedDecider
• type ShedReason
  • func (r ShedReason) String() string
• type Shedder
  • func New(cfg Config) *Shedder
  • func NewWithLimits(hardLimit, softLimit int64) *Shedder
  • func (s *Shedder) Inflight() int64
  • func (s *Shedder) IsOverloaded() bool
  • func (s *Shedder) IsSoftOverloaded() bool
  • func (s *Shedder) Middleware(next http.Handler) http.Handler
  • func (s *Shedder) MiddlewareFunc() func(http.Handler) http.Handler
  • func (s *Shedder) ReadyHandler() http.Handler
  • func (s *Shedder) ReadyHandlerFunc() http.HandlerFunc

Functions

func HealthHandler

func HealthHandler() http.Handler

HealthHandler returns a simple health check handler that always returns 200 OK. This is suitable for Kubernetes liveness probes.

Types

type Config

type Config struct {
	// HardLimit is the maximum number of in-flight requests before the
	// readiness endpoint returns 503. This is required and must be > 0.
	HardLimit int64

	// SoftLimit is the threshold for soft overload behavior.
	// If SoftLimit > 0 and inflight > SoftLimit (but <= HardLimit),
	// the ShedDecider is consulted to determine if requests should be fast-failed.
	// If SoftLimit is 0 or negative, soft overload behavior is disabled.
	SoftLimit int64

	// ShedDecider is called when in soft overload state to determine
	// whether to shed a request. If nil and SoftLimit > 0, soft shedding
	// is effectively disabled unless ShedHeader is set.
	ShedDecider ShedDecider

	// ShedHeader specifies a header name and value for automatic shedding.
	// When in soft overload state, requests with this header matching will be shed.
	// This is an alternative to ShedDecider for simple priority-based shedding.
	// If both ShedDecider and ShedHeader are set, ShedDecider takes precedence.
	ShedHeader *HeaderMatcher

	// OnShed is an optional callback invoked when a request is shed.
	// Useful for logging or metrics (without adding direct dependencies).
	OnShed func(r *http.Request, reason ShedReason)
}

Config holds the configuration for a Shedder instance.

type HeaderMatcher

type HeaderMatcher struct {
	Name  string // Header name, e.g., "X-Priority"
	Value string // Header value to match, e.g., "low"
}

HeaderMatcher defines a header name and value to match for shedding.

type ShedDecider

type ShedDecider func(r *http.Request) bool

ShedDecider is a callback function that determines whether a request should be shed when in soft overload state. It receives the incoming request and returns true if the request should be rejected.

type ShedReason

type ShedReason int

ShedReason indicates why a request was shed.

const (
	// ShedReasonHardLimit indicates the request was shed because
	// in-flight requests exceeded HardLimit.
	ShedReasonHardLimit ShedReason = iota

	// ShedReasonSoftLimit indicates the request was shed because
	// in-flight requests exceeded SoftLimit and the ShedDecider
	// (or header match) determined it should be shed.
	ShedReasonSoftLimit
)

func (ShedReason) String

func (r ShedReason) String() string

type Shedder

type Shedder struct {
	// contains filtered or unexported fields
}

Shedder tracks in-flight requests and provides load shedding capabilities.

func New

func New(cfg Config) *Shedder

New creates a new Shedder with the given configuration. It panics if HardLimit is <= 0.

func NewWithLimits

func NewWithLimits(hardLimit, softLimit int64) *Shedder

NewWithLimits creates a new Shedder with just hard and soft limits. This is a convenience function for simple use cases without callbacks.

func (*Shedder) Inflight

func (s *Shedder) Inflight() int64

Inflight returns the current number of in-flight requests.

func (*Shedder) IsOverloaded

func (s *Shedder) IsOverloaded() bool

IsOverloaded returns true if in-flight requests exceed HardLimit.

func (*Shedder) IsSoftOverloaded

func (s *Shedder) IsSoftOverloaded() bool

IsSoftOverloaded returns true if soft limit is configured and in-flight requests exceed SoftLimit (but not HardLimit).

func (*Shedder) Middleware

func (s *Shedder) Middleware(next http.Handler) http.Handler

Middleware returns an http.Handler that wraps the given handler with load shedding logic.

The middleware:

1. Increments the in-flight counter
2. Checks if HardLimit is exceeded - if so, returns 503 immediately
3. If SoftLimit is exceeded and ShedDecider returns true, returns 503
4. Otherwise, calls the wrapped handler
5. Decrements the in-flight counter when done (even on panic)

func (*Shedder) MiddlewareFunc

func (s *Shedder) MiddlewareFunc() func(http.Handler) http.Handler

MiddlewareFunc is a convenience wrapper that returns a function suitable for use with middleware chains that expect func(http.Handler) http.Handler.

func (*Shedder) ReadyHandler

func (s *Shedder) ReadyHandler() http.Handler

ReadyHandler returns an http.Handler that implements a Kubernetes readiness probe endpoint.

Returns:

• 200 OK when in-flight requests <= HardLimit
• 503 Service Unavailable when in-flight requests > HardLimit

func (*Shedder) ReadyHandlerFunc

func (s *Shedder) ReadyHandlerFunc() http.HandlerFunc

ReadyHandlerFunc is a convenience function that returns the readiness handler as an http.HandlerFunc.