levee

README

Levee: Self Tuning Circuit Breaker and Concurrency Limiter

lev·ee /ˈlevi/ noun

An embankment built to prevent the overflow of a river or body of water; specifically: an artificial bank confining a river channel or limiting adjacent areas subject to flooding.

Levee keeps more of your business flowing under dynamic conditions than any other circuit-breaker and rate-limiter combination.

• 248 bytes memory overhead (yes, under a quarter of a kB)
• multi-million request processing capacity -- per CPU core

What is Levee?

Levee is a self-tuning circuit breaker and concurrency-based rate limiter for Go services.

• Always watching, always adapting
• Fully self-contained, 100% in-process operation
• No external dependencies

Works as a circuit breaker on outbound requests to prevent cascading failures from degraded or faulty dependencies. Works as a rate limiter on incoming requests to prevent failure due to overload.

Levee is designed to be dead simple to integrate and take the guesswork out of configuring operational parameters.

Inspired by Hystrix from Netflix.

Why Levee?

Circuit breakers and concurrency limiters are essential components of any distributed system. However, the operating parameters of services can change over time, both short term (e.g., due to a sudden spike in traffic) and long term (e.g., due to changes in the service's dependencies).

This means that the parameters of the circuit breaker and concurrency limiter need to be adjusted frequently to ensure optimal performance, but they're rarely updated often enough. Besides, circuit breaker tuning is done unscientifically, based on heuristics and guesswork.

This can lead to suboptimal performance, with the circuit breaker either being too aggressive (causing unnecessary service denial) or too lenient (allowing cascading failures).

Levee continuously monitors the RED metrics -- R: Requests per Second, E: Error Rate, D: Duration aka Response Time or Latency -- as well as in-flight concurrents. It computes statistical properties of these signals to adjust its operating parameters dynamically, ensuring that the circuit breaker and concurrency limiter are always optimally tuned.

Adding levee instances throughout the network can provide the resiliency benefits of a decentralised service mesh while keeping yaml-hell away.

Levee is also painstakingly designed to consume a fixed, small amount of memory, making it suitable for use in high-performance, low-latency services.

How to use Levee?

Basic, in-band usage:

package main

import (
	"fmt"
	"time"

	"github.com/codemartial/levee"
)

func main() {
	slo := levee.SLO{
		SuccessRate: 0.95,
		Timeout:     time.Millisecond * 100,
	}

	l := levee.NewLevee(slo)

	stateChange, err := l.Call(func() error {
		// Call the upstream service
		return nil
	})

	switch stateChange.State {
	case levee.OPEN:
		fmt.Println("Circuit breaker is Open")
	case levee.THROTTLED:
		fmt.Println("Circuit breaker is Throttling")
	case levee.CLOSED:
		fmt.Println("Circuit breaker is Closed")
	}
}

Advanced, out-of-band usage:

Levee can be called out-of-band so you can better organise your code and just call Levee at the start and end of your tasks. You can even control the timing, e.g. for stream processors that work on event-time or ingestion-time. Here's an example:

package main

import (
	"fmt"
	"time"

	"github.com/codemartial/levee"
)

func main() {
	slo := levee.SLO{
		SuccessRate: 0.95,
		Timeout:     time.Millisecond * 100,
	}

	l := levee.NewLevee(slo)

	// Check circuit state before starting the task
	start := time.Now()
	stateChange, err := l.Start(start)
	if err != nil {
		fmt.Println("Circuit is open, request rejected")
		return
	}

	// Perform the actual task
	taskErr := callUpstreamService()
	end := time.Now()
	duration := end.Sub(start)

	// Report the outcome
	if taskErr != nil {
		stateChange = l.Fail(end, duration)
	} else {
		stateChange = l.Success(end, duration)
	}

	fmt.Printf("Circuit state: %v\n", stateChange.State)
}

Benchmark

Levee is validated in a closed-loop distributed simulation where circuit-breaker decisions shape backend load, autoscaling, and queue backpressure. Across seven traffic and error-rate variations, Levee beats a meticulously tuned static breaker in every one, with its widest margins under extreme overload: where a binary open/close cannot keep up, Levee's concurrency control matches admission to available capacity.

See benchmarks/ for the full methodology and results.

Run with (from the benchmarks/ directory):

go test -v -run TestDistributedBenchmarkFirstIncident -timeout 15m

TODO

Levee is still a work in progress. Here are some of the things that need to be done:

1. Implement concurrent access (done)
2. Implement save state and restore state capability (done)
3. Implement state updates over channels
4. Implement system load monitoring

The last one is rather tricky. There is no standard way to access the environment load in Go. The best I may be able to do is to make it Linux specific. Even that is complicated being split between VM/BM and containers.

Documentation

Index

• Variables
• type Levee
  • func NewLevee(slo SLO) *Levee
  • func RestoreState(s *LeveeState) *Levee
  • func (l *Levee) Call(f func() error) (StateChange, error)
  • func (l *Levee) Fail(ts time.Time, duration time.Duration) StateChange
  • func (l *Levee) SaveState() (*LeveeState, error)
  • func (l *Levee) Start(ts time.Time) (StateChange, error)
  • func (l *Levee) State() State
  • func (l *Levee) Success(ts time.Time, duration time.Duration) StateChange
• type LeveeState
• type SLO
• type State
• type StateChange
• type Trigger

Variables

var ErrCircuitOpen = errors.New("circuit breaker is open")

Types

type Levee

type Levee struct {
	// contains filtered or unexported fields
}

func NewLevee

func NewLevee(slo SLO) *Levee

func RestoreState

func RestoreState(s *LeveeState) *Levee

func (*Levee) Call

func (l *Levee) Call(f func() error) (StateChange, error)

func (*Levee) Fail

func (l *Levee) Fail(ts time.Time, duration time.Duration) StateChange

func (*Levee) SaveState

func (l *Levee) SaveState() (*LeveeState, error)

func (*Levee) Start

func (l *Levee) Start(ts time.Time) (StateChange, error)

func (*Levee) State

func (l *Levee) State() State

func (*Levee) Success

func (l *Levee) Success(ts time.Time, duration time.Duration) StateChange

type LeveeState

type LeveeState struct {
	SLO              SLO     `json:"slo"`
	StateVal         uint8   `json:"state"`
	StateEnteredAtNS int64   `json:"state_entered_at_ns"`
	ErrEWMA          float64 `json:"err_ewma"`
	ErrLastTSNS      int64   `json:"err_last_ts_ns"`
	Initialized      bool    `json:"initialized"`
	Samples          int64   `json:"samples"`
	ConsecFails      int     `json:"consec_fails"`
	Inflight         int64   `json:"inflight"`
	InflightLimit    float64 `json:"inflight_limit"`
	Goodput          float64 `json:"goodput"`
	AvgLatency       float64 `json:"avg_latency"`
	LastSuccessTSNS  int64   `json:"last_success_ts_ns"`
	LastEvalTSNS     int64   `json:"last_eval_ts_ns"`
	EvalSuccesses    int64   `json:"eval_successes"`
	EvalFailures     int64   `json:"eval_failures"`
	OpenStreak       int     `json:"open_streak"`
}

type SLO

type SLO struct {
	SuccessRate float64
	Timeout     time.Duration
}

type State

type State uint8

const (
	CLOSED    State = iota
	OPEN            // Tripped: block all traffic, wait for cooldown
	THROTTLED       // SLO breach: inflight-limited admission, MIMD toward capacity
	HALF_OPEN       // Post-OPEN probe: conservative admission, adaptive eval interval
)

type StateChange

type StateChange struct {
	State   State
	Trigger Trigger
}

type Trigger

type Trigger error