brownout

README

brownout

Go HTTP middleware for browning out endpoints at runtime. You can shed some or all traffic to selected routes without restarting or redeploying the server.

Use it when a downstream dependency is struggling and you want to turn off an expensive endpoint, cut a noisy route to 10% of its traffic, or make writes read-only for a while. When the incident is over, you remove the rule.

• Plain net/http middleware, so it works with any router built on http.Handler
• Lock-free request path: rules are swapped atomically and adding one costs nothing on unmatched routes
• Partial brownouts: let a set percentage of traffic through
• Rules can auto-expire, so a forgotten brownout doesn't stay on forever
• Three ways to change rules: Go API, built-in JSON admin API, or a watched JSON file (works well with Kubernetes ConfigMaps)
• No dependencies outside the standard library

go get github.com/rossedman/brownout

Quick start

c := brownout.New()

// Wrap your app.
go http.ListenAndServe(":8080", c.Middleware(appHandler))

// Expose the admin API on an internal-only listener.
go http.ListenAndServe("127.0.0.1:9090", c.AdminHandler())

Then, while the server is running:

# Shed all traffic to /reports/* for 10 minutes
curl -X PUT localhost:9090/rules/reports \
  -d '{"path": "/reports/*", "ttl": "10m", "retry_after_seconds": 60}'

# Let only 20% of searches through
curl -X PUT localhost:9090/rules/search -d '{"path": "/search", "allow_percent": 20}'

# Make the API read-only
curl -X PUT localhost:9090/rules/readonly \
  -d '{"path": "/api/*", "methods": ["POST","PUT","PATCH","DELETE"]}'

# See what's active, then restore
curl localhost:9090/rules
curl -X DELETE localhost:9090/rules/reports

📖 Docs and an interactive rule playground: https://rossedman.github.io/brownout/

Examples

Each example is a small runnable server (go run ./examples/<name>). The curl commands to try it are at the top of each file.

Example	Shows
basic	The middleware plus the admin API on a separate internal port
admin-auth	App and admin API on one port, with bearer-token auth and the admin API mounted outside the middleware
filewatch	Rules loaded from a JSON file and reloaded when it changes (Kubernetes ConfigMap pattern)
auto	Automatically browning out an endpoint when its dependency fails its health checks, and restoring it on recovery
degrade	Using Check to serve a cheaper response instead of a 503
metrics	Counting shed requests per rule with WithOnShed, and JSON error bodies with WithResponder

There are also runnable snippets in example_test.go, which appear on pkg.go.dev.

Rules

Field	JSON	Description
ID	id	Unique name. Setting a rule with an existing ID replaces it.
Path	path	Exact path, or a prefix if it ends in * (/api/*). * alone matches everything.
Methods	methods	Optional. Limits the rule to these HTTP methods.
AllowPercent	allow_percent	Percentage (0–100) of matching requests still allowed through. The default of 0 sheds everything.
StatusCode	status_code	Response status for shed requests. Default 503.
RetryAfterSeconds	retry_after_seconds	Sets the Retry-After header.
Message	message	Response body. Defaults to the status text.
ExpiresAt	expires_at	RFC 3339 time after which the rule stops applying. In the admin API you can send "ttl": "15m" instead.

Rules are checked in order and the first match decides. If the first matching rule lets a request through (for example, because of allow_percent), later rules are not checked. You can use this to carve out exceptions:

c.Set(brownout.Rule{ID: "health", Path: "/api/health", AllowPercent: 100}) // always allowed
c.Set(brownout.Rule{ID: "api", Path: "/api/*"})                            // everything else shed

Changing rules

From Go (for example, from a feature-flag callback or a health check that watches a dependency):

c.Set(rule)           // add or update one rule
c.Remove("reports")   // remove one rule
c.Replace(rules)      // swap the whole set atomically (all-or-nothing validation)
c.Clear()             // remove all rules
c.Rules()             // list active rules

Admin API: c.AdminHandler(). Mount it under a prefix with http.StripPrefix:

Method	Path
GET	/rules	List active rules: {"rules": [...]}
PUT	/rules	Replace all rules: {"rules": [...]}
DELETE	/rules	Remove all rules
GET	/rules/{id}	Get one rule
PUT	/rules/{id}	Create or update one rule
DELETE	/rules/{id}	Remove one rule

⚠️ The admin handler has no authentication. Serve it on an internal port or put your own auth middleware in front of it.

Watched file: poll a JSON file and reload it when its contents change. If a reload fails (for example, the file is invalid JSON or only half-written), the current rules stay in place.

go c.WatchFile(ctx, "/etc/brownout/rules.json", 5*time.Second, func(err error) {
	log.Printf("brownout: %v", err)
})

{ "rules": [ { "id": "reports", "path": "/reports/*", "retry_after_seconds": 60 } ] }

Customizing

c := brownout.New(
	brownout.WithDefaultStatus(http.StatusTooManyRequests),
	brownout.WithOnShed(func(r *http.Request, rule brownout.Rule) {
		shedCounter.WithLabelValues(rule.ID).Inc()
	}),
	brownout.WithResponder(func(w http.ResponseWriter, r *http.Request, rule brownout.Rule) {
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(rule.StatusCode)
		json.NewEncoder(w).Encode(map[string]string{"error": "temporarily unavailable"})
	}),
)

To make the decision yourself (for example, to degrade a response instead of rejecting it), call c.Check(r), which returns the matching rule and whether the request should be shed.

Performance

The request path takes no locks and makes no allocations unless a request is shed. It reads the clock only when a request matches a rule that has an expiry. Rule evaluation is linear in the number of rules, about 12ns per rule that is checked. Representative numbers from go test -bench . -benchmem on a 4-vCPU Xeon VM:

Scenario	Time/request	Allocs
No rules installed	~24 ns	0
1 rule, no match	~37 ns	0
10 rules, none match	~160 ns	0
100 rules, none match	~1.2 µs	0
Matched, shed with the default 503 response	~370 ns	3
10 rules, 4 goroutines in parallel, rules changing concurrently	~85 ns	0

Run them yourself with:

go test -run '^$' -bench . -benchmem ./...

CI runs every benchmark on each push and pull request. Pull requests get a benchstat comparison against the base branch in the job summary. TestHotPathAllocations fails the build if the request path starts allocating.

Multiple instances

Rules live in memory for each process. When you run several replicas, either use WatchFile with a shared file such as a ConfigMap, or call the admin API on each instance.

License

MIT

Documentation

Overview

Package brownout provides HTTP middleware for shedding traffic to selected endpoints at runtime ("browning out") without restarting the server.

A Controller holds a set of Rules. Each Rule matches requests by method and path and decides what fraction of matching traffic is allowed through; the rest is rejected with a configurable status code (503 by default). Rules can be changed at any time — programmatically, through the JSON admin API returned by AdminHandler, or by watching a file with WatchFile — and the change takes effect on the very next request.

c := brownout.New()
mux := http.NewServeMux()
mux.Handle("/admin/brownout/", http.StripPrefix("/admin/brownout", c.AdminHandler()))
http.ListenAndServe(":8080", c.Middleware(appHandler))

// Later, from anywhere:
c.Set(brownout.Rule{ID: "reports", Path: "/reports/*"})

Example

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/rossedman/brownout"
)

func get(h http.Handler, method, path string) int {
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, httptest.NewRequest(method, path, nil))
	return rec.Code
}

func main() {
	c := brownout.New()
	app := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {})
	h := c.Middleware(app)

	fmt.Println(get(h, "GET", "/reports/q3"))

	// Brown out every path under /reports/ without restarting.
	_ = c.Set(brownout.Rule{ID: "reports", Path: "/reports/*"})
	fmt.Println(get(h, "GET", "/reports/q3"))
	fmt.Println(get(h, "GET", "/users/1"))

	// And bring it back.
	c.Remove("reports")
	fmt.Println(get(h, "GET", "/reports/q3"))
}

Output:
200
503
200
200

Index

• type Controller
  • func New(opts ...Option) *Controller
  • func (c *Controller) AdminHandler() http.Handler
  • func (c *Controller) Check(req *http.Request) (Rule, bool)
  • func (c *Controller) Clear()
  • func (c *Controller) LoadFile(path string) error
  • func (c *Controller) Middleware(next http.Handler) http.Handler
  • func (c *Controller) Remove(id string) bool
  • func (c *Controller) Replace(rules []Rule) error
  • func (c *Controller) Rule(id string) (Rule, bool)
  • func (c *Controller) Rules() []Rule
  • func (c *Controller) Set(rule Rule) error
  • func (c *Controller) WatchFile(ctx context.Context, path string, interval time.Duration, onError func(error)) error
• type Option
  • func WithClock(now func() time.Time) Option
  • func WithDefaultStatus(code int) Option
  • func WithOnShed(fn func(req *http.Request, rule Rule)) Option
  • func WithResponder(fn Responder) Option
• type Responder
• type Rule
  • func (r Rule) Matches(req *http.Request, now time.Time) bool
  • func (r Rule) Validate() error

Examples

• Package
• Controller.AdminHandler
• Controller.Check
• Controller.Set (Exception)
• Rule (Methods)
• WithResponder

Types

type Controller

type Controller struct {
	// contains filtered or unexported fields
}

Controller holds the active brownout rules and produces middleware that enforces them. It is safe for concurrent use; the request path is lock-free.

func New

func New(opts ...Option) *Controller

New returns a Controller with no rules.

func (*Controller) AdminHandler

func (c *Controller) AdminHandler() http.Handler

AdminHandler returns an http.Handler exposing a JSON API for managing rules at runtime. Mount it under a prefix with http.StripPrefix:

GET    /rules        list active rules        → {"rules": [...]}
PUT    /rules        replace all rules        ← {"rules": [...]}
DELETE /rules        remove all rules
GET    /rules/{id}   fetch one rule
PUT    /rules/{id}   create or update a rule  ← {"path": "/x/*", ...}
DELETE /rules/{id}   remove a rule

Rule bodies may include "ttl" (e.g. "10m") instead of "expires_at".

The handler performs no authentication or authorization. Do not expose it publicly: serve it on an internal listener or wrap it with your own auth.

Example

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"
	"strings"

	"github.com/rossedman/brownout"
)

func main() {
	c := brownout.New()
	admin := http.StripPrefix("/admin/brownout", c.AdminHandler())

	body := `{"path": "/search", "allow_percent": 20, "ttl": "15m"}`
	rec := httptest.NewRecorder()
	admin.ServeHTTP(rec, httptest.NewRequest("PUT", "/admin/brownout/rules/search", strings.NewReader(body)))
	fmt.Println(rec.Code)

	r, _ := c.Rule("search")
	fmt.Println(r.Path, r.AllowPercent)
}

Output:
200
/search 20

func (*Controller) Check

func (c *Controller) Check(req *http.Request) (Rule, bool)

Check reports whether req should be shed and, if so, the rule responsible. Rules are evaluated in order and the first match decides: if it lets the request through (per AllowPercent), later rules are not consulted.

Example

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/rossedman/brownout"
)

func main() {
	c := brownout.New()
	_ = c.Set(brownout.Rule{ID: "personalization", Path: "/products/*"})

	handler := func(w http.ResponseWriter, r *http.Request) {
		if rule, degraded := c.Check(r); degraded {
			fmt.Fprintf(w, "generic page (degraded by %s)", rule.ID)
			return
		}
		fmt.Fprint(w, "personalized page")
	}

	rec := httptest.NewRecorder()
	handler(rec, httptest.NewRequest("GET", "/products/7", nil))
	fmt.Println(rec.Code, rec.Body)
}

Output:
200 generic page (degraded by personalization)

func (*Controller) Clear

func (c *Controller) Clear()

Clear removes all rules.

func (*Controller) LoadFile

func (c *Controller) LoadFile(path string) error

LoadFile replaces the controller's rules with those in a JSON file of the form {"rules": [...]}, using the same rule fields as the admin API (except "ttl"; use "expires_at" for an absolute deadline).

func (*Controller) Middleware

func (c *Controller) Middleware(next http.Handler) http.Handler

Middleware wraps next so that requests matching an active rule are shed.

func (*Controller) Remove

func (c *Controller) Remove(id string) bool

Remove deletes the rule with the given ID, reporting whether it existed.

func (*Controller) Replace

func (c *Controller) Replace(rules []Rule) error

Replace atomically swaps the entire rule set. Either all rules are valid and installed, or an error is returned and nothing changes.

func (*Controller) Rule

func (c *Controller) Rule(id string) (Rule, bool)

Rule returns the active rule with the given ID.

func (*Controller) Rules

func (c *Controller) Rules() []Rule

Rules returns a copy of the current rules, in evaluation order, excluding any that have expired.

func (*Controller) Set

func (c *Controller) Set(rule Rule) error

Set adds a rule, or replaces the existing rule with the same ID while keeping its position in the evaluation order.

Example (Exception)

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/rossedman/brownout"
)

func get(h http.Handler, method, path string) int {
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, httptest.NewRequest(method, path, nil))
	return rec.Code
}

func main() {
	c := brownout.New()
	h := c.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))

	// The first matching rule decides, so an earlier rule that allows 100%
	// of traffic carves an exception out of a later, broader rule.
	_ = c.Set(brownout.Rule{ID: "health", Path: "/api/health", AllowPercent: 100})
	_ = c.Set(brownout.Rule{ID: "api", Path: "/api/*"})

	fmt.Println(get(h, "GET", "/api/health"))
	fmt.Println(get(h, "GET", "/api/orders"))
}

Output:
200
503

func (*Controller) WatchFile

func (c *Controller) WatchFile(ctx context.Context, path string, interval time.Duration, onError func(error)) error

WatchFile loads path and then polls it every interval, reloading the rules whenever its contents change. It blocks until ctx is canceled and returns ctx.Err(), or returns early if the initial load fails.

Errors on later reloads (a missing file or invalid JSON, e.g. mid-write) leave the current rules in place and are passed to onError if non-nil. This works well with Kubernetes ConfigMap volumes, which update files in place without restarting the pod.

type Option

type Option func(*Controller)

Option configures a Controller.

func WithClock

func WithClock(now func() time.Time) Option

WithClock overrides the time source used to evaluate rule expiry. Intended for tests.

func WithDefaultStatus

func WithDefaultStatus(code int) Option

WithDefaultStatus sets the status code used for rules that don't specify one. The default is 503 Service Unavailable.

func WithOnShed

func WithOnShed(fn func(req *http.Request, rule Rule)) Option

WithOnShed registers a callback invoked for every shed request, e.g. to record metrics or logs. It runs synchronously before the response is written, so it should be fast.

func WithResponder

func WithResponder(fn Responder) Option

WithResponder replaces the default response writer for shed requests. The Rule passed to the responder has StatusCode already defaulted.

Example

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/rossedman/brownout"
)

func main() {
	c := brownout.New(brownout.WithResponder(func(w http.ResponseWriter, r *http.Request, rule brownout.Rule) {
		w.Header().Set("Content-Type", "application/json")
		w.WriteHeader(rule.StatusCode)
		fmt.Fprintf(w, `{"error":"brownout","rule":%q}`, rule.ID)
	}))
	_ = c.Set(brownout.Rule{ID: "export", Path: "/export", StatusCode: http.StatusTooManyRequests})

	rec := httptest.NewRecorder()
	c.Middleware(http.NotFoundHandler()).ServeHTTP(rec, httptest.NewRequest("GET", "/export", nil))
	fmt.Println(rec.Code, rec.Body)
}

Output:
429 {"error":"brownout","rule":"export"}

type Responder

type Responder func(w http.ResponseWriter, req *http.Request, rule Rule)

Responder writes the response for a shed request.

type Rule

type Rule struct {
	// ID uniquely identifies the rule. Setting a rule with an existing ID
	// replaces it in place.
	ID string `json:"id"`

	// Path is matched against the request's URL path. A trailing "*" makes
	// it a prefix match ("/api/reports/*" matches "/api/reports/123"); a
	// lone "*" matches every path. Otherwise the match is exact.
	Path string `json:"path"`

	// Methods restricts the rule to the given HTTP methods. Empty matches
	// any method.
	Methods []string `json:"methods,omitempty"`

	// AllowPercent is the percentage (0–100) of matching requests that are
	// still let through. The zero value sheds all matching traffic; 25 lets
	// roughly one in four requests through.
	AllowPercent float64 `json:"allow_percent,omitempty"`

	// StatusCode is the response status for shed requests. Defaults to the
	// Controller's default (503 Service Unavailable unless overridden).
	StatusCode int `json:"status_code,omitempty"`

	// RetryAfterSeconds, when positive, is sent as the Retry-After header.
	RetryAfterSeconds int `json:"retry_after_seconds,omitempty"`

	// Message is written as the response body for shed requests. Defaults
	// to the status text of StatusCode.
	Message string `json:"message,omitempty"`

	// ExpiresAt, when non-zero, automatically disables the rule after the
	// given time.
	ExpiresAt time.Time `json:"expires_at,omitzero"`
}

Rule describes an endpoint (or group of endpoints) to brown out.

Example (Methods)

package main

import (
	"fmt"
	"net/http"
	"net/http/httptest"

	"github.com/rossedman/brownout"
)

func get(h http.Handler, method, path string) int {
	rec := httptest.NewRecorder()
	h.ServeHTTP(rec, httptest.NewRequest(method, path, nil))
	return rec.Code
}

func main() {
	c := brownout.New()
	h := c.Middleware(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {}))

	// Make the API read-only.
	_ = c.Set(brownout.Rule{
		ID:      "readonly",
		Path:    "/api/*",
		Methods: []string{"POST", "PUT", "PATCH", "DELETE"},
	})
	fmt.Println(get(h, "GET", "/api/orders"))
	fmt.Println(get(h, "POST", "/api/orders"))
}

Output:
200
503

func (Rule) Matches

func (r Rule) Matches(req *http.Request, now time.Time) bool

Matches reports whether the rule applies to the request at time now, ignoring AllowPercent.

func (Rule) Validate

func (r Rule) Validate() error

Validate reports whether the rule is well formed.