floodgate

package module
v0.0.0-...-ac52e68 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Nov 13, 2026 License: Apache-2.0 Imports: 7 Imported by: 0

README

Floodgate 🌊

HTTP Circuit Breaker for Go

Floodgate is a circuit breaker for HTTP requests. It tracks host status and blocks requests to hosts that that are known to be unavailable, with an automatic recovery that periodically checks if unavailable hosts have recovered.

gate := floodgate.New(store)

decision, err := gate.Check(ctx, "remote.example")

switch decision.Action {
case floodgate.ActionSend:  // deliver, then report gate.Success or gate.Failure
case floodgate.ActionRetry: // the host is unhealthy; fail fast and retry at decision.RetryAt
case floodgate.ActionWait:  // the host asked for fewer requests; reschedule for decision.RetryAt
case floodgate.ActionDrop:  // the host is dormant or paused; abandon the delivery
}

States

  • Healthy: deliveries are allowed.
  • Unhealthy: deliveries are held back until a cooldown passes.
  • Testing: the cooldown has passed, and trial deliveries are let through. A success makes the host healthy; a failure makes it unhealthy again for twice as long.
  • Dormant: the host has failed on seven separate days. Its deliveries are abandoned, and it is checked on a slow schedule. This is a separate mark (Host.IsDormant), not a value that Host.State returns.

Sensible Default Rules

These rules can be modified using With... options

  • Rate limits: RateLimited pauses a host until the time it asked for, capped at one hour, or for one minute when it named none.
  • Unhealthy hosts: after five consecutive failures, a host is marked unhealthy for one minute, doubling for up to 32 minutes at a time.
  • Dormant hosts: a host that fails on seven separate days with no success between them is marked dormant, and its deliveries are abandoned. It is checked again weekly for four checks, then every 30 days. Revive clears the mark.
  • Operator pause: SetPaused stops deliveries to a host until it is resumed.

Using it with remote

The remotegate subpackage connects a Gate to requests made with remote. Its option asks before the request is sent, refuses the request when it must be held back, and records the outcome afterwards. The caller never reports to the Gate.

err := remote.Post(inboxURL).
	With(remotegate.Option(gate)). // build a new one for every request
	Body(body).
	Send()

A refused request is never sent. Its error is a remotegate.Refusal, which carries a status code and a wait, so code written for a server's own answers handles the Gate's too:

The host is Code Wait
rate-limited 429 until the pause ends
unhealthy 503 until the cooldown ends
dormant or paused 423 none

derp.IsTooManyRequests(err) therefore returns the Gate's wait for a rate-limited host, the same way it returns a server's own Retry-After.

Reporting Failures

The remotegate option does this reporting for you. Without it, callers report failures to floodgate after each transaction. Report Failure when the host could not be reached or answered with a server error. Report RateLimited for a 429. Report neither for a refusal that says nothing about the host's health, such as a 401 or a 404.

Storage

Floodgate keeps no state of its own. The application implements Store over a database that all of its servers share, and each write must be one atomic operation on the host's record. MemoryStore is for tests and single-process tools.

A healthy host costs no record and no write: Success writes only when the record shows earlier trouble.

Testing a Store

The storetest package checks a Store of your own against MemoryStore, which is the reference. storetest.Run turns a slice of bytes into a script of Store operations and returns a Snapshot of what the Store holds afterwards. Run the same script on both Stores and compare the two Snapshots; a native fuzz test can generate the scripts.

expected, _ := storetest.Run(ctx, floodgate.NewMemoryStore(), script)
actual, _ := storetest.Run(ctx, myStore, script) // myStore must start empty
require.Equal(t, expected, actual)

Pull Requests Welcome

This library is a work in progress, and will benefit from your experience reports, use cases, and contributions. If you have an idea for making this library better, send in a pull request. We're all in this together! 🌊

Documentation

Overview

Package floodgate is a circuit breaker for outbound deliveries, keyed by remote host.

It answers one question before each delivery: may this server send to that host right now? The answer comes from a small record of how recent deliveries to the host went. A host that keeps failing is unhealthy for a cooldown, then tested, and marked dormant when it has failed on enough separate days. A host that asks for fewer requests is paused until the time it names.

The names describe the remote host, not the gate: healthy, unhealthy, testing, and dormant. The circuit breaker terms "open" and "closed" are not used, because they read both ways.

The package keeps no state of its own. Every record lives behind the Store interface, which the application implements over a database that all of its servers share, so that every server sees the same host the same way. MemoryStore is for tests and single-process tools.

Index

Examples

Constants

View Source
const ActionDrop = "DROP"

ActionDrop means that the host is dormant or paused, so the delivery is abandoned

View Source
const ActionRetry = "RETRY"

ActionRetry means that the host is unhealthy, so the delivery fails fast and is retried later

View Source
const ActionSend = "SEND"

ActionSend means that the delivery may go ahead

View Source
const ActionWait = "WAIT"

ActionWait means that the host asked for fewer requests, so the delivery is rescheduled

View Source
const StateHealthy = "HEALTHY"

StateHealthy means that deliveries to the host are allowed

View Source
const StateTesting = "TESTING"

StateTesting means that the cooldown has passed, and trial deliveries are let through

View Source
const StateUnhealthy = "UNHEALTHY"

StateUnhealthy means that deliveries to the host are held back until its cooldown passes

Variables

This section is empty.

Functions

This section is empty.

Types

type Blocklist

type Blocklist map[string]struct{}

Blocklist is the set of hosts that nothing should be queued for, held in memory for one fan-out

func (Blocklist) Contains

func (blocklist Blocklist) Contains(host string) bool

Contains returns TRUE if the host is dormant or paused

type Decision

type Decision struct {
	Action  string    // What to do with the delivery: ActionSend, ActionRetry, ActionWait, or ActionDrop
	RetryAt time.Time // For ActionWait and ActionRetry, when to try again
	Host    Host      // The record that the decision was made from, to hand back with the result
}

Decision is the answer to "may this server deliver to that host right now?"

func (Decision) CanSend

func (decision Decision) CanSend() bool

CanSend returns TRUE if the delivery may go ahead

type Gate

type Gate struct {
	// contains filtered or unexported fields
}

Gate decides whether deliveries may go to a remote host, and records how each one went

func New

func New(store Store, options ...GateOption) Gate

New returns a Gate that keeps its records in the provided Store

func (Gate) Blocked

func (gate Gate) Blocked(ctx context.Context) (Blocklist, error)

Blocked returns the hosts that nothing should be queued for, read from the Store once. A fan-out loads it at the start and checks every recipient against it in memory.

Example

ExampleGate_Blocked shows a fan-out skipping the hosts that nothing should be queued for

ctx := context.Background()
gate := New(NewMemoryStore())

// The server's operator stops deliveries to one host
_ = gate.SetPaused(ctx, "paused.example", true)

// Load the list once, then check every recipient against it in memory
blocked, _ := gate.Blocked(ctx)

for _, host := range []string{"paused.example", "remote.example"} {
	fmt.Println(host, blocked.Contains(host))
}
Output:
paused.example true
remote.example false

func (Gate) Check

func (gate Gate) Check(ctx context.Context, host string) (Decision, error)

Check decides what to do with one delivery to a host, at the moment it is about to be sent

Example

ExampleGate_Check shows the whole life of one delivery: ask, send, and report how it went

ctx := context.Background()

// In production, the Store is a database that all of your servers share
gate := New(NewMemoryStore())

// Ask before each delivery
decision, err := gate.Check(ctx, "remote.example")

if err != nil {
	fmt.Println(err)
	return
}

fmt.Println(decision.Action)

// Send the delivery, then report the result. Hand back the record from the Decision,
// so that a host with no trouble on record costs no write.
if decision.CanSend() {
	_ = gate.Success(ctx, decision.Host)
}
Output:
SEND

func (Gate) Failure

func (gate Gate) Failure(ctx context.Context, host string) error

Failure records that the host could not be reached or answered with a server error, and marks it unhealthy or dormant when that is due. Never call it for a rate limit or a refusal.

Example

ExampleGate_Failure shows a host becoming unhealthy after five failures in a row

ctx := context.Background()
gate := New(NewMemoryStore(), WithJitter(0), WithClock(func() time.Time { return exampleStart }))

// Report each delivery that could not connect, timed out, or got a server error
for range 5 {
	_ = gate.Failure(ctx, "remote.example")
}

// The next delivery fails fast, without connecting, and is retried after the cooldown
decision, _ := gate.Check(ctx, "remote.example")

fmt.Println(decision.Action)
fmt.Println(decision.Host.State(exampleStart))
fmt.Println(decision.RetryAt.Format(time.RFC3339))
Output:
RETRY
UNHEALTHY
2026-11-11T12:01:00Z

func (Gate) Now

func (gate Gate) Now() time.Time

Now returns the current time as this Gate reads it, which a test may have replaced

func (Gate) ProbeFailed

func (gate Gate) ProbeFailed(ctx context.Context, record Host) error

ProbeFailed records that a check of a dormant host failed, and schedules the next one

func (Gate) ProbeSucceeded

func (gate Gate) ProbeSucceeded(ctx context.Context, host string, software string, version string) error

ProbeSucceeded records that a check of a dormant host succeeded, along with the software that the host reported

func (Gate) ProbesDue

func (gate Gate) ProbesDue(ctx context.Context) (iter.Seq[Host], error)

ProbesDue returns the dormant hosts that should be checked now

Example

ExampleGate_ProbesDue shows a scheduled task checking whether a dormant host has come back

ctx := context.Background()
now := exampleStart

// For this example, one day of failure makes a host dormant. The default is seven.
gate := New(NewMemoryStore(), WithDormantDays(1), WithClock(func() time.Time { return now }))
_ = gate.Failure(ctx, "remote.example")

// A dormant host gets nothing
decision, _ := gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)

// Eight days later, the first weekly check is due
now = now.Add(8 * 24 * time.Hour)
hosts, _ := gate.ProbesDue(ctx)

for host := range hosts {

	fmt.Println("checking", host.Host)

	// Contact the host yourself, then report what happened: ProbeFailed or ProbeSucceeded
	_ = gate.ProbeSucceeded(ctx, host.Host, "mastodon", "4.5.0")
}

// The host is healthy again
decision, _ = gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)
Output:
DROP
checking remote.example
SEND

func (Gate) RateLimited

func (gate Gate) RateLimited(ctx context.Context, host string, retryAfter time.Duration) (time.Time, error)

RateLimited records that the host asked for fewer requests, and returns the time until which it is paused. A wait of zero or less means that the host named none.

Example

ExampleGate_RateLimited shows a host that answered "429 Too Many Requests"

ctx := context.Background()
gate := New(NewMemoryStore(), WithJitter(0), WithClock(func() time.Time { return exampleStart }))

// Pass the wait that the host asked for, or zero if it named none
until, _ := gate.RateLimited(ctx, "remote.example", 10*time.Minute)
fmt.Println(until.Format(time.RFC3339))

// Every delivery to that host now waits, on every server that shares the Store
decision, _ := gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)

// No host can ask for more than the longest pause, which is one hour by default
until, _ = gate.RateLimited(ctx, "remote.example", 6*time.Hour)
fmt.Println(until.Format(time.RFC3339))
Output:
2026-11-11T12:10:00Z
WAIT
2026-11-11T13:00:00Z

func (Gate) Revive

func (gate Gate) Revive(ctx context.Context, host string) error

Revive marks a host healthy and clears its history. Call it when a delivery or a check succeeds after trouble, or when the host sends a request of its own.

func (Gate) SetPaused

func (gate Gate) SetPaused(ctx context.Context, host string, isPaused bool) error

SetPaused records whether the server operator has stopped deliveries to a host

func (Gate) SetSignatureFormat

func (gate Gate) SetSignatureFormat(ctx context.Context, host string, format string) error

SetSignatureFormat records the HTTP signature format that a host accepts

func (Gate) Success

func (gate Gate) Success(ctx context.Context, record Host) error

Success records that a delivery reached the host. It takes the record from the Decision, and writes nothing for a host with no history of trouble.

type GateOption

type GateOption func(*Gate)

GateOption changes one setting of a Gate

func WithClock

func WithClock(now func() time.Time) GateOption

WithClock replaces the clock, so that a test can choose the time

func WithCooldown

func WithCooldown(first time.Duration, longest time.Duration) GateOption

WithCooldown sets how long a host stays unhealthy before its first test, and the longest it waits after its cooldown has doubled

func WithDormantDays

func WithDormantDays(days int) GateOption

WithDormantDays sets the number of separate days of failure that marks a host dormant

func WithFailureThreshold

func WithFailureThreshold(count int) GateOption

WithFailureThreshold sets the number of consecutive failures that makes a host unhealthy

func WithJitter

func WithJitter(longest time.Duration) GateOption

WithJitter sets the longest random delay added to a rescheduled delivery. Zero adds none.

func WithProbeSchedule

func WithProbeSchedule(early time.Duration, earlyCount int, late time.Duration) GateOption

WithProbeSchedule sets the wait between the first `earlyCount` checks of a dormant host, and the wait between every check after those

func WithRateLimit

func WithRateLimit(fallback time.Duration, longest time.Duration) GateOption

WithRateLimit sets the pause used when a host names none, and the longest pause it can ask for

type Host

type Host struct {
	Host            string   `json:"host"              bson:"_id"`             // Hostname of the remote server, and the record's key
	IsUnhealthy     bool     `json:"isUnhealthy"       bson:"isUnhealthy"`     // TRUE while deliveries are held back
	TestAfter       int64    `json:"testAfter"         bson:"testAfter"`       // Unix seconds when an unhealthy host starts testing
	Cooldown        int64    `json:"cooldown"          bson:"cooldown"`        // Seconds that the host last waited before testing
	FailureCount    int      `json:"failureCount"      bson:"failureCount"`    // Consecutive failures since the last success
	FailureDates    []string `json:"failureDates"      bson:"failureDates"`    // Separate days (UTC) with a failure since the last success
	DormantSince    int64    `json:"dormantSince"      bson:"dormantSince"`    // Unix seconds when the host was marked dormant, or zero
	LastRevived     int64    `json:"lastRevived"       bson:"lastRevived"`     // Unix seconds when a success last cleared a history of trouble
	LastFailure     int64    `json:"lastFailure"       bson:"lastFailure"`     // Unix seconds of the last failed delivery
	IsPaused        bool     `json:"isPaused"          bson:"isPaused"`        // TRUE if the server operator has stopped deliveries to this host
	RateLimitUntil  int64    `json:"rateLimitUntil"    bson:"rateLimitUntil"`  // Unix seconds until which the host asked for no requests
	Software        string   `json:"software"          bson:"software"`        // Server software reported by NodeInfo
	Version         string   `json:"version"           bson:"version"`         // Software version reported by NodeInfo
	SignatureFormat string   `json:"signatureFormat"   bson:"signatureFormat"` // HTTP signature format that the host accepts, once known
	NextProbe       int64    `json:"nextProbe"         bson:"nextProbe"`       // Unix seconds when a dormant host is next checked
	ProbeCount      int      `json:"probeCount"        bson:"probeCount"`      // Number of checks made since the host became dormant
}

Host is the record of how deliveries to one remote host have gone. A host with no record is healthy, so the zero value describes a healthy host.

func (Host) IsBlocked

func (host Host) IsBlocked() bool

IsBlocked returns TRUE if nothing should be queued for the host at all

func (Host) IsDormant

func (host Host) IsDormant() bool

IsDormant returns TRUE if the host has failed for long enough that deliveries are abandoned

func (Host) IsRateLimited

func (host Host) IsRateLimited(now time.Time) bool

IsRateLimited returns TRUE if the host has asked for no requests until a time after the one given

func (Host) State

func (host Host) State(now time.Time) string

State returns the host's state at the given time: healthy, unhealthy, or testing

Example

ExampleHost_State shows the three states that a host's record can be in

// A host with no record, or with no trouble on record, is healthy
fmt.Println(Host{}.State(exampleStart))

// An unhealthy host is left alone until its cooldown passes, and is then tested
record := Host{IsUnhealthy: true, TestAfter: exampleStart.Add(time.Minute).Unix()}

fmt.Println(record.State(exampleStart))
fmt.Println(record.State(exampleStart.Add(2 * time.Minute)))
Output:
HEALTHY
UNHEALTHY
TESTING

type MemoryStore

type MemoryStore struct {
	// contains filtered or unexported fields
}

MemoryStore is a Store that keeps its records in one process's memory. It is for tests and single-process tools: servers that share work must share a Store that they can all reach.

func NewMemoryStore

func NewMemoryStore() *MemoryStore

NewMemoryStore returns an empty MemoryStore

func (*MemoryStore) Load

func (store *MemoryStore) Load(_ context.Context, host string) (Host, error)

Load returns the record for a host, or a zero Host when there is none

func (*MemoryStore) MarkDormant

func (store *MemoryStore) MarkDormant(_ context.Context, host string, since time.Time, nextProbe time.Time) error

MarkDormant marks the host dormant, unless it already is or it has no record

func (*MemoryStore) MarkUnhealthy

func (store *MemoryStore) MarkUnhealthy(_ context.Context, host string, testAfter time.Time, cooldown time.Duration) error

MarkUnhealthy holds deliveries back until the given time. It writes nothing when the host has no record.

func (*MemoryStore) RangeBlocked

func (store *MemoryStore) RangeBlocked(_ context.Context) (iter.Seq[Host], error)

RangeBlocked returns every host that is dormant or paused

func (*MemoryStore) RangeProbesDue

func (store *MemoryStore) RangeProbesDue(_ context.Context, now time.Time) (iter.Seq[Host], error)

RangeProbesDue returns every dormant host whose next check is at or before the given time

func (*MemoryStore) RecordFailure

func (store *MemoryStore) RecordFailure(_ context.Context, host string, now time.Time, date string) (Host, error)

RecordFailure counts one failure and its day, and returns the record afterwards

func (*MemoryStore) RecordProbeFailure

func (store *MemoryStore) RecordProbeFailure(_ context.Context, host string, nextProbe time.Time) error

RecordProbeFailure counts one failed check and schedules the next. It writes nothing when the host has no record.

func (*MemoryStore) RecordSuccess

func (store *MemoryStore) RecordSuccess(_ context.Context, host string, now time.Time) error

RecordSuccess marks the host healthy and clears the failure history of a host that has a record

func (*MemoryStore) SetNodeInfo

func (store *MemoryStore) SetNodeInfo(_ context.Context, host string, software string, version string) error

SetNodeInfo records the software that the host runs

func (*MemoryStore) SetPaused

func (store *MemoryStore) SetPaused(_ context.Context, host string, isPaused bool) error

SetPaused records whether the server operator has stopped deliveries to the host

func (*MemoryStore) SetRateLimit

func (store *MemoryStore) SetRateLimit(_ context.Context, host string, until time.Time) error

SetRateLimit pauses the host until the given time, and never shortens a pause

func (*MemoryStore) SetSignatureFormat

func (store *MemoryStore) SetSignatureFormat(_ context.Context, host string, format string) error

SetSignatureFormat records the HTTP signature format that the host accepts

type Store

type Store interface {

	// Load returns the record for a host. A host with no record yields a zero Host and no error.
	Load(ctx context.Context, host string) (Host, error)

	// RangeBlocked returns every host that is dormant or paused
	RangeBlocked(ctx context.Context) (iter.Seq[Host], error)

	// RangeProbesDue returns every dormant host whose next check is at or before the given time
	RangeProbesDue(ctx context.Context, now time.Time) (iter.Seq[Host], error)

	// RecordSuccess marks the host healthy and clears the failure history, the dormant mark and
	// the probe schedule. It writes nothing when the host has no record.
	RecordSuccess(ctx context.Context, host string, now time.Time) error

	// RecordFailure adds one to the failure count, adds the day to the failure dates if it is
	// new, creates the record if there is none, and returns the record as it stands afterwards
	RecordFailure(ctx context.Context, host string, now time.Time, date string) (Host, error)

	// MarkUnhealthy holds deliveries back until the given time, and records the cooldown that was
	// used. It writes nothing when the host has no record.
	MarkUnhealthy(ctx context.Context, host string, testAfter time.Time, cooldown time.Duration) error

	// MarkDormant marks the host dormant and schedules its first check. It changes nothing
	// when the host is already marked, or has no record.
	MarkDormant(ctx context.Context, host string, since time.Time, nextProbe time.Time) error

	// RecordProbeFailure adds one to the probe count and schedules the next check. It writes
	// nothing when the host has no record.
	RecordProbeFailure(ctx context.Context, host string, nextProbe time.Time) error

	// SetRateLimit pauses the host until the given time, creating the record if there is none.
	// It never shortens a pause: the later of the stored time and the new time is kept.
	SetRateLimit(ctx context.Context, host string, until time.Time) error

	// SetPaused records whether the server operator has stopped deliveries to the host,
	// creating the record if there is none
	SetPaused(ctx context.Context, host string, isPaused bool) error

	// SetNodeInfo records the software that the host runs, creating the record if there is none
	SetNodeInfo(ctx context.Context, host string, software string, version string) error

	// SetSignatureFormat records the HTTP signature format that the host accepts, creating the
	// record if there is none
	SetSignatureFormat(ctx context.Context, host string, format string) error
}

Store keeps the Host records. Every server of an application shares one Store, so each write must be one atomic operation on one record, or two reports at once lose a count.

Directories

Path Synopsis
Package remotegate connects a floodgate.Gate to requests made with the benpate/remote package.
Package remotegate connects a floodgate.Gate to requests made with the benpate/remote package.
Package storetest checks that an implementation of floodgate.Store behaves like the others.
Package storetest checks that an implementation of floodgate.Store behaves like the others.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL