Documentation
¶
Overview ¶
Package floodgate is a circuit breaker for outbound deliveries, keyed by remote host.
It answers one question before each delivery: may this server send to that host right now? The answer comes from a small record of how recent deliveries to the host went. A host that keeps failing is unhealthy for a cooldown, then tested, and marked dormant when it has failed on enough separate days. A host that asks for fewer requests is paused until the time it names.
The names describe the remote host, not the gate: healthy, unhealthy, testing, and dormant. The circuit breaker terms "open" and "closed" are not used, because they read both ways.
The package keeps no state of its own. Every record lives behind the Store interface, which the application implements over a database that all of its servers share, so that every server sees the same host the same way. MemoryStore is for tests and single-process tools.
Index ¶
- Constants
- type Blocklist
- type Decision
- type Gate
- func (gate Gate) Blocked(ctx context.Context) (Blocklist, error)
- func (gate Gate) Check(ctx context.Context, host string) (Decision, error)
- func (gate Gate) Failure(ctx context.Context, host string) error
- func (gate Gate) Now() time.Time
- func (gate Gate) ProbeFailed(ctx context.Context, record Host) error
- func (gate Gate) ProbeSucceeded(ctx context.Context, host string, software string, version string) error
- func (gate Gate) ProbesDue(ctx context.Context) (iter.Seq[Host], error)
- func (gate Gate) RateLimited(ctx context.Context, host string, retryAfter time.Duration) (time.Time, error)
- func (gate Gate) Revive(ctx context.Context, host string) error
- func (gate Gate) SetPaused(ctx context.Context, host string, isPaused bool) error
- func (gate Gate) SetSignatureFormat(ctx context.Context, host string, format string) error
- func (gate Gate) Success(ctx context.Context, record Host) error
- type GateOption
- func WithClock(now func() time.Time) GateOption
- func WithCooldown(first time.Duration, longest time.Duration) GateOption
- func WithDormantDays(days int) GateOption
- func WithFailureThreshold(count int) GateOption
- func WithJitter(longest time.Duration) GateOption
- func WithProbeSchedule(early time.Duration, earlyCount int, late time.Duration) GateOption
- func WithRateLimit(fallback time.Duration, longest time.Duration) GateOption
- type Host
- type MemoryStore
- func (store *MemoryStore) Load(_ context.Context, host string) (Host, error)
- func (store *MemoryStore) MarkDormant(_ context.Context, host string, since time.Time, nextProbe time.Time) error
- func (store *MemoryStore) MarkUnhealthy(_ context.Context, host string, testAfter time.Time, cooldown time.Duration) error
- func (store *MemoryStore) RangeBlocked(_ context.Context) (iter.Seq[Host], error)
- func (store *MemoryStore) RangeProbesDue(_ context.Context, now time.Time) (iter.Seq[Host], error)
- func (store *MemoryStore) RecordFailure(_ context.Context, host string, now time.Time, date string) (Host, error)
- func (store *MemoryStore) RecordProbeFailure(_ context.Context, host string, nextProbe time.Time) error
- func (store *MemoryStore) RecordSuccess(_ context.Context, host string, now time.Time) error
- func (store *MemoryStore) SetNodeInfo(_ context.Context, host string, software string, version string) error
- func (store *MemoryStore) SetPaused(_ context.Context, host string, isPaused bool) error
- func (store *MemoryStore) SetRateLimit(_ context.Context, host string, until time.Time) error
- func (store *MemoryStore) SetSignatureFormat(_ context.Context, host string, format string) error
- type Store
Examples ¶
Constants ¶
const ActionDrop = "DROP"
ActionDrop means that the host is dormant or paused, so the delivery is abandoned
const ActionRetry = "RETRY"
ActionRetry means that the host is unhealthy, so the delivery fails fast and is retried later
const ActionSend = "SEND"
ActionSend means that the delivery may go ahead
const ActionWait = "WAIT"
ActionWait means that the host asked for fewer requests, so the delivery is rescheduled
const StateHealthy = "HEALTHY"
StateHealthy means that deliveries to the host are allowed
const StateTesting = "TESTING"
StateTesting means that the cooldown has passed, and trial deliveries are let through
const StateUnhealthy = "UNHEALTHY"
StateUnhealthy means that deliveries to the host are held back until its cooldown passes
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type Blocklist ¶
type Blocklist map[string]struct{}
Blocklist is the set of hosts that nothing should be queued for, held in memory for one fan-out
type Decision ¶
type Decision struct {
Action string // What to do with the delivery: ActionSend, ActionRetry, ActionWait, or ActionDrop
RetryAt time.Time // For ActionWait and ActionRetry, when to try again
Host Host // The record that the decision was made from, to hand back with the result
}
Decision is the answer to "may this server deliver to that host right now?"
type Gate ¶
type Gate struct {
// contains filtered or unexported fields
}
Gate decides whether deliveries may go to a remote host, and records how each one went
func New ¶
func New(store Store, options ...GateOption) Gate
New returns a Gate that keeps its records in the provided Store
func (Gate) Blocked ¶
Blocked returns the hosts that nothing should be queued for, read from the Store once. A fan-out loads it at the start and checks every recipient against it in memory.
Example ¶
ExampleGate_Blocked shows a fan-out skipping the hosts that nothing should be queued for
ctx := context.Background()
gate := New(NewMemoryStore())
// The server's operator stops deliveries to one host
_ = gate.SetPaused(ctx, "paused.example", true)
// Load the list once, then check every recipient against it in memory
blocked, _ := gate.Blocked(ctx)
for _, host := range []string{"paused.example", "remote.example"} {
fmt.Println(host, blocked.Contains(host))
}
Output: paused.example true remote.example false
func (Gate) Check ¶
Check decides what to do with one delivery to a host, at the moment it is about to be sent
Example ¶
ExampleGate_Check shows the whole life of one delivery: ask, send, and report how it went
ctx := context.Background()
// In production, the Store is a database that all of your servers share
gate := New(NewMemoryStore())
// Ask before each delivery
decision, err := gate.Check(ctx, "remote.example")
if err != nil {
fmt.Println(err)
return
}
fmt.Println(decision.Action)
// Send the delivery, then report the result. Hand back the record from the Decision,
// so that a host with no trouble on record costs no write.
if decision.CanSend() {
_ = gate.Success(ctx, decision.Host)
}
Output: SEND
func (Gate) Failure ¶
Failure records that the host could not be reached or answered with a server error, and marks it unhealthy or dormant when that is due. Never call it for a rate limit or a refusal.
Example ¶
ExampleGate_Failure shows a host becoming unhealthy after five failures in a row
ctx := context.Background()
gate := New(NewMemoryStore(), WithJitter(0), WithClock(func() time.Time { return exampleStart }))
// Report each delivery that could not connect, timed out, or got a server error
for range 5 {
_ = gate.Failure(ctx, "remote.example")
}
// The next delivery fails fast, without connecting, and is retried after the cooldown
decision, _ := gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)
fmt.Println(decision.Host.State(exampleStart))
fmt.Println(decision.RetryAt.Format(time.RFC3339))
Output: RETRY UNHEALTHY 2026-11-11T12:01:00Z
func (Gate) Now ¶
Now returns the current time as this Gate reads it, which a test may have replaced
func (Gate) ProbeFailed ¶
ProbeFailed records that a check of a dormant host failed, and schedules the next one
func (Gate) ProbeSucceeded ¶
func (gate Gate) ProbeSucceeded(ctx context.Context, host string, software string, version string) error
ProbeSucceeded records that a check of a dormant host succeeded, along with the software that the host reported
func (Gate) ProbesDue ¶
ProbesDue returns the dormant hosts that should be checked now
Example ¶
ExampleGate_ProbesDue shows a scheduled task checking whether a dormant host has come back
ctx := context.Background()
now := exampleStart
// For this example, one day of failure makes a host dormant. The default is seven.
gate := New(NewMemoryStore(), WithDormantDays(1), WithClock(func() time.Time { return now }))
_ = gate.Failure(ctx, "remote.example")
// A dormant host gets nothing
decision, _ := gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)
// Eight days later, the first weekly check is due
now = now.Add(8 * 24 * time.Hour)
hosts, _ := gate.ProbesDue(ctx)
for host := range hosts {
fmt.Println("checking", host.Host)
// Contact the host yourself, then report what happened: ProbeFailed or ProbeSucceeded
_ = gate.ProbeSucceeded(ctx, host.Host, "mastodon", "4.5.0")
}
// The host is healthy again
decision, _ = gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)
Output: DROP checking remote.example SEND
func (Gate) RateLimited ¶
func (gate Gate) RateLimited(ctx context.Context, host string, retryAfter time.Duration) (time.Time, error)
RateLimited records that the host asked for fewer requests, and returns the time until which it is paused. A wait of zero or less means that the host named none.
Example ¶
ExampleGate_RateLimited shows a host that answered "429 Too Many Requests"
ctx := context.Background()
gate := New(NewMemoryStore(), WithJitter(0), WithClock(func() time.Time { return exampleStart }))
// Pass the wait that the host asked for, or zero if it named none
until, _ := gate.RateLimited(ctx, "remote.example", 10*time.Minute)
fmt.Println(until.Format(time.RFC3339))
// Every delivery to that host now waits, on every server that shares the Store
decision, _ := gate.Check(ctx, "remote.example")
fmt.Println(decision.Action)
// No host can ask for more than the longest pause, which is one hour by default
until, _ = gate.RateLimited(ctx, "remote.example", 6*time.Hour)
fmt.Println(until.Format(time.RFC3339))
Output: 2026-11-11T12:10:00Z WAIT 2026-11-11T13:00:00Z
func (Gate) Revive ¶
Revive marks a host healthy and clears its history. Call it when a delivery or a check succeeds after trouble, or when the host sends a request of its own.
func (Gate) SetPaused ¶
SetPaused records whether the server operator has stopped deliveries to a host
func (Gate) SetSignatureFormat ¶
SetSignatureFormat records the HTTP signature format that a host accepts
type GateOption ¶
type GateOption func(*Gate)
GateOption changes one setting of a Gate
func WithClock ¶
func WithClock(now func() time.Time) GateOption
WithClock replaces the clock, so that a test can choose the time
func WithCooldown ¶
func WithCooldown(first time.Duration, longest time.Duration) GateOption
WithCooldown sets how long a host stays unhealthy before its first test, and the longest it waits after its cooldown has doubled
func WithDormantDays ¶
func WithDormantDays(days int) GateOption
WithDormantDays sets the number of separate days of failure that marks a host dormant
func WithFailureThreshold ¶
func WithFailureThreshold(count int) GateOption
WithFailureThreshold sets the number of consecutive failures that makes a host unhealthy
func WithJitter ¶
func WithJitter(longest time.Duration) GateOption
WithJitter sets the longest random delay added to a rescheduled delivery. Zero adds none.
func WithProbeSchedule ¶
WithProbeSchedule sets the wait between the first `earlyCount` checks of a dormant host, and the wait between every check after those
func WithRateLimit ¶
func WithRateLimit(fallback time.Duration, longest time.Duration) GateOption
WithRateLimit sets the pause used when a host names none, and the longest pause it can ask for
type Host ¶
type Host struct {
Host string `json:"host" bson:"_id"` // Hostname of the remote server, and the record's key
IsUnhealthy bool `json:"isUnhealthy" bson:"isUnhealthy"` // TRUE while deliveries are held back
TestAfter int64 `json:"testAfter" bson:"testAfter"` // Unix seconds when an unhealthy host starts testing
Cooldown int64 `json:"cooldown" bson:"cooldown"` // Seconds that the host last waited before testing
FailureCount int `json:"failureCount" bson:"failureCount"` // Consecutive failures since the last success
FailureDates []string `json:"failureDates" bson:"failureDates"` // Separate days (UTC) with a failure since the last success
DormantSince int64 `json:"dormantSince" bson:"dormantSince"` // Unix seconds when the host was marked dormant, or zero
LastRevived int64 `json:"lastRevived" bson:"lastRevived"` // Unix seconds when a success last cleared a history of trouble
LastFailure int64 `json:"lastFailure" bson:"lastFailure"` // Unix seconds of the last failed delivery
IsPaused bool `json:"isPaused" bson:"isPaused"` // TRUE if the server operator has stopped deliveries to this host
RateLimitUntil int64 `json:"rateLimitUntil" bson:"rateLimitUntil"` // Unix seconds until which the host asked for no requests
Software string `json:"software" bson:"software"` // Server software reported by NodeInfo
Version string `json:"version" bson:"version"` // Software version reported by NodeInfo
SignatureFormat string `json:"signatureFormat" bson:"signatureFormat"` // HTTP signature format that the host accepts, once known
NextProbe int64 `json:"nextProbe" bson:"nextProbe"` // Unix seconds when a dormant host is next checked
ProbeCount int `json:"probeCount" bson:"probeCount"` // Number of checks made since the host became dormant
}
Host is the record of how deliveries to one remote host have gone. A host with no record is healthy, so the zero value describes a healthy host.
func (Host) IsDormant ¶
IsDormant returns TRUE if the host has failed for long enough that deliveries are abandoned
func (Host) IsRateLimited ¶
IsRateLimited returns TRUE if the host has asked for no requests until a time after the one given
func (Host) State ¶
State returns the host's state at the given time: healthy, unhealthy, or testing
Example ¶
ExampleHost_State shows the three states that a host's record can be in
// A host with no record, or with no trouble on record, is healthy
fmt.Println(Host{}.State(exampleStart))
// An unhealthy host is left alone until its cooldown passes, and is then tested
record := Host{IsUnhealthy: true, TestAfter: exampleStart.Add(time.Minute).Unix()}
fmt.Println(record.State(exampleStart))
fmt.Println(record.State(exampleStart.Add(2 * time.Minute)))
Output: HEALTHY UNHEALTHY TESTING
type MemoryStore ¶
type MemoryStore struct {
// contains filtered or unexported fields
}
MemoryStore is a Store that keeps its records in one process's memory. It is for tests and single-process tools: servers that share work must share a Store that they can all reach.
func NewMemoryStore ¶
func NewMemoryStore() *MemoryStore
NewMemoryStore returns an empty MemoryStore
func (*MemoryStore) MarkDormant ¶
func (store *MemoryStore) MarkDormant(_ context.Context, host string, since time.Time, nextProbe time.Time) error
MarkDormant marks the host dormant, unless it already is or it has no record
func (*MemoryStore) MarkUnhealthy ¶
func (store *MemoryStore) MarkUnhealthy(_ context.Context, host string, testAfter time.Time, cooldown time.Duration) error
MarkUnhealthy holds deliveries back until the given time. It writes nothing when the host has no record.
func (*MemoryStore) RangeBlocked ¶
RangeBlocked returns every host that is dormant or paused
func (*MemoryStore) RangeProbesDue ¶
RangeProbesDue returns every dormant host whose next check is at or before the given time
func (*MemoryStore) RecordFailure ¶
func (store *MemoryStore) RecordFailure(_ context.Context, host string, now time.Time, date string) (Host, error)
RecordFailure counts one failure and its day, and returns the record afterwards
func (*MemoryStore) RecordProbeFailure ¶
func (store *MemoryStore) RecordProbeFailure(_ context.Context, host string, nextProbe time.Time) error
RecordProbeFailure counts one failed check and schedules the next. It writes nothing when the host has no record.
func (*MemoryStore) RecordSuccess ¶
RecordSuccess marks the host healthy and clears the failure history of a host that has a record
func (*MemoryStore) SetNodeInfo ¶
func (store *MemoryStore) SetNodeInfo(_ context.Context, host string, software string, version string) error
SetNodeInfo records the software that the host runs
func (*MemoryStore) SetPaused ¶
SetPaused records whether the server operator has stopped deliveries to the host
func (*MemoryStore) SetRateLimit ¶
SetRateLimit pauses the host until the given time, and never shortens a pause
func (*MemoryStore) SetSignatureFormat ¶
SetSignatureFormat records the HTTP signature format that the host accepts
type Store ¶
type Store interface {
// Load returns the record for a host. A host with no record yields a zero Host and no error.
Load(ctx context.Context, host string) (Host, error)
// RangeBlocked returns every host that is dormant or paused
RangeBlocked(ctx context.Context) (iter.Seq[Host], error)
// RangeProbesDue returns every dormant host whose next check is at or before the given time
RangeProbesDue(ctx context.Context, now time.Time) (iter.Seq[Host], error)
// RecordSuccess marks the host healthy and clears the failure history, the dormant mark and
// the probe schedule. It writes nothing when the host has no record.
RecordSuccess(ctx context.Context, host string, now time.Time) error
// RecordFailure adds one to the failure count, adds the day to the failure dates if it is
// new, creates the record if there is none, and returns the record as it stands afterwards
RecordFailure(ctx context.Context, host string, now time.Time, date string) (Host, error)
// MarkUnhealthy holds deliveries back until the given time, and records the cooldown that was
// used. It writes nothing when the host has no record.
MarkUnhealthy(ctx context.Context, host string, testAfter time.Time, cooldown time.Duration) error
// MarkDormant marks the host dormant and schedules its first check. It changes nothing
// when the host is already marked, or has no record.
MarkDormant(ctx context.Context, host string, since time.Time, nextProbe time.Time) error
// RecordProbeFailure adds one to the probe count and schedules the next check. It writes
// nothing when the host has no record.
RecordProbeFailure(ctx context.Context, host string, nextProbe time.Time) error
// SetRateLimit pauses the host until the given time, creating the record if there is none.
// It never shortens a pause: the later of the stored time and the new time is kept.
SetRateLimit(ctx context.Context, host string, until time.Time) error
// SetPaused records whether the server operator has stopped deliveries to the host,
// creating the record if there is none
SetPaused(ctx context.Context, host string, isPaused bool) error
// SetNodeInfo records the software that the host runs, creating the record if there is none
SetNodeInfo(ctx context.Context, host string, software string, version string) error
// SetSignatureFormat records the HTTP signature format that the host accepts, creating the
// record if there is none
SetSignatureFormat(ctx context.Context, host string, format string) error
}
Store keeps the Host records. Every server of an application shares one Store, so each write must be one atomic operation on one record, or two reports at once lose a count.
Source Files
¶
Directories
¶
| Path | Synopsis |
|---|---|
|
Package remotegate connects a floodgate.Gate to requests made with the benpate/remote package.
|
Package remotegate connects a floodgate.Gate to requests made with the benpate/remote package. |
|
Package storetest checks that an implementation of floodgate.Store behaves like the others.
|
Package storetest checks that an implementation of floodgate.Store behaves like the others. |