README
¶
retry
Package retry implements a wrapper to retry failing function calls.
About
retry is a package for calling functions repeatedly until they either succeed or the action is cancelled, for example due to a timeout. Retrying operations is a common strategy to deal with temporary failures in distributed systems, for example when using Remote Procedure Calls (RPCs).
This package provides all features needed in a large scale distributed environment and does so with a small and idiomatic API surface. Highlights are:
SRE best practices
Features and defaults in this package are heavily influenced by the SRE book, particularly the chapters Handling Overload and Addressing Cascading Failures. By default this package uses jitter to evenly distribute retries over the retry period and limits the number of attempts per request. A retry budget optionally limits the number of retries sent to a backend to prevent overload.
context aware
retry supports Go's context package.
Contexts are the idiomatic way to make a call with a deadline and to cancel
ongoing calls early. Refer to Go Concurrency Patterns:
Context for more information on contexts.
The Do() function takes a context.Context as its first argument and returns
immediately when the context is cancelled, for example when a timeout is reached
or when a client connection has been closed. The context is also passed to the
callback, so the callback can also implement the concurrency pattern.
The documentation has examples
demonstrating how the retry and context packages interact.
Permanent failures
If the client code detects a permanent failure, for example an "access forbidden" error, it can abort, preventing additional retries. This returns the (inevitable) error right away instead of wasting additional time and resources on retrying needlessly.
User code can signal a permanent failure by wrapping the error with
Abort or by returning an
error implementing the Error
interface. The Error interface is a subset of net.Error, i.e. errors created
by the net package will automatically do the right thing.
HTTP transport
A Transport type
implements all the logic required for retrying HTTP requests. The Transport
retries requests returning an HTTP 5xx status code, i.e. status codes signalling
a server-side error, in addition to temporary errors.
Examples
Cancel retries after timeout
This example, which is taken from the
documentation, demonstrates how
retries can be cancelled after 10 seconds using the context package.
// Create a context which is cancelled after 10 seconds.
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
// cb is a function that may or may not fail.
cb := func(_ context.Context) error {
return nil // or error
}
// Call cb via Do() until it succeeds or the 10 second timeout is reached.
if err := retry.Do(ctx, cb); err != nil {
log.Printf("cb() = %v", err)
}
Retry HTTP 5xx errors
This example, which is taken from the documentation, demonstrates how to retry an HTTP POST request until it succeeds or the 30 second timeout is reached.
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
c := &http.Client{
Transport: &Transport{},
}
req, err := http.NewRequest(http.MethodPost, "https://example.com/",
strings.NewReader(`{"example":true}`))
if err != nil {
log.Fatalf("NewRequest() = %v", err)
}
res, err := c.Do(req.WithContext(ctx))
if err != nil {
log.Printf("Do() = %v", err)
return
}
defer res.Body.Close()
// ...
Stability
This package is still a bit rough around the edges and there might be a backwards compatibility breaking change or two in its future, though none are planned at the moment.
License
Author
Florian Forster
Documentation
¶
Overview ¶
Package retry implements a wrapper to retry failing function calls.
Index ¶
Examples ¶
Constants ¶
This section is empty.
Variables ¶
var ErrExhausted = errors.New("retry budget exhausted")
ErrExhausted is returned by Do() when the retry budget is exhausted.
Functions ¶
func Attempt ¶
Attempt returns the number of previous attempts. In other words, it returns the zero-based index of the request.
Only call this function from within a retried function.
func Do ¶
Do repeatedly calls cb until it succeeds. After cb fails (returns a non-nil error), execution is paused for an exponentially increasing time. Execution can be cancelled at any time by cancelling the context.
By default, this function behaves as if the following options were passed:
Attempts(4),
ExpBackoff{
Base: 100 * time.Millisecond,
Max: 2 * time.Second,
Factor: 2.0,
},
FullJitter,
Example ¶
ctx := context.Background()
// cb is a function that may or may not fail.
cb := func(_ context.Context) error {
return nil // or error
}
// Call cb via Do() until it succeeds.
if err := Do(ctx, cb); err != nil {
log.Printf("cb() = %v", err)
}
Example (WithTimeout) ¶
This example demonstrates how to cancel a retried function call after a specific time.
// Create a context which is cancelled after 10 seconds.
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
// cb is a function that may or may not fail.
cb := func(_ context.Context) error {
return nil // or error
}
// Call cb via Do() until is succeeds or the 10 second timeout is reached.
if err := Do(ctx, cb); err != nil {
log.Printf("cb() = %v", err)
}
Types ¶
type Attempts ¶
type Attempts int
Attempts sets the number of calls made to the callback, i.e. the call is attempted at most n times. If all calls fail, the error of the last call is returned by Do().
Special case: the zero value retries indefinitely.
Implements the Option interface.
Example ¶
ctx := context.Background()
// cb is a function that may or may not fail.
cb := func(_ context.Context) error {
return nil // or error
}
// Call cb via Do() at most 5 times.
if err := Do(ctx, cb, Attempts(5)); err != nil {
log.Printf("cb() = %v", err)
}
type Budget ¶
type Budget struct {
// Rate is the minimum rate of retries (in calls per second).
// If fewer retries are attempted than this rate, retries are never throttled.
Rate float64
// Ratio is the maximum ratio of retries.
// When used as an option to Do(), it's the ratio of retries to initial
// calls. In that case ratio is a number in the [0.0, Attempts()]
// range. The initial request is never dropped.
// When used as part of BudgetHandler, it's the ratio of retries to
// total requests. In that case ratio is a number in the [0.0, 1.0]
// range.
Ratio float64
// contains filtered or unexported fields
}
Budget implements a retry budget, i.e. a limit for retries. Limiting the amount of retries sent to a service helps to mitigate cascading failures.
To add a retry budget for a specific service or backend, declare a Budget variable that is shared by all Do() calls. See the example for a demonstration.
Budget calculates the rate of initial calls and the rate of retries over a moving one minute window. If the rate of retries exceeds Budget.Rate and the ratio of retries exceeds Budget.Ratio, then retries are dropped. The Do() function returns ErrExhausted in this case.
Implements the Option interface.
Example ¶
ctx := context.Background()
// fooRetryBudget is a global variable holding the state of foo's retry budget.
// You should have one retry budget per backend service.
var fooRetryBudget = Budget{
Rate: 1.0,
Ratio: 0.1,
}
// failingRPC is a fake RPC call simulating a temporary backend failure.
failingRPC := func(_ context.Context) error {
return errors.New("temporary failure")
}
// Simulate 100 concurrent requests. Each request is tried initially,
// but only ~10 requests are retried, i.e. there will be approximately
// 110 calls of failingRPC in total.
for i := 0; i < 100; i++ {
go func() {
// Pass a pointer to fooRetryBudget to all Do() calls,
// i.e. all Do() calls receive a pointer to the same Budget{}.
// This allows state to be shared between Do() calls.
if err := Do(ctx, failingRPC, &fooRetryBudget); err != nil {
log.Println(err)
}
}()
}
type BudgetHandler ¶
type BudgetHandler struct {
http.Handler
// Budget is the server side retry budget. While Handler is handling
// fewer than Budget.Rate requests, responses are never modified. If
// the ratio of retries to total requests exceeds Budget.Ratio, this is
// taken as an indicator that the cluster as a whole is overloaded.
Budget
}
BudgetHandler wraps an http.Handler and applies a server-side retry budget. When the ratio of retries exceeds BudgetHandler.Ratio while the rate of requests is at least BudgetHandler.Rate, i.e. when the retry budget is exhausted, then temporary errors are changed to permanent errors. A high ratio of retries is an indicator that the cluster as a whole is overloaded. Returning permanent errors in an overload situation mitigates the risk that retries are keeping the system in overload.
An HTTP request is considered a retry if it has the "Retry-Attempt" HTTP header set, as created by Transport. The value of the header is not relevant, as long as it is not empty.
Temporary errors are primarily responses with 5xx status codes, but there are exceptions. See the documentation of "Transport" type for a detailed discussion.
When in an overload situation, BudgetHandler:
• sets the status code to 429 "Too Many Requests" if the status code indicates a temporary failure, and
• removes the "Retry-After" header if set.
Note that this is not a rate limiter. BudgetHandler will never decline a request itself, it only makes sure that if a request is declined, for example with 503 "Service Unavailable", the status code is upgraded to a permanent error when the retry budget is exhausted, i.e. when in overload.
func (*BudgetHandler) ServeHTTP ¶
func (h *BudgetHandler) ServeHTTP(w http.ResponseWriter, req *http.Request)
ServeHTTP proxies the HTTP request to the embedded http.Handler.
type Error ¶
Error is an error type that controls retry behavior. If Temporary() returns false, Do() returns immediately and does not continue to call the callback function.
Error is specifically designed to be a subset of net.Error.
func Abort ¶
Abort wraps err so it implements the Error interface and reports a permanent condition. This causes Do() to return immediately with the wrapped error.
Example ¶
This example demonstrates how responses to an HTTP request might be handled. Responses with an error code between 400 and 499 will abort the Do() call, since the server indicates that there is problem on the client side and retrying the callback would just do the same thing again.
ctx := context.Background()
cb := func(ctx context.Context) error {
req, err := http.NewRequest(http.MethodGet, "http://example.com/", nil)
if err != nil {
return err
}
req = req.WithContext(ctx)
res, err := http.DefaultClient.Do(req)
if err != nil {
// This is likely a networking problem since the default client doesn't have any policies configured.
// Specifically, it may be a net.Error which implements the Error interface.
// Returning this may or may not abort the Do() call, depending on the error.
return err
}
if res.StatusCode >= 400 && res.StatusCode < 500 {
// Client error, i.e. we're doing someting wrong
// -> Abort
return Abort(fmt.Errorf("HTTP status %d (%q)", res.StatusCode, res.Status))
}
if res.StatusCode >= 500 {
// Server error, i.e. not our fault
// -> Try again
return fmt.Errorf("HTTP status %d (%q)", res.StatusCode, res.Status)
}
// TODO: do something meaningful with res.
return nil // Success
}
// Call cb via Do() until it succeeds or Abort() is returned.
if err := Do(ctx, cb); err != nil {
log.Printf("cb() = %v", err)
}
type ExpBackoff ¶
ExpBackoff sets custom backoff parameters. After the first failure, execution pauses for the duration specified by base. After each subsequent failure the delay is multiplied by Factor until max is reached. Execution is never paused for longer than the Max duration.
Implements the Option interface.
Example ¶
ctx := context.Background()
// cb is a function that may or may not fail.
cb := func(_ context.Context) error {
return nil // or error
}
opts := []Option{
ExpBackoff{
Base: 10 * time.Millisecond,
Max: 5 * time.Second,
Factor: 2.0,
},
}
// Call cb via Do() with custom backoff parameters.
if err := Do(ctx, cb, opts...); err != nil {
log.Printf("cb() = %v", err)
}
type Jitter ¶
type Jitter float64
Jitter is a randomization of the backoff delay. Randomizing the delay avoids thundering herd problems, for example when using optimistic locking.
This generic jitter implementation has two components: a fixed delay (calculated by ExpBackoff) and a random component in the range [0,delay). The Jitter type is a floating point number in the (0-1] range that controls the ratio of the random component. Assuming that the current backoff delay is 100ms, Jitter 1.0 means the result is in the range [0,100) ms, Jitter 0.2 means the result is in the range [80,100) ms.
The following formula is used:
delay = Jitter * random_between(0, delay) + (1 - Jitter) * delay
Special cases: the zero value is treated equally to FullJitter. Minus one (-1.0) deactivates jitter.
An in-depth discussion of different jitter strategies and their impact on client work and server load is available at: https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
Implements the Option interface.
const EqualJitter Jitter = 0.5
EqualJitter produces random delays in the [max/2,max) range. The name refers to the fact that the obligatory delay and the random range are of equal length.
const FullJitter Jitter = 1.0
FullJitter produces random delays in the [0,max) range. This is the recommanded instance and the default behavior.
const WithoutJitter Jitter = -1.0
WithoutJitter deactivates jitter and always returns delay unchanged.
type Option ¶
type Option interface {
// contains filtered or unexported methods
}
Option is an option for Do().
The following types implement Option:
• Attempts
• Budget
• ExpBackoff
• Jitter
• Timeout
type Timeout ¶
Timeout specifies the timeout for each individual attempt. When specified, the context passed to the callback is cancelled after this duration. When the timeout expires, the callback should return as quickly as possible. The retry logic continues without waiting for the callback to return, though, so callbacks should be thread-safe.
Implements the Option interface.
type Transport ¶
type Transport struct {
http.RoundTripper
// contains filtered or unexported fields
}
Transport is a retrying "net/http".RoundTripper. The zero value of Transport is a valid "net/http".RoundTripper that is using "net/http".DefaultTransport.
Custom options can be set by initializing Transport with NewTransport().
One consequence of using this transport is that HTTP 5xx errors will be reported as errors, with one exception:
• The 501 "Not Implemented" status code is treated as a permanent failure.
HTTP 4xx errors are generally not retried (and therefore don't result in an error being returned), with two exceptions:
• The 423 "Locked" status code is treated like a temporary issue.
• If the response has a 4xx status code and the "Retry-After" header, the request is retried.
Transport needs to be able to read the request body multiple times. Depending on the provided Request.Body, this happens in one of two ways:
• If Request.Body implements the io.Seeker interface, Body is rewound by calling Seek().
• Otherwise, Request.Body is copied into an internal buffer, which consumes additional memory.
When re-sending HTTP requests the transport adds the "Retry-Attempt" HTTP header indicating that a request is a retry. The header value is an integer counting the retries, i.e. "1" for the first retry (the second attempt overall). Note: there is currently no standard or even de-facto standard way of indicating retries to an HTTP server. When an appropriate RFC is published or an industry standard emerges, this header will be changed accordingly.
Use "net/http".Request.WithContext() to pass a context to Do(). By default, the request is associated with the background context.
Example ¶
c := &http.Client{
Transport: &Transport{},
}
// Caveat: there is no specific context associated with this request.
// The net/http package uses the background context in that case.
// That means that this request will be retried indefinitely until it succeeds.
res, err := c.Get("http://example.com/")
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
if res.StatusCode >= 500 && res.StatusCode < 600 {
panic("this does not happen. HTTP 5xx errors are reported as errors.")
}
// use "res"
Example (WithOptions) ¶
c := &http.Client{
Transport: NewTransport(http.DefaultTransport, Attempts(3)),
}
// Caveat: there is no specific context associated with this request.
// The net/http package uses the background context, i.e. cancellation does not work.
res, err := c.Get("http://example.com/")
if err != nil {
log.Fatal(err)
}
defer res.Body.Close()
// use "res"
Example (WithTimeout) ¶
ctx, cancel := context.WithTimeout(context.Background(), 30*time.Second)
defer cancel()
c := &http.Client{
Transport: &Transport{},
}
// The context needs to be added to the request via http.Request.WithContext().
// The net/http package defaults to using the background context, which is never cancelled.
// That's why NewRequest()/Do() is used here instead of the more
// convenient Get(), Head() and Post() short-hands.
req, err := http.NewRequest(http.MethodPost, "https://example.com/",
strings.NewReader(`{"example":true}`))
if err != nil {
log.Fatalf("NewRequest() = %v", err)
}
res, err := c.Do(req.WithContext(ctx))
if err != nil {
log.Printf("Do() = %v", err)
return
}
defer res.Body.Close()
// use "res"
func NewTransport ¶
func NewTransport(base http.RoundTripper, opts ...Option) *Transport
NewTransport initializes a new Transport with the provided options.
base may be nil in which case it defaults to "net/http".DefaultTransport.
Source Files