rehttp

README

rehttp

Package rehttp implements an HTTP Transport (an http.RoundTripper) that handles retries. See the godoc for details.

Please note that rehttp requires Go1.6+, because it uses the http.Request.Cancel field to check for cancelled requests. It should work on Go1.5, but only if there is no timeout set on the *http.Client. Go's stdlib will return an error on the first request if that's the case, because it requires a RoundTripper that implements the (now deprecated in Go1.6) CancelRequest method.

On Go1.7+, it uses the context returned by http.Request.Context to check for cancelled requests.

Installation

$ go get github.com/PuerkitoBio/rehttp

License

The BSD 3-Clause license.

Documentation

Overview

Package rehttp implements an HTTP transport that handles retries. An HTTP client can be created with a *rehttp.Transport as RoundTripper and it will apply the retry strategy to its requests.

The retry strategy is provided by the Transport, which determines whether or not the request should be retried, and if so, what delay to apply before retrying, based on the RetryFn and DelayFn functions passed to NewTransport.

The package offers common delay strategies as ready-made functions that return a DelayFn:

• ConstDelay(delay time.Duration) DelayFn
• ExpJitterDelay(base, max time.Duration) DelayFn
• ExpJitterDelayWithRand(base, max time.Duration, generator func(int64) int64) DelayFn

It also provides common retry helpers that return a RetryFn:

• RetryIsErr(func(error) bool) RetryFn
• RetryHTTPMethods(methods ...string) RetryFn
• RetryMaxRetries(max int) RetryFn
• RetryStatuses(statuses ...int) RetryFn
• RetryStatusInterval(fromStatus, toStatus int) RetryFn
• RetryTimeoutErr() RetryFn
• RetryTemporaryErr() RetryFn

Those can be combined with RetryAny or RetryAll as needed. RetryAny enables retries if any of the RetryFn return true, while RetryAll enables retries if all RetryFn return true. Typically, the RetryFn of the Transport should use at least RetryMaxRetries and some other retry condition(s), combined using RetryAll.

By default, the Transport will buffer the request's body in order to be able to retry the request, as a request attempt will consume and close the existing body. Sometimes this is not desirable, so it can be prevented by setting PreventRetryWithBody to true on the Transport. Doing so will disable retries when a request has a non-nil body.

This package requires Go version 1.6+, since it uses the new http.Request.Cancel field in order to cancel requests. It doesn't implement the deprecated http.Transport.CancelRequest method (https://golang.org/pkg/net/http/#Transport.CancelRequest).

On Go1.7+, it uses the context returned by http.Request.Context to check for cancelled requests. Before Go1.7, PerAttemptTimeout has no effect.

It should work on Go1.5, but only if there is no timeout set on the *http.Client. Go's stdlib will return an error on the first request if that's the case, because it requires a RoundTripper that implements the CancelRequest method.

Example

package main

import (
	"net/http"
	"time"

	"github.com/PuerkitoBio/rehttp"
)

func main() {
	tr := rehttp.NewTransport(
		nil, // will use http.DefaultTransport
		rehttp.RetryAll(rehttp.RetryMaxRetries(3), rehttp.RetryTemporaryErr()), // max 3 retries for Temporary errors
		rehttp.ConstDelay(time.Second),                                         // wait 1s between retries
	)
	client := &http.Client{
		Transport: tr,
		Timeout:   10 * time.Second, // Client timeout applies to all retries as a whole
	}
	res, err := client.Get("http://example.com")
	if err != nil {
		// handle err
	}
	// handle response
	res.Body.Close()
}

Index

• Variables
• type Attempt
• type DelayFn
  • func ConstDelay(delay time.Duration) DelayFn
  • func ExpJitterDelay(base, max time.Duration) DelayFn
  • func ExpJitterDelayWithRand(base, max time.Duration, generator func(n int64) int64) DelayFn
• type RetryFn
  • func RetryAll(retryFns ...RetryFn) RetryFn
  • func RetryAny(retryFns ...RetryFn) RetryFn
  • func RetryHTTPMethods(methods ...string) RetryFn
  • func RetryIsErr(fn func(error) bool) RetryFn
  • func RetryMaxRetries(max int) RetryFn
  • func RetryStatusInterval(fromStatus, toStatus int) RetryFn
  • func RetryStatuses(statuses ...int) RetryFn
  • func RetryTemporaryErr() RetryFn
  • func RetryTimeoutErr() RetryFn
• type Transport
  • func NewTransport(rt http.RoundTripper, retry RetryFn, delay DelayFn) *Transport
  • func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error)

Examples

• Package

Variables

var PRNG = rand.New(rand.NewSource(time.Now().UnixNano()))

PRNG is the *math.Rand value to use to add jitter to the backoff algorithm used in ExpJitterDelay. By default it uses a *rand.Rand initialized with a source based on the current time in nanoseconds.

Deprecated: math/rand sources can panic if used concurrently without synchronization. PRNG is no longer used by this package and its use outside this package is discouraged. https://github.com/PuerkitoBio/rehttp/issues/12

Types

type Attempt

type Attempt struct {
	// Index is the attempt index starting at 0.
	Index int

	// Request is the request for this attempt. If a non-nil Response
	// is present, this is the same as Response.Request, but since a
	// Response may not be available, it is guaranteed to be set on this
	// field.
	Request *http.Request

	// Response is the response for this attempt. It may be nil if an
	// error occurred an no response was received.
	Response *http.Response

	// Error is the error returned by the attempt, if any.
	Error error
}

Attempt holds the data describing a RoundTrip attempt.

type DelayFn

type DelayFn func(attempt Attempt) time.Duration

DelayFn is the signature for functions that return the delay to apply before the next retry.

func ConstDelay

func ConstDelay(delay time.Duration) DelayFn

ConstDelay returns a DelayFn that always returns the same delay.

func ExpJitterDelay

func ExpJitterDelay(base, max time.Duration) DelayFn

ExpJitterDelay is identical to ExpJitterDelayWithRand, using math/rand.Int63n as the random generator function. This package does not call rand.Seed, so it is the caller's responsibility to ensure the default generator is properly seeded.

func ExpJitterDelayWithRand

func ExpJitterDelayWithRand(base, max time.Duration, generator func(n int64) int64) DelayFn

ExpJitterDelayWithRand returns a DelayFn that returns a delay between 0 and base * 2^attempt capped at max (an exponential backoff delay with jitter). The generator argument is expected to generate a random int64 in the half open interval [0, n). It is the caller's responsibility to ensure that the function is safe for concurrent use.

See the full jitter algorithm in: http://www.awsarchitectureblog.com/2015/03/backoff.html

type RetryFn

type RetryFn func(attempt Attempt) bool

RetryFn is the signature for functions that return whether a retry should be done for the request.

func RetryAll

func RetryAll(retryFns ...RetryFn) RetryFn

RetryAll returns a RetryFn that allows a retry if all retryFns return true. If retryFns is empty, it always returns true.

func RetryAny

func RetryAny(retryFns ...RetryFn) RetryFn

RetryAny returns a RetryFn that allows a retry as long as one of the retryFns returns true. If retryFns is empty, it always returns false.

func RetryHTTPMethods

func RetryHTTPMethods(methods ...string) RetryFn

RetryHTTPMethods returns a RetryFn that retries if the request's HTTP method is one of the provided methods. It is meant to be used in conjunction with another RetryFn such as RetryTimeoutErr combined using RetryAll, otherwise this function will retry any successful request made with one of the provided methods.

func RetryIsErr

func RetryIsErr(fn func(error) bool) RetryFn

RetryIsErr returns a RetryFn that retries if the provided error predicate - a function that receives an error and returns a boolean - returns true for the error associated with the Attempt. Note that fn may be called with a nil error.

func RetryMaxRetries

func RetryMaxRetries(max int) RetryFn

RetryMaxRetries returns a RetryFn that retries if the number of retries is less than or equal to max.

func RetryStatusInterval

func RetryStatusInterval(fromStatus, toStatus int) RetryFn

RetryStatusInterval returns a RetryFn that retries if the response's status code is in the provided half-closed interval [fromStatus, toStatus) (that is, it retries if fromStatus <= Response.StatusCode < toStatus, so RetryStatusInterval(400, 500) would retry for any 4xx code, but not for 500).

func RetryStatuses

func RetryStatuses(statuses ...int) RetryFn

RetryStatuses returns a RetryFn that retries if the response's status code is one of the provided statuses.

func RetryTemporaryErr

func RetryTemporaryErr() RetryFn

RetryTemporaryErr returns a RetryFn that retries if the Attempt's error is a temporary error. A temporary error is one that implements the Temporary() bool method. Most errors from the net package implement this. This interface was deprecated in go 1.18. Favor RetryTimeoutErr. https://github.com/golang/go/issues/45729

func RetryTimeoutErr

func RetryTimeoutErr() RetryFn

RetryTimeoutErr returns a RetryFn that retries if the Attempt's error is a timeout error. Before go 1.13, a timeout error is one that implements the Timeout() bool method. Most errors from the net package implement this. After go 1.13, a timeout error is one that implements the net.Error interface which includes both Timeout() and Temporary() to make it less likely to falsely identify errors that occurred outside of the net package.

type Transport

type Transport struct {
	http.RoundTripper

	// PreventRetryWithBody prevents retrying if the request has a body. Since
	// the body is consumed on a request attempt, in order to retry a request
	// with a body, the body has to be buffered in memory. Setting this
	// to true avoids this buffering: the retry logic is bypassed if the body
	// is non-nil.
	PreventRetryWithBody bool

	// PerAttemptTimeout can be optionally set to add per-attempt timeouts.
	// These may be used in place of or in conjunction with overall timeouts.
	// For example, a per-attempt timeout of 5s would mean an attempt will
	// be canceled after 5s, then the delay fn will be consulted before
	// potentially making another attempt, which will again be capped at 5s.
	// This means that the overall duration may be up to
	// (PerAttemptTimeout + delay) * n, where n is the maximum attempts.
	// If using an overall timeout (whether on the http client or the request
	// context), the request will stop at whichever timeout is reached first.
	// Your RetryFn can determine if a request hit the per-attempt timeout by
	// checking if attempt.Error == context.DeadlineExceeded (or use errors.Is
	// on go 1.13+).
	// time.Duration(0) signals that no per-attempt timeout should be used.
	// Note that before go 1.7 this option has no effect.
	PerAttemptTimeout time.Duration
	// contains filtered or unexported fields
}

Transport wraps a RoundTripper such as *http.Transport and adds retry logic.

func NewTransport

func NewTransport(rt http.RoundTripper, retry RetryFn, delay DelayFn) *Transport

NewTransport creates a Transport with a retry strategy based on retry and delay to control the retry logic. It uses the provided RoundTripper to execute the requests. If rt is nil, http.DefaultTransport is used.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip implements http.RoundTripper for the Transport type. It calls its underlying http.RoundTripper to execute the request, and adds retry logic as per its configuration.