README
¶
go-retryablehttp
The retryablehttp package provides a familiar HTTP client interface with
automatic retries and exponential backoff. It is a thin wrapper over the
standard net/http client library and exposes nearly the same public API. This
makes retryablehttp very easy to drop into existing programs.
retryablehttp performs automatic retries under certain conditions. Mainly, if
an error is returned by the client (connection errors, etc.), or if a 500-range
response code is received (except 501), then a retry is invoked after a wait
period. Otherwise, the response is returned and left to the caller to
interpret.
The main difference from net/http is that requests which take a request body
(POST/PUT et. al) can have the body provided in a number of ways (some more or
less efficient) that allow "rewinding" the request body if the initial request
fails so that the full request can be attempted again. See the
godoc for more
details.
Version 0.6.0 and before are compatible with Go prior to 1.12. From 0.6.1 onward, Go 1.12+ is required. From 0.6.7 onward, Go 1.13+ is required.
Example Use
Using this library should look almost identical to what you would do with
net/http. The most simple example of a GET request is shown below:
resp, err := retryablehttp.Get("/foo")
if err != nil {
panic(err)
}
The returned response object is an *http.Response, the same thing you would
usually get from net/http. Had the request failed one or more times, the above
call would block and retry with exponential backoff.
Retrying cases that fail after a seeming success
It's possible for a request to succeed in the sense that the expected response headers are received, but then to encounter network-level errors while reading the response body. In go-retryablehttp's most basic usage, this error would not be retryable, due to the out-of-band handling of the response body. In some cases it may be desirable to handle the response body as part of the retryable operation.
A toy example (which will retry the full request and succeed on the second attempt) is shown below:
c := retryablehttp.NewClient()
r := retryablehttp.NewRequest("GET", "://foo", nil)
handlerShouldRetry := true
r.SetResponseHandler(func(*http.Response) error {
if !handlerShouldRetry {
return nil
}
handlerShouldRetry = false
return errors.New("retryable error")
})
Getting a stdlib *http.Client with retries
It's possible to convert a *retryablehttp.Client directly to a *http.Client.
This makes use of retryablehttp broadly applicable with minimal effort. Simply
configure a *retryablehttp.Client as you wish, and then call StandardClient():
retryClient := retryablehttp.NewClient()
retryClient.RetryMax = 10
standardClient := retryClient.StandardClient() // *http.Client
For more usage and examples see the godoc.
Documentation
¶
Overview ¶
Package retryablehttp provides a familiar HTTP client interface with automatic retries and exponential backoff. It is a thin wrapper over the standard net/http client library and exposes nearly the same public API. This makes retryablehttp very easy to drop into existing programs.
retryablehttp performs automatic retries under certain conditions. Mainly, if an error is returned by the client (connection errors etc), or if a 500-range response is received, then a retry is invoked. Otherwise, the response is returned and left to the caller to interpret.
Requests which take a request body should provide a non-nil function parameter. The best choice is to provide either a function satisfying ReaderFunc which provides multiple io.Readers in an efficient manner, a *bytes.Buffer (the underlying raw byte slice will be used) or a raw byte slice. As it is a reference type, and we will wrap it as needed by readers, we can efficiently re-use the request body without needing to copy it. If an io.Reader (such as a *bytes.Reader) is provided, the full body will be read prior to the first request, and will be efficiently re-used for any retries. ReadSeeker can be used, but some users have observed occasional data races between the net/http library and the Seek functionality of some implementations of ReadSeeker, so should be avoided if possible.
Index ¶
- func DefaultBackoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
- func DefaultRetryPolicy(ctx context.Context, resp *http.Response, err error) (bool, error)
- func ErrorPropagatedRetryPolicy(ctx context.Context, resp *http.Response, err error) (bool, error)
- func Get(url string) (*http.Response, error)
- func Head(url string) (*http.Response, error)
- func LinearJitterBackoff(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
- func PassthroughErrorHandler(resp *http.Response, err error, _ int) (*http.Response, error)
- func Post(url, bodyType string, body interface{}) (*http.Response, error)
- func PostForm(url string, data url.Values) (*http.Response, error)
- type Backoff
- type CheckRetry
- type Client
- func (c *Client) Do(req *Request) (*http.Response, error)
- func (c *Client) Get(url string) (*http.Response, error)
- func (c *Client) Head(url string) (*http.Response, error)
- func (c *Client) Post(url, bodyType string, body interface{}) (*http.Response, error)
- func (c *Client) PostForm(url string, data url.Values) (*http.Response, error)
- func (c *Client) StandardClient() *http.Client
- type ErrorHandler
- type LenReader
- type LeveledLogger
- type Logger
- type ReaderFunc
- type Request
- type RequestLogHook
- type ResponseHandlerFunc
- type ResponseLogHook
- type RoundTripper
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func DefaultBackoff ¶
DefaultBackoff provides a default callback for Client.Backoff which will perform exponential backoff based on the attempt number and limited by the provided minimum and maximum durations.
It also tries to parse Retry-After response header when a http.StatusTooManyRequests (HTTP Code 429) is found in the resp parameter. Hence it will return the number of seconds the server states it may be ready to process more requests from this client.
func DefaultRetryPolicy ¶
DefaultRetryPolicy provides a default callback for Client.CheckRetry, which will retry on connection errors and server errors.
func ErrorPropagatedRetryPolicy ¶
ErrorPropagatedRetryPolicy is the same as DefaultRetryPolicy, except it propagates errors back instead of returning nil. This allows you to inspect why it decided to retry or not.
func LinearJitterBackoff ¶
LinearJitterBackoff provides a callback for Client.Backoff which will perform linear backoff based on the attempt number and with jitter to prevent a thundering herd.
min and max here are *not* absolute values. The number to be multiplied by the attempt number will be chosen at random from between them, thus they are bounding the jitter.
For instance: * To get strictly linear backoff of one second increasing each retry, set both to one second (1s, 2s, 3s, 4s, ...) * To get a small amount of jitter centered around one second increasing each retry, set to around one second, such as a min of 800ms and max of 1200ms (892ms, 2102ms, 2945ms, 4312ms, ...) * To get extreme jitter, set to a very wide spread, such as a min of 100ms and a max of 20s (15382ms, 292ms, 51321ms, 35234ms, ...)
func PassthroughErrorHandler ¶
PassthroughErrorHandler is an ErrorHandler that directly passes through the values from the net/http library for the final request. The body is not closed.
Types ¶
type Backoff ¶
Backoff specifies a policy for how long to wait between retries. It is called after a failing request to determine the amount of time that should pass before trying again.
type CheckRetry ¶
CheckRetry specifies a policy for handling retries. It is called following each request with the response and error values returned by the http.Client. If CheckRetry returns false, the Client stops retrying and returns the response to the caller. If CheckRetry returns an error, that error value is returned in lieu of the error from the request. The Client will close any response body when retrying, but if the retry is aborted it is up to the CheckRetry callback to properly close any response body before returning.
type Client ¶
type Client struct {
HTTPClient *http.Client // Internal HTTP client.
Logger interface{} // Customer logger instance. Can be either Logger or LeveledLogger
RetryWaitMin time.Duration // Minimum time to wait
RetryWaitMax time.Duration // Maximum time to wait
RetryMax int // Maximum number of retries
// RequestLogHook allows a user-supplied function to be called
// before each retry.
RequestLogHook RequestLogHook
// ResponseLogHook allows a user-supplied function to be called
// with the response from each HTTP request executed.
ResponseLogHook ResponseLogHook
// CheckRetry specifies the policy for handling retries, and is called
// after each request. The default policy is DefaultRetryPolicy.
CheckRetry CheckRetry
// Backoff specifies the policy for how long to wait between retries
Backoff Backoff
// ErrorHandler specifies the custom error handler to use, if any
ErrorHandler ErrorHandler
// contains filtered or unexported fields
}
Client is used to make HTTP requests. It adds additional functionality like automatic retries to tolerate minor outages.
func (*Client) PostForm ¶
PostForm is a convenience method for doing simple POST operations using pre-filled url.Values form data.
func (*Client) StandardClient ¶
StandardClient returns a stdlib *http.Client with a custom Transport, which shims in a *retryablehttp.Client for added retries.
type ErrorHandler ¶
ErrorHandler is called if retries are expired, containing the last status from the http library. If not specified, default behavior for the library is to close the body and return an error indicating how many tries were attempted. If overriding this, be sure to close the body if needed.
type LenReader ¶
type LenReader interface {
Len() int
}
LenReader is an interface implemented by many in-memory io.Reader's. Used for automatically sending the right Content-Length header when possible.
type LeveledLogger ¶
type LeveledLogger interface {
Error(msg string, keysAndValues ...interface{})
Info(msg string, keysAndValues ...interface{})
Debug(msg string, keysAndValues ...interface{})
Warn(msg string, keysAndValues ...interface{})
}
LeveledLogger is an interface that can be implemented by any logger or a logger wrapper to provide leveled logging. The methods accept a message string and a variadic number of key-value pairs. For log.Printf style formatting where message string contains a format specifier, use Logger interface.
type Logger ¶
type Logger interface {
Printf(string, ...interface{})
}
Logger interface allows to use other loggers than standard log.Logger.
type ReaderFunc ¶
ReaderFunc is the type of function that can be given natively to NewRequest
type Request ¶
type Request struct {
// Embed an HTTP request directly. This makes a *Request act exactly
// like an *http.Request so that all meta methods are supported.
*http.Request
// contains filtered or unexported fields
}
Request wraps the metadata needed to create HTTP requests.
func FromRequest ¶
FromRequest wraps an http.Request in a retryablehttp.Request
func NewRequest ¶
NewRequest creates a new wrapped request.
func NewRequestWithContext ¶
func NewRequestWithContext(ctx context.Context, method, url string, rawBody interface{}) (*Request, error)
NewRequestWithContext creates a new wrapped request with the provided context.
The context controls the entire lifetime of a request and its response: obtaining a connection, sending the request, and reading the response headers and body.
func (*Request) BodyBytes ¶
BodyBytes allows accessing the request body. It is an analogue to http.Request's Body variable, but it returns a copy of the underlying data rather than consuming it.
This function is not thread-safe; do not call it at the same time as another call, or at the same time this request is being used with Client.Do.
func (*Request) SetBody ¶
SetBody allows setting the request body.
It is useful if a new body needs to be set without constructing a new Request.
func (*Request) SetResponseHandler ¶
func (r *Request) SetResponseHandler(fn ResponseHandlerFunc)
SetResponseHandler allows setting the response handler.
func (*Request) WithContext ¶
WithContext returns wrapped Request with a shallow copy of underlying *http.Request with its context changed to ctx. The provided ctx must be non-nil.
func (*Request) WriteTo ¶
WriteTo allows copying the request body into a writer.
It writes data to w until there's no more data to write or when an error occurs. The return int64 value is the number of bytes written. Any error encountered during the write is also returned. The signature matches io.WriterTo interface.
type RequestLogHook ¶
RequestLogHook allows a function to run before each retry. The HTTP request which will be made, and the retry number (0 for the initial request) are available to users. The internal logger is exposed to consumers.
type ResponseHandlerFunc ¶
ResponseHandlerFunc is a type of function that takes in a Response, and does something with it. It only runs if the initial part of the request was successful. If an error is returned, the client's retry policy will be used to determine whether to retry the whole request.
type ResponseLogHook ¶
ResponseLogHook is like RequestLogHook, but allows running a function on each HTTP response. This function will be invoked at the end of every HTTP request executed, regardless of whether a subsequent retry needs to be performed or not. If the response body is read or closed from this method, this will affect the response returned from Do().
type RoundTripper ¶
type RoundTripper struct {
// The client to use during requests. If nil, the default retryablehttp
// client and settings will be used.
Client *Client
// contains filtered or unexported fields
}
RoundTripper implements the http.RoundTripper interface, using a retrying HTTP client to execute requests.
It is important to note that retryablehttp doesn't always act exactly as a RoundTripper should. This is highly dependent on the retryable client's configuration.
Source Files