README
¶
retryablehttp
Heavily inspired from https://github.com/hashicorp/go-retryablehttp.
Usage
Example of using retryablehttp in Go Code is available in examples folder
Examples of using Nuclei From Go Code to run templates on targets are provided in the examples folder.
url encoding and parsing issues
retryablehttp.Request by default handles some url encoding and parameters issues. since http.Request internally uses url.Parse() to parse url specified in request it creates some inconsistencies for below urls and other non-RFC compilant urls
// below urls are either normalized or returns error when used in `http.NewRequest()`
https://scanme.sh/%invalid
https://scanme.sh/w%0d%2e/
scanme.sh/with/path?some'param=`'+OR+ORDER+BY+1--
All above mentioned cases are handled internally in retryablehttp.
Note
It is not recommended to update url.URL instance of Request once a new request is created (ex req.URL.Path = xyz) due to internal logic of urls.
In any case if it is not possible to follow above point due to some reason helper methods are available to reflect such changes
Request.Update()commits any changes made to query parameters (ex:Request.URL.Query().Add(x,y))
Documentation
¶
Overview ¶
Package retryablehttp provides a familiar HTTP client interface with automatic retries and exponential backoff. It is a thin wrapper over the standard net/http client library and exposes nearly the same public API. This makes retryablehttp very easy to drop into existing programs.
retryablehttp performs automatic retries under certain conditions. Mainly, if an error is returned by the client (connection errors etc), or if a 500-range response is received, then a retry is invoked. Otherwise, the response is returned and left to the caller to interpret.
Requests which take a request body should provide a non-nil function parameter. The best choice is to provide either a function satisfying ReaderFunc which provides multiple io.Readers in an efficient manner, a *bytes.Buffer (the underlying raw byte slice will be used) or a raw byte slice. As it is a reference type, and we will wrap it as needed by readers, we can efficiently re-use the request body without needing to copy it. If an io.Reader (such as a *bytes.Reader) is provided, the full body will be read prior to the first request, and will be efficiently re-used for any retries. ReadSeeker can be used, but some users have observed occasional data races between the net/http library and the Seek functionality of some implementations of ReadSeeker, so should be avoided if possible.
Index ¶
- Variables
- func CheckRecoverableErrors(ctx context.Context, resp *http.Response, err error) (bool, error)
- func DefaultBackoff() ...
- func DefaultClient() *http.Client
- func DefaultHostSprayingTransport() *http.Transport
- func DefaultPooledClient() *http.Client
- func DefaultRetryPolicy() func(ctx context.Context, resp *http.Response, err error) (bool, error)
- func DefaultReusePooledTransport() *http.Transport
- func Discard(req *Request, resp *http.Response, RespReadLimit int64)
- func ExponentialJitterBackoff() ...
- func FullJitterBackoff() ...
- func Get(url string) (*http.Response, error)
- func Head(url string) (*http.Response, error)
- func HostSprayRetryPolicy() func(ctx context.Context, resp *http.Response, err error) (bool, error)
- func LinearJitterBackoff() ...
- func PassthroughErrorHandler(resp *http.Response, err error, _ int) (*http.Response, error)
- func Post(url, bodyType string, body interface{}) (*http.Response, error)
- func PostForm(url string, data url.Values) (*http.Response, error)
- type Auth
- type AuthType
- type Backoff
- type CheckRetry
- type Client
- func (c *Client) Do(req *Request) (*http.Response, error)
- func (c *Client) Get(url string) (*http.Response, error)
- func (c *Client) Head(url string) (*http.Response, error)
- func (c *Client) Post(url, bodyType string, body interface{}) (*http.Response, error)
- func (c *Client) PostForm(url string, data url.Values) (*http.Response, error)
- type ContextOverride
- type ErrorHandler
- type Metrics
- type Options
- type Request
- func FromRequest(r *http.Request) (*Request, error)
- func FromRequestWithTrace(r *http.Request) (*Request, error)
- func NewRequest(method, url string, body interface{}) (*Request, error)
- func NewRequestFromURL(method string, urlx *urlutil.URL, body interface{}) (*Request, error)
- func NewRequestFromURLWithContext(ctx context.Context, method string, urlx *urlutil.URL, body interface{}) (*Request, error)
- func NewRequestWithContext(ctx context.Context, method, url string, body interface{}) (*Request, error)
- type RequestLogHook
- type ResponseLogHook
Constants ¶
This section is empty.
Variables ¶
var DefaultOptionsSingle = Options{ RetryWaitMin: 1 * time.Second, RetryWaitMax: 30 * time.Second, Timeout: 30 * time.Second, RetryMax: 5, RespReadLimit: 4096, KillIdleConn: false, NoAdjustTimeout: true, }
DefaultOptionsSingle contains the default options for host bruteforce scenarios where lots of requests need to be sent to a single host.
var DefaultOptionsSpraying = Options{ RetryWaitMin: 1 * time.Second, RetryWaitMax: 30 * time.Second, Timeout: 30 * time.Second, RetryMax: 5, RespReadLimit: 4096, KillIdleConn: true, NoAdjustTimeout: true, }
DefaultOptionsSpraying contains the default options for host spraying scenarios where lots of requests need to be sent to different hosts.
var DisableZTLSFallback = false
DisableZTLSFallback disables use of ztls when there is error in tls handshake can also be disabled by setting DISABLE_ZTLS_FALLBACK env variable to true
var PreferHTTP bool
When True . Request uses `http` as scheme instead of `https`
Functions ¶
func CheckRecoverableErrors ¶ added in v1.0.4
Check recoverable errors
func DefaultBackoff ¶
func DefaultBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
DefaultBackoff provides a default callback for Client.Backoff which will perform exponential backoff based on the attempt number and limited by the provided minimum and maximum durations.
func DefaultClient ¶
DefaultClient returns a new http.Client with similar default values to http.Client, but with a non-shared Transport, idle connections disabled, and keepalives disabled.
func DefaultHostSprayingTransport ¶
DefaultHostSprayingTransport returns a new http.Transport with similar default values to http.DefaultTransport, but with idle connections and keepalives disabled.
func DefaultPooledClient ¶
DefaultPooledClient returns a new http.Client with similar default values to http.Client, but with a shared Transport. Do not use this function for transient clients as it can leak file descriptors over time. Only use this for clients that will be re-used for the same host(s).
func DefaultRetryPolicy ¶
DefaultRetryPolicy provides a default callback for Client.CheckRetry, which will retry on connection errors and server errors.
func DefaultReusePooledTransport ¶
DefaultReusePooledTransport returns a new http.Transport with similar default values to http.DefaultTransport. Do not use this for transient transports as it can leak file descriptors over time. Only use this for transports that will be re-used for the same host(s).
func Discard ¶
Discard is an helper function that discards the response body and closes the underlying connection
func ExponentialJitterBackoff ¶
func ExponentialJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
ExponentialJitterBackoff provides a callback for Client.Backoff which will perform en exponential backoff based on the attempt number and with jitter to prevent a thundering herd.
min and max here are *not* absolute values. The number to be multipled by the attempt number will be chosen at random from between them, thus they are bounding the jitter.
func FullJitterBackoff ¶
func FullJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
FullJitterBackoff implements capped exponential backoff with jitter. Algorithm is fast because it does not use floating point arithmetics. It returns a random number between [0...n] https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/
func HostSprayRetryPolicy ¶
HostSprayRetryPolicy provides a callback for Client.CheckRetry, which will retry on connection errors and server errors.
func LinearJitterBackoff ¶
func LinearJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration
LinearJitterBackoff provides a callback for Client.Backoff which will perform linear backoff based on the attempt number and with jitter to prevent a thundering herd.
min and max here are *not* absolute values. The number to be multipled by the attempt number will be chosen at random from between them, thus they are bounding the jitter.
For instance: - To get strictly linear backoff of one second increasing each retry, set both to one second (1s, 2s, 3s, 4s, ...) - To get a small amount of jitter centered around one second increasing each retry, set to around one second, such as a min of 800ms and max of 1200ms (892ms, 2102ms, 2945ms, 4312ms, ...) - To get extreme jitter, set to a very wide spread, such as a min of 100ms and a max of 20s (15382ms, 292ms, 51321ms, 35234ms, ...)
func PassthroughErrorHandler ¶
PassthroughErrorHandler is an ErrorHandler that directly passes through the values from the net/http library for the final request. The body is not closed.
Types ¶
type Backoff ¶
Backoff specifies a policy for how long to wait between retries. It is called after a failing request to determine the amount of time that should pass before trying again.
type CheckRetry ¶
CheckRetry specifies a policy for handling retries. It is called following each request with the response and error values returned by the http.Client. If CheckRetry returns false, the Client stops retrying and returns the response to the caller. If CheckRetry returns an error, that error value is returned in lieu of the error from the request. The Client will close any response body when retrying, but if the retry is aborted it is up to the CheckRetry callback to properly close any response body before returning.
type Client ¶
type Client struct {
// HTTPClient is the internal HTTP client (http1x + http2 via connection upgrade upgrade).
HTTPClient *http.Client
// HTTPClient is the internal HTTP client configured to fallback to native http2 at transport level
HTTPClient2 *http.Client
// RequestLogHook allows a user-supplied function to be called
// before each retry.
RequestLogHook RequestLogHook
// ResponseLogHook allows a user-supplied function to be called
// with the response from each HTTP request executed.
ResponseLogHook ResponseLogHook
// ErrorHandler specifies the custom error handler to use, if any
ErrorHandler ErrorHandler
// CheckRetry specifies the policy for handling retries, and is called
// after each request. The default policy is DefaultRetryPolicy.
CheckRetry CheckRetry
// Backoff specifies the policy for how long to wait between retries
Backoff Backoff
// contains filtered or unexported fields
}
Client is used to make HTTP requests. It adds additional functionality like automatic retries to tolerate minor outages.
var DefaultHTTPClient *Client
DefaultHTTPClient is the http client with DefaultOptionsSingle options.
func NewWithHTTPClient ¶
NewWithHTTPClient creates a new Client with custom http client Deprecated: Use options.HttpClient
type ContextOverride ¶ added in v1.0.3
type ContextOverride string
const (
RETRY_MAX ContextOverride = "retry-max"
)
type ErrorHandler ¶
ErrorHandler is called if retries are expired, containing the last status from the http library. If not specified, default behavior for the library is to close the body and return an error indicating how many tries were attempted. If overriding this, be sure to close the body if needed.
type Metrics ¶
type Metrics struct {
// Failures is the number of failed requests
Failures int
// Retries is the number of retries for the request
Retries int
// DrainErrors is number of errors occured in draining response body
DrainErrors int
}
Metrics contains the metrics about each request
type Options ¶
type Options struct {
// RetryWaitMin is the minimum time to wait for retry
RetryWaitMin time.Duration
// RetryWaitMax is the maximum time to wait for retry
RetryWaitMax time.Duration
// Timeout is the maximum time to wait for the request
Timeout time.Duration
// RetryMax is the maximum number of retries
RetryMax int
// RespReadLimit is the maximum HTTP response size to read for
// connection being reused.
RespReadLimit int64
// Verbose specifies if debug messages should be printed
Verbose bool
// KillIdleConn specifies if all keep-alive connections gets killed
KillIdleConn bool
// Custom CheckRetry policy
CheckRetry CheckRetry
// Custom Backoff policy
Backoff Backoff
// NoAdjustTimeout disables automatic adjustment of HTTP request timeout
NoAdjustTimeout bool
// Custom http client
HttpClient *http.Client
}
Options contains configuration options for the client
type Request ¶
type Request struct {
// Embed an HTTP request directly. This makes a *Request act exactly
// like an *http.Request so that all meta methods are supported.
*http.Request
//URL
*urlutil.URL
// Metrics contains the metrics for the request.
Metrics Metrics
Auth *Auth
}
Request wraps the metadata needed to create HTTP requests. Request is not threadsafe. A request cannot be used by multiple goroutines concurrently.
func FromRequest ¶
FromRequest wraps an http.Request in a retryablehttp.Request
func FromRequestWithTrace ¶
FromRequestWithTrace wraps an http.Request in a retryablehttp.Request with trace enabled
func NewRequest ¶
NewRequest creates a new wrapped request
func NewRequestFromURL ¶ added in v1.0.11
NewRequest creates a new wrapped request.
func NewRequestFromURLWithContext ¶ added in v1.0.11
func NewRequestFromURLWithContext(ctx context.Context, method string, urlx *urlutil.URL, body interface{}) (*Request, error)
NewRequestWithContext creates a new wrapped request with context
func NewRequestWithContext ¶
func NewRequestWithContext(ctx context.Context, method, url string, body interface{}) (*Request, error)
NewRequest creates a new wrapped request with given context
func (*Request) BodyBytes ¶
BodyBytes allows accessing the request body. It is an analogue to http.Request's Body variable, but it returns a copy of the underlying data rather than consuming it.
This function is not thread-safe; do not call it at the same time as another call, or at the same time this request is being used with Client.Do.
func (*Request) SetURL ¶ added in v1.0.10
SetURL updates request url (i.e http.Request.URL) with given url
type RequestLogHook ¶
RequestLogHook allows a function to run before each retry. The HTTP request which will be made, and the retry number (0 for the initial request) are available to users. The internal logger is exposed to consumers.
type ResponseLogHook ¶
ResponseLogHook is like RequestLogHook, but allows running a function on each HTTP response. This function will be invoked at the end of every HTTP request executed, regardless of whether a subsequent retry needs to be performed or not. If the response body is read or closed from this method, this will affect the response returned from Do().
Source Files
Directories