hqgohttp

package module
v0.0.0-...-fdb48fa Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 24, 2023 License: MIT Imports: 25 Imported by: 2

README

hqgohttp

made with go license maintenance open issues closed issues contribution

Resource

Contribution

Issues and Pull Requests are welcome! Check out the contribution guidelines.

Licensing

This utility is distributed under the MIT license.

Documentation

Index

Constants

This section is empty.

Variables

View Source 
var DefaultOptionsSingle = &Options{
	RetryWaitMin:    1 * time.Second,
	RetryWaitMax:    30 * time.Second,
	Timeout:         30 * time.Second,
	RetryMax:        4,
	RespReadLimit:   4096,
	KillIdleConn:    false,
	NoAdjustTimeout: true,
}

DefaultOptionsSingle is an instance of Options with default values suitable for "host brute force" scenarios, where lots of requests need to be sent to a single host. For example, it sets KillIdleConn to false to allow keep-alive connections, as they can improve performance when connecting repeatedly to the same host.

View Source 
var DefaultOptionsSpraying = &Options{
	RetryWaitMin:    1 * time.Second,
	RetryWaitMax:    30 * time.Second,
	Timeout:         30 * time.Second,
	RetryMax:        4,
	RespReadLimit:   4096,
	KillIdleConn:    true,
	NoAdjustTimeout: true,
}

DefaultOptionsSpraying is an instance of Options with default values suitable for "host spraying" scenarios, where lots of requests need to be sent to different hosts. For example, it sets KillIdleConn to true to kill all keep-alive connections, as they are not useful when connecting to many different hosts.

Functions

func CheckRecoverableErrors 

func CheckRecoverableErrors(ctx context.Context, _ *http.Response, err error) (bool, error)

CheckRecoverableErrors checks if an error is recoverable and decides whether to retry the request. The conditions it checks are: 1. If the context has been canceled or its deadline has been exceeded, it doesn't retry. 2. If the error is related to too many redirects or an unsupported protocol scheme, it doesn't retry. 3. If the error is due to a TLS certificate verification failure (specifically an unknown authority error), it doesn't retry. If none of the above conditions are met, it considers the error as likely recoverable and decides to retry.

func DefaultBackoff 

func DefaultBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

DefaultBackoff provides a callback for client.Backoff implements the standard exponential backoff without jitter. i.e The delay between retries is doubled with each attempt, up to a maximum delay.

func DefaultHTTPClient 

func DefaultHTTPClient() *http.Client

DefaultHTTPClient returns a new http.Client with similar default values to http.Client, but with a non-shared transport, idle connections disabled, and keep-alives disabled. It does this by setting the Transport field of the http.Client struct to the transport returned by DefaultHTTPTransport.

func DefaultHTTPPooledTransport 

func DefaultHTTPPooledTransport() (transport *http.Transport)

DefaultHTTPPooledTransport returns a new http.Transport with similar default values to http.DefaultTransport, but with a custom configuration that is suitable for transports that will be reused for the same hosts. It sets various fields of the http.Transport struct, such as Proxy, DialContext, MaxIdleConns, IdleConnTimeout, TLSHandshakeTimeout, ExpectContinueTimeout, ForceAttemptHTTP2, and MaxIdleConnsPerHost.

Do not use this for transient transports as it can leak file descriptors over time. Only use this for transports that will be re-used for the same host(s).

func DefaultHTTPTransport 

func DefaultHTTPTransport() (transport *http.Transport)

DefaultHTTPTransport returns a new http.Transport with similar default values to http.DefaultTransport, but with idle connections and keepalives disabled. It does this by first creating a transport with pooled connections (by calling DefaultHTTPPooledTransport) and then setting DisableKeepAlives to true and MaxIdleConnsPerHost to -1.

func DefaultPooledClient 

func DefaultPooledClient() *http.Client

DefaultPooledClient returns a new http.Client with similar default values to http.Client, but with a shared transport. It sets the Transport field of the http.Client struct to the transport returned by DefaultHTTPPooledTransport.

Do not use this function for transient clients as it can leak file descriptors over time. Only use this for clients that will be re-used for the same host(s).

func DefaultRetryPolicy 

func DefaultRetryPolicy() func(ctx context.Context, resp *http.Response, err error) (bool, error)

DefaultRetryPolicy provides a default callback for client.CheckRetry, which will retry on connection errors and server errors.

func ExponentialJitterBackoff 

func ExponentialJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

ExponentialJitterBackoff provides a callback for Client.Backoff which will perform an exponential backoff based on the attempt number and with jitter to prevent a thundering herd.

min and max here are *not* absolute values. The number to be multiplied by the attempt number will be chosen at random from between them, thus they are bounding the jitter.

func FullJitterBackoff 

func FullJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

FullJitterBackoff provides a callback for client.Backoff which implements a variation of exponential backoff with full jitter. i.e Instead of doubling the delay with each attempt, it randomizes the delay completely within the exponential window.

Algorithm is fast because it does not use floating point arithmetics. It returns a random number between [0...n] https://aws.amazon.com/blogs/architecture/exponential-backoff-and-jitter/

func Get 

func Get(URL string) (*http.Response, error)

Get issues a GET to the specified URL. 

func Head(URL string) (*http.Response, error)

Head issues a HEAD to the specified URL.

func HostSprayRetryPolicy 

func HostSprayRetryPolicy() func(ctx context.Context, resp *http.Response, err error) (bool, error)

HostSprayRetryPolicy provides a callback for client.CheckRetry, which will retry on connection errors and server errors.

func LinearJitterBackoff 

func LinearJitterBackoff() func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

LinearJitterBackoff provides a callback for client.Backoff which implements linear backoff with jitter. i.e The delay between retries is increased linearly with each attempt, but a random jitter is added to this delay.

This jitter helps in distributed systems to avoid situations where many clients retry simultaneously, commonly known as "thundering herd".

min and max here are *not* absolute values. The number to be multiplied by the attempt number will be chosen at random from between them, thus they are bounding the jitter.

For instance: - To get strictly linear backoff of one second increasing each retry, set both to one second (1s, 2s, 3s, 4s, ...) - To get a small amount of jitter centered around one second increasing each retry, set to around one second, such as a min of 800ms and max of 1200ms (892ms, 2102ms, 2945ms, 4312ms, ...) - To get extreme jitter, set to a very wide spread, such as a min of 100ms and a max of 20s (15382ms, 292ms, 51321ms, 35234ms, ...)

func Post 

func Post(URL, bodyType string, body interface{}) (*http.Response, error)

Post issues a POST to the specified URL.

func PostForm 

func PostForm(URL string, data url.Values) (*http.Response, error)

PostForm issues a POST to the specified URL, with data's keys and values

Types

type Auth 

type Auth struct {
	Type     AuthType
	Username string
	Password string
}

Auth specific information

type AuthType 

type AuthType uint8
const (
	DigestAuth AuthType = iota
)

type Backoff 

type Backoff func(min, max time.Duration, attemptNum int, resp *http.Response) time.Duration

Backoff specifies a policy for how long to wait between retries.

type CheckRetry 

type CheckRetry func(ctx context.Context, resp *http.Response, err error) (bool, error)

CheckRetry specifies a policy for handling retries. It is called following each request with the response and error values returned by the http.Client. If CheckRetry returns false, the Client stops retrying and returns the response to the caller. If CheckRetry returns an error, that error value is returned in lieu of the error from the request. The Client will close any response body when retrying, but if the retry is aborted it is up to the CheckRetry callback to properly close any response body before returning.

type Client 

type Client struct {
	// HTTPClient is the internal HTTP client (http1x + http2 via connection upgrade upgrade).
	HTTPClient *http.Client
	// HTTP2Client is the internal HTTP client configured to fallback to native http2 at transport level
	HTTP2Client *http.Client
	// RequestLogHook allows a user-supplied function to be called before each retry.
	RequestLogHook RequestLogHook
	// ResponseLogHook allows a user-supplied function to be called with the response from each HTTP request executed.
	ResponseLogHook ResponseLogHook
	// ErrorHandler specifies the custom error handler to use, if any
	ErrorHandler ErrorHandler
	// CheckRetry specifies the policy for handling retries, and is called after each request
	CheckRetry CheckRetry
	// Backoff specifies the policy for how long to wait between retries
	Backoff Backoff
	// contains filtered or unexported fields
}

Client represents the main HTTP client. It is used to make HTTP requests and adds additional functionality like automatic retries to tolerate minor outages. 

var DefaultClient *Client

DefaultClient is the http client with DefaultOptionsSingle options.

func New 

func New(options *Options) (client *Client, err error)

New creates a new client instance based on provided options. It configures the internal HTTP clients, sets up HTTP/2 for the second client, applies retry and backoff policies, and Adjusts client timeouts and other settings based on the provided options.

func (*Client) Do 

func (c *Client) Do(req *Request) (res *http.Response, err error)

Do wraps calling an HTTP method with retries.

func (*Client) Get 

func (c *Client) Get(URL string) (*http.Response, error)

Get is a convenience helper for doing simple GET requests.

func (*Client) Head 

func (c *Client) Head(URL string) (*http.Response, error)

Head is a convenience method for doing simple HEAD requests.

func (*Client) Post 

func (c *Client) Post(URL, bodyType string, body interface{}) (*http.Response, error)

Post is a convenience method for doing simple POST requests.

func (*Client) PostForm 

func (c *Client) PostForm(URL string, data url.Values) (*http.Response, error)

PostForm is a convenience method for doing simple POST operations using pre-filled url.Values form data.

type ContextOverride 

type ContextOverride string
const (
	RetryMax ContextOverride = "retry-max"
)

type ErrorHandler 

type ErrorHandler func(resp *http.Response, err error, numTries int) (*http.Response, error)

ErrorHandler is called if retries are expired, containing the last status from the http library. If not specified, default behavior for the library is to close the body and return an error indicating how many tries were attempted. If overriding this, be sure to close the body if needed.

type Metrics 

type Metrics struct {
	// Failures is the number of failed requests
	Failures int
	// Retries is the number of retries for the request
	Retries int
	// DrainErrors is number of errors occurred in draining response body
	DrainErrors int
}

Metrics contains the metrics about each request

type Options 

type Options struct {
	// Custom http client
	HTTPClient *http.Client
	// KillIdleConn specifies if all keep-alive connections gets killed
	KillIdleConn bool
	// RespReadLimit is the maximum HTTP response size to read for connection being reused.
	RespReadLimit int64
	// Timeout is the maximum time to wait for the request
	Timeout time.Duration
	// NoAdjustTimeout disables automatic adjustment of HTTP request timeout
	NoAdjustTimeout bool

	// Custom CheckRetry policy
	CheckRetry CheckRetry
	// RetryMax is the maximum number of retries
	RetryMax int
	// Custom Backoff policy
	Backoff Backoff
	// RetryWaitMin is the minimum time to wait for retry
	RetryWaitMin time.Duration
	// RetryWaitMax is the maximum time to wait for retry
	RetryWaitMax time.Duration

	// Verbose specifies if debug messages should be printed
	Verbose bool
}

Options represents configuration fields to customize the behavior of the HTTP client

type Request 

type Request struct {
	// Embed an HTTP request directly. This makes a *Request act exactly
	// like an *http.Request so that all meta methods are supported.
	*http.Request

	// Metrics contains the metrics for the request.
	Metrics Metrics

	Auth *Auth
}

Request wraps the metadata needed to create HTTP requests. Request is not threadsafe. A request cannot be used by multiple goroutines concurrently.

func FromRequest 

func FromRequest(r *http.Request) (*Request, error)

FromRequest wraps an http.Request in a client.Request

func FromRequestWithTrace 

func FromRequestWithTrace(r *http.Request) (*Request, error)

FromRequestWithTrace wraps an http.Request in a client.Request with trace enabled

func NewRequest 

func NewRequest(method, url string, body interface{}) (*Request, error)

NewRequest creates a new wrapped request

func NewRequestFromURL 

func NewRequestFromURL(url, method string, body interface{}) (*Request, error)

NewRequest creates a new wrapped request.

func NewRequestFromURLWithContext 

func NewRequestFromURLWithContext(ctx context.Context, url, method string, body interface{}) (*Request, error)

NewRequestWithContext creates a new wrapped request with context

func NewRequestWithContext 

func NewRequestWithContext(ctx context.Context, method, url string, body interface{}) (*Request, error)

NewRequest creates a new wrapped request with given context

func (*Request) BodyBytes 

func (r *Request) BodyBytes() ([]byte, error)

BodyBytes allows accessing the request body. It is an analogue to http.Request's Body variable, but it returns a copy of the underlying data rather than consuming it.

This function is not thread-safe; do not call it at the same time as another call, or at the same time this request is being used with Client.Do.

func (*Request) Clone 

func (r *Request) Clone(ctx context.Context) *Request

Clones and returns new Request

func (*Request) Dump 

func (r *Request) Dump() ([]byte, error)

Dump returns request dump in bytes

func (*Request) WithContext 

func (r *Request) WithContext(ctx context.Context) *Request

WithContext returns wrapped Request with a shallow copy of underlying *http.Request with its context changed to ctx. The provided ctx must be non-nil.

type RequestLogHook 

type RequestLogHook func(*http.Request, int)

RequestLogHook allows a function to run before each retry. The HTTP request which will be made, and the retry number (0 for the initial request) are available to users. The internal logger is exposed to consumers.

type ResponseLogHook 

type ResponseLogHook func(*http.Response)

ResponseLogHook is like RequestLogHook, but allows running a function on each HTTP response. This function will be invoked at the end of every HTTP request executed, regardless of whether a subsequent retry needs to be performed or not. If the response body is read or closed from this method, this will affect the response returned from Do().

Source Files

View all Source files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.