httpx

package module
v1.1.4 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Aug 20, 2021 License: MIT Imports: 11 Imported by: 7

README

httpx - Best HTTP client for reliability (retry, plugins, racing, and more!)

Build Status Go Report Card PkgGoDev

Package httpx is the best-in-class GoLang HTTP client for making HTTP requests with enterprise-level reliability.

Features include:

  • Flexible retry policies
  • Flexible timeout policies (including adaptive timeouts)
  • Optional concurrent request support smooths over rough service patches ("racing")
  • Plugin and customization support via event handlers

Getting Started

Install httpx:

$ go get github.com/gogama/httpx

Import the httpx package and create a Client to begin making reliable HTTP requests:

package main

import "github.com/gogama/httpx"

func main() {
	client := &httpx.Client{} // Use default retry and timeout policies
	client.Get("http://example.com")
}

Quick Hits

Retry

The httpx.Client provides a reasonable default retry policy. To replace it with your own custom policy, use package retry, for example

client := &httpx.Client{
	RetryPolicy: retry.NewPolicy(
		retry.Times(10).And(
			retry.StatusCode(501, 502, 504).Or(retry.TransientErr)
        ),
        retry.NewExpWaiter(500*time.Millisecond, 30*time.Second, nil)
    )
}

For more elaborate policies, write your own retry.Decider or retry.Waiter implementation. To disable retry altogether, use the built-in policy retry.Never:

Timeouts

The httpx.Client provides a reasonable default timeout policy. To replace it with your own constant timeout, adaptive timeout, or custom policy, use package timeout, for example:

client := &httpx.Client{
	TimeoutPolicy: timeout.Fixed(30*time.Second) // Constant 30 second timeout
}

For more elaborate timeouts, use timeout.Adaptive or write your own timeout.Policy implementation. To disable timeouts altogether, use the built-in policy timeout.Infinite:

Concurrent requests ("racing")

The httpx.Client advanced racing feature is disabled by default. Enable it by specifying a racing policy using built in components from package racing, or by writing your own racing.Scheduler and racing.Starter implementations. Here is a simple example using the built-ins:

client := &httpx.Client{
	// Use up to two extra parallel request attempts. Start the first extra attempt
	// if the response to the initial attempt is not received within 300ms. Start
	// the second extra attempt if neither the initial attempt nor the first extra
	// attempt have received a response after one second. 
	RacingPolicy: racing.NewPolicy( 
		racing.NewStaticScheduler(300*time.Millisecond, 1*time.Second),
		racing.AlwaysStart)
}

More Info

See the USAGE.md for a more detailed usage guide and FAQ.md for answers to frequently asked questions, or click here for the full httpx API reference documentation.

Plugins

Customize the behavior of httpx.Client by adding event handlers to the client's handler group.

handlers := &httpx.HandlerGroup{}
handlers.PushBack(httpx.BeforeReadBody, myReadBodyHandler)
client := &httpx.Client{Handlers: handlers}

Besides writing your own, you can add install one of the following open source httpx plugins into your client:

  1. aws-xray-httpx - Adds AWS X-Ray tracing support into httpx.Client.
  2. reconnx - Discards slow HTTP connections from connection pool.

License

This project is licensed under the terms of the MIT License.

Acknowledgements

Developer happiness on this project was boosted by JetBrains' generous donation of an open source license for their lovely GoLand IDE. ❤

Documentation

Overview

Package httpx provides a robust HTTP client with retry support and other advanced features within a simple and familiar interface.

Create a Client to begin making requests. 

client := &httpx.Client{}
ex, err := client.Get("https://www.example.com")
...
ex, err := client.Post("https://www.example.com/upload",
	"application/json", &buf)
...
ex, err := client.PostForm("http://example.com/form",
	url.Values{"key": {"Value"}, "id": {"123"}})

For control over how the client sends HTTP requests and receives HTTP responses, use a custom HTTPDoer. For example, use a GoLang standard HTTP client: 

doer := &http.Client{
	..., // See package "net/http" for detailed documentation
}
client := &httpx.Client{
	HTTPDoer: doer,
}

For control over the client's retry decisions and timing, create a custom retry policy using components from package retry: 

retryWaiter := retry.NewExpWaiter(250*time.Millisecond, 5*time.Second(), time.Now())
retryPolicy := retry.NewPolicy(retry.DefaultDecider, retryWaiter)
client := httpx.Client{
	RetryPolicy: retryPolicy,
}

For control over the client's individual attempt timeouts, set a custom timeout policy using package timeout: 

client := &httpx.Client{
	TimeoutPolicy: timeout.Fixed(10*time.Second)
}

To hook into the fine-grained details of the client's request execution logic, install a handler into the appropriate handler chain: 

log := log.New(os.Stdout, "", log.LstdFlags)
handlers := &httpx.HandlerGroup{}
handlers.PushBack(httpx.BeforeAttempt, httpx.HandlerFunc(
	func(_ httpx.Event, e *request.Execution) {
		log.Printf("Attempt %d to %s", e.Attempt, e.Request.URL.String())
	})
)
client := &httpx.Client{
	HTTPDoer: doer,
	Handlers: handlers,
}

Package httpx provides basic interfaces for each method of the robust client (Doer, Getter, Header, Poster, FormPoster, and IdleCloser); a combined interface that composes all the basic methods (Executor); and utility functions for working with a Doer (Inflate, Get, Head, Post, and PostForm).

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func Get 

func Get(d Doer, url string) (*request.Execution, error)

Get uses the specified Doer to issue a GET to the specified URL, using the same policies as d.Do.

To make a request plan with custom headers, use request.NewPlan and d.Do. 

func Head(d Doer, url string) (*request.Execution, error)

Head uses the specified Doer to issue a HEAD to the specified URL, using the same policies as d.Do.

To make a request plan with custom headers, use request.NewPlan and d.Do.

func Post 

func Post(d Doer, url, contentType string, body interface{}) (*request.Execution, error)

Post uses the specified Doer to issue a POST to the specified URL, using the same policies as d.Do.

The body parameter may be nil for an empty body, or may be any of the types supported by Client.Post, request.NewPlan, and request.BodyBytes, namely: string; []byte; io.Reader; and io.ReadCloser.

To make a request plan with custom headers, use request.NewPlan and d.Do.

func PostForm 

func PostForm(d Doer, url string, data url.Values) (*request.Execution, error)

PostForm uses the specified Doer to issue a POST to the specified URL, with data's keys and values URL-encoded as the request body.

The Content-Type header is set to application/x-www-form-urlencoded. To set other headers, use request.NewPlan and d.Do.

Types

type Client 

type Client struct {
	// HTTPDoer specifies the mechanics of sending HTTP requests and
	// receiving responses.
	//
	// If HTTPDoer is nil, http.DefaultClient from the standard net/http
	// package is used.
	HTTPDoer HTTPDoer
	// TimeoutPolicy specifies how to set timeouts on individual request
	// attempts.
	//
	// If TimeoutPolicy is nil, timeout.DefaultPolicy is used.
	TimeoutPolicy timeout.Policy
	// RetryPolicy decides when to retry failed attempts and how long
	// to sleep after a failed attempt before retrying.
	//
	// If RetryPolicy is nil, retry.DefaultPolicy is used.
	RetryPolicy retry.Policy
	// RacingPolicy specifies how to race concurrent requests if this
	// advanced feature is desired.
	//
	// If RacingPolicy is nil, racing.Disabled, which never races
	// concurrent requests, is used.
	RacingPolicy racing.Policy
	// Handlers allows custom handler chains to be invoked when
	// designated events occur during execution of a request plan.
	//
	// If Handlers is nil, no custom handlers will be run.
	Handlers *HandlerGroup
}

A Client is a robust HTTP client with retry support. Its zero value is a valid configuration.

The zero value client uses http.DefaultClient (from net/http) as the HTTPDoer, timeout.DefaultPolicy as the timeout policy, retry.DefaultPolicy as the retry policy, and an empty handler group (no event handlers/plug-ins).

Client's HTTPDoer typically has an internal state (cached TCP connections) so Client instances should be reused instead of created as needed. Client is safe for concurrent use by multiple goroutines.

A Client is higher-level than an HTTPDoer. The HTTPDoer is responsible for all details of sending the HTTP request and receiving the response, while Client builds on top of the HTTPDoer's feature set. For example, the HTTPDoer is responsible for redirects, so consult the HTTPDoer's documentation to understand how redirects are handled. Typically the Go standard HTTP client http.Client) will be used as the HTTPDoer, but this is not required.

On top of the HTTP request features provided by the HTTPDoer, Client adds the following features:

• Client reads and buffers the entire HTTP response body into a []byte (returned as the Execution.Body field);

• Client retries failed request attempts using a customizable retry policy;

• Client sets individual request attempt timeouts using a customizable timeout policy;

• Client may race multiple concurrent HTTP request attempts against one another to improve performance using a customizable racing policy (see package racing for more detail);

• Client invokes user-provided handler functions at designated plug-in points within the attempt/retry loop, allowing new features to be mixed in from outside libraries; and

• Client implements the httpx.Executor interface.

Client's HTTP methods should feel familiar to anyone who has used the Go standard HTTP client (http.Client). The methods use the same names, and follow the same rough parameter schema, as the Go standard client. The main differences are:

• instead of consuming an http.Request, which is only suitable for making a one-off request attempt, Client.Do consumes a request.Plan which is suitable for making multiple attempts if necessary (the plan execution logic converts the plan into http.Request as needed); and

• instead of producing an http.Response, all of Client's HTTP methods return a request.Execution, which contains some metadata about the plan execution as well as a fully-buffered response body.

func (*Client) CloseIdleConnections 

func (c *Client) CloseIdleConnections()

CloseIdleConnections invokes the same method on the client's underlying HTTPDoer.

If the HTTPDoer has no CloseIdleConnections method, this method does nothing.

If the HTTPDoer does have a CloseIdleConnections method, then the effect of this method depends entirely on its implementation in the HTTPDoer. For example, the http.Client type forwards the call to its Transport, but only if the Transport itself has a CloseIdleConnections method (otherwise it does nothing).

func (*Client) Do 

func (c *Client) Do(p *request.Plan) (*request.Execution, error)

Do executes an HTTP request plan and returns the results, following timeout and retry policy set on Client, and low-level policy set on the underlying HTTPDoer.

The result returned is the result after the final HTTP request attempt made during the plan execution, as determined by the retry policy.

An error is returned if, after doing any retries mandated by the retry policy, the final attempt resulted in an error. An attempt may end in error due to failure to speak HTTP (for example a network connectivity problem), or because of policy in the robust client (such as timeout), or because of policy on the underlying HTTPDoer (for example relating to redirects body). A non-2XX status code in the final attempt does not result in an error.

The returned Execution is never nil, but may contain a nil Response and will contain a nil Body if an error occurred (if the initial HTTP request caused an error, both Response and Body are nil, but if the initial HTTP request succeeded and the error occurred while reading Body from the request, then Response is non-nil but body is nil). If an error was returned, the Err field of the Execution always references the same error.

If the returned error is nil, the returned Execution will contain both a non-nil Response and a non-nil Body (although Body may have zero length).

Any returned error will be of type *url.Error. The url.Error's Timeout method, and the Execution's Timeout method, will return true if the final request attempt timed out, or if the entire plan timed out.

For simple use cases, the Get, Head, Post, and PostForm methods may prove easier to use than Do.

func (*Client) Get 

func (c *Client) Get(url string) (*request.Execution, error)

Get issues a GET to the specified URL, using the same policies followed by Do.

To make a request plan with custom headers, use request.NewPlan and Client.Do.

func (*Client) Head 

func (c *Client) Head(url string) (*request.Execution, error)

Head issues a HEAD to the specified URL, using the same policies followed by Do.

To make a request plan with custom headers, use request.NewPlan and Client.Do.

func (*Client) Post 

func (c *Client) Post(url, contentType string, body interface{}) (*request.Execution, error)

Post issues a POST to the specified URL, using the same policies followed by Do.

The body parameter may be nil for an empty body, or may be any of the types supported by request.NewPlan, request.BodyBytes, and httpx.Post, namely: string; []byte; io.Reader; and io.ReadCloser.

To make a request plan with custom headers, use request.NewPlan and Client.Do.

func (*Client) PostForm 

func (c *Client) PostForm(url string, data url.Values) (*request.Execution, error)

PostForm issues a POST to the specified URL, with data's keys and values URL-encoded as the request body.

The Content-Type header is set to application/x-www-form-urlencoded. To set other headers, use request.NewPlan and Client.Do.

type Doer 

type Doer interface {
	Do(p *request.Plan) (*request.Execution, error)
}

Doer is the interface that wraps the basic Do method.

Do executes an HTTP request plan and returns the final execution state (and error, if any). Client implements the Doer interface, and any other Doer implementation must behave substantially the same as Client.Do.

Any Doer can be converted into an Executor via the Inflate function.

type Event 

type Event int

An Event identifies the event type when installing or running a Handler. Install event handlers in a Client to extend it with custom functionality. 

const (
	// BeforeExecutionStart identifies the event that occurs before the
	// plan execution starts.
	//
	// When Client fires BeforeExecutionStart, the execution is
	// non-nil but the only field that has been set is the plan.
	BeforeExecutionStart Event = iota
	// BeforeAttempt identifies the event that occurs before each
	// individual HTTP request attempt during the plan execution.
	//
	// When Client fires BeforeAttempt, the execution's request
	// field is set to the HTTP request that WILL BE sent after all
	// BeforeAttempt handlers have finished.
	//
	// BeforeAttempt Handlers may modify the execution's request, or
	// some of its fields, thus changing the HTTP request that will be
	// sent. However, BeforeAttempt handlers should clone request fields
	// which have reference types (URL and Header) before changing them
	// to avoid side effects, as these fields initially reference the
	// same-named fields in the plan.
	BeforeAttempt
	// BeforeReadBody identifies the event that occurs after an HTTP
	// request attempt has resulted in an HTTP response (as opposed to
	// an error) but before the response body is read and buffered.
	//
	// When Client fires BeforeReadyBody, the execution's
	// response field is set to the HTTP response whose body WILL BE
	// read after all BeforeReadBody handlers have finished (however,
	// handlers may modify this field).
	//
	// Note that BeforeReadBody never fires if the HTTP request attempt
	// ended in error, but always fires if an HTTP response is received,
	// regardless of HTTP response status code, and regardless of
	// whether there is a non-empty body in the request.
	BeforeReadBody
	// AfterAttemptTimeout identifies the event that occurs after an
	// HTTP request attempt failed because of a timeout error.
	//
	// When Client fires AfterAttemptTimeout, the execution's
	// error field is set to the timeout error, and its attempt timeout
	// counter has been incremented.
	AfterAttemptTimeout
	// AfterAttempt identifies the event that occurs after an HTTP
	// request attempt is concluded, regardless of whether it concluded
	// successfully or not.
	//
	// When Client fires AfterAttempt, either the execution's
	// response field or its error field OR BOTH may be set to non-nil
	// values, but it will never be the case that both are nil. The
	// response will only be non-nil when the error is also non-nil if
	// there was an error reading the response body.
	//
	// Note that AfterAttempt always fires on every HTTP request attempt,
	// regardless of whether it ended in error, and that it runs before
	// the retry policy is consulted for a retry decision.
	AfterAttempt
	// AfterPlanTimeout identifies the event that occurs after a timeout
	// on the request plan level, not just the request attempt level
	// (i.e. the context deadline on the plan's context is exceeded).
	// A plan timeout can be detected either at the same time as an
	// attempt timeout, or during the retry wait period.
	//
	// When Client fires AfterPlanTimeout, the execution's
	// response and body fields are both nil.
	//
	// Note that AfterPlanTimeout always occurs after AfterAttempt,
	// even if the plan timeout was actually detected at the same time
	// as an attempt timeout.
	AfterPlanTimeout
	// AfterExecutionEnd identifies the event that occurs after the plan
	// execution ends.
	//
	// When Client fires AfterExecutionEnd, the execution is in
	// the same state it was in after the final HTTP request attempt
	// (and last AfterAttempt event) EXCEPT that the end time is set to
	// the time the execution ended.
	AfterExecutionEnd
)

func Events 

func Events() []Event

Events returns a slice containing all events which can occur in an HTTP request plan execution by Client, in the order in which they would occur.

func (Event) Name deprecated 

func (evt Event) Name() string

Name returns the name of the event.

Deprecated: This method is redundant with String and will be removed in the near future.

func (Event) String added in v1.1.2 

func (evt Event) String() string

String returns the name of the event.

type Executor 

type Executor interface {
	Doer
	Getter
	Header
	Poster
	FormPoster
	IdleCloser
}

Executor is the interface that groups the basic Do, Get, Head, Post, PostForm, and CloseIdleConnections methods.

Any Doer can be converted into an Executor via the Inflate function.

func Inflate 

func Inflate(d Doer) Executor

Inflate converts any non-nil Doer into an Executor. This may be helpful for interop across library boundaries, i.e. if code that only has access to a Doer needs to call a function that requires an Executor.

type FormPoster 

type FormPoster interface {
	PostForm(url string, data url.Values) (*request.Execution, error)
}

FormPoster is the interface that wraps the basic PostForm method.

PostForm creates an HTTP request plan to issue a form POST to the specified URL, executes the plan, and returns the final execution state (and error, if any). Client implements the FormPoster interface, and any other FormPoster implementation must behave substantially the same as Client.PostForm.

The request plan body is set to the URL-encoded keys and values from data, and the content type is set to application/x-www-form-urlencoded.

Any Doer can be used to emulate a FormPoster via the PostForm function.

type Getter 

type Getter interface {
	Get(url string) (*request.Execution, error)
}

Getter is the interface that wraps the basic Get method.

Get creates an HTTP request plan to issue a GET to the specified URL, executes the plan, and returns the final execution state (and error, if any). Client implements the Getter interface, and any other Getter implementation must behave substantially the same as Client.Get.

Any Doer can be used to emulate a Getter via the Get function.

type HTTPDoer 

type HTTPDoer interface {
	// Do sends an HTTP request and returns an HTTP response following
	// policy (such as redirects, cookies, auth) configured on the
	// HTTPDoer.
	//
	// The Do method must follow the contract documented on the GoLang
	// standard library http.Client from the net/http package.
	Do(r *http.Request) (*http.Response, error)
}

An HTTPDoer implements a Do method in the same manner as the GoLang standard library http.Client from the net/http package.

type Handler 

type Handler interface {
	Handle(Event, *request.Execution)
}

A Handler handles the occurrence of an event during a request plan execution. Install event handlers in a Client to extend the Client with custom functionality.

type HandlerFunc 

type HandlerFunc func(Event, *request.Execution)

The HandlerFunc type is an adapter to allow the use of ordinary functions as event handlers. If f is a function with appropriate signature, then HandlerFunc(f) is a Handler that calls f.

func (HandlerFunc) Handle 

func (f HandlerFunc) Handle(evt Event, e *request.Execution)

Handle calls f(evt, e).

type HandlerGroup 

type HandlerGroup struct {
	// contains filtered or unexported fields
}

A HandlerGroup is a group of event handler chains which can be installed in a Client. Install event handlers to extend Client with custom functionality.

func (*HandlerGroup) PushBack 

func (g *HandlerGroup) PushBack(evt Event, h Handler)

PushBack adds an event handler to the back of the event handler chain for a specific event type. 

type Header interface {
	Head(url string) (*request.Execution, error)
}

Header is the interface that wraps the basic Head method.

Head creates an HTTP request plan to issue a HEAD to the specified URL, executes the plan, and returns the final execution state (and error, if any). Client implements the Header interface, and any other Header implementation must behave substantially the same as Client.Head.

Any Doer can be used to emulate a Header via the Head function.

type IdleCloser 

type IdleCloser interface {
	CloseIdleConnections()
}

IdleCloser is the interface that wraps the basic CloseIdleConnections method.

If the underlying implementation supports it, CloseIdleConnections closes any idle which were previously connected from previous requests but are now sitting idle in a "keep-alive" state. It does not interrupt any connections currently in use.

If the underlying implementation does not support this ability, CloseIdleConnections does nothing.

type Poster 

type Poster interface {
	Post(url, contentType string, body interface{}) (*request.Execution, error)
}

Poster is the interface that wraps the basic Post method.

Post creates an HTTP request plan to issue a POST to the specified URL, executes the plan, and returns the final execution state (and error, if any). Client implements the Poster interface, and any other Poster implementation must behave substantially the same as Client.Post.

The body parameter may be nil for an empty body, or may be any of the types supported by request.NewPlan, request.BodyBytes, and httpx.Post, namely: string; []byte; io.Reader; and io.ReadCloser.

Any Doer can be used to emulate a Poster via the Post function.

Source Files

View all Source files

Directories

Path Synopsis
Package racing provides flexible policies for running multiple HTTP requests simultaneously in a "race" to improve performance.
 Package racing provides flexible policies for running multiple HTTP requests simultaneously in a "race" to improve performance.
Package request contains the core types Plan (describes an HTTP request plan) and Execution (describes a Plan execution).
 Package request contains the core types Plan (describes an HTTP request plan) and Execution (describes a Plan execution).
Package retry provides flexible policies for retrying failed attempts during an HTTP request plan execution, and how long to wait before retrying.
 Package retry provides flexible policies for retrying failed attempts during an HTTP request plan execution, and how long to wait before retrying.
Package timeout defines flexible policies for setting HTTP timeouts during an HTTP request plan execution, including on retries.
 Package timeout defines flexible policies for setting HTTP timeouts during an HTTP request plan execution, including on retries.
Package transient classifies errors from HTTP request execution as transient or non-transient.
 Package transient classifies errors from HTTP request execution as transient or non-transient.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.