httpclient

README

httpclient

Basic HTTP client functionality that I found myself writing over and over again.

Documentation

• pkg.go.dev

Authors

• Christophe Lambin

License

This project is licensed under the MIT License - see the LICENSE.md file for details.

Documentation

Overview

Package httpclient provides a set of http.RoundTripper implementations that extend http.Client's transport layer.

RoundTrippers can be chained together to create a complex client middleware that is transparent at the application layer.

Example (Chained)

package main

import (
	"fmt"
	"github.com/clambin/go-common/httpclient"
	"github.com/prometheus/client_golang/prometheus"
	"io"
	"net/http"
	"time"
)

func main() {
	tp := httpclient.NewRoundTripper(
		httpclient.WithCache(httpclient.DefaultCacheTable, time.Second, time.Minute),
		httpclient.WithMetrics("foo", "bar", "example"),
	)

	// The RoundTripper needs to be registered with a Prometheus registry so Prometheus will collect it.
	prometheus.MustRegister(tp)

	c := http.Client{Transport: tp}

	if resp, err := c.Get("https://example.com"); err == nil {
		body, _ := io.ReadAll(resp.Body)
		fmt.Print(string(body))
		_ = resp.Body.Close()
	}
}

Index

• type CacheTable
• type CacheTableEntry
• type Option
  • func WithCache(table CacheTable, defaultExpiry, cleanupInterval time.Duration) Option
  • func WithCustomMetrics(o RequestMeasurer) Option
  • func WithInstrumentedCache(table CacheTable, defaultExpiry, cleanupInterval time.Duration, ...) Option
  • func WithInstrumentedLimiter(maxParallel int64, namespace, subsystem, application string) Option
  • func WithLimiter(maxParallel int64) Option
  • func WithMetrics(namespace, subsystem, application string) Option
  • func WithRoundTripper(roundTripper http.RoundTripper) Option
• type RequestMeasurer
• type RoundTripper
  • func NewRoundTripper(options ...Option) *RoundTripper
  • func (r RoundTripper) Collect(metrics chan<- prometheus.Metric)
  • func (r RoundTripper) Describe(descs chan<- *prometheus.Desc)
• type RoundTripperFunc
  • func (f RoundTripperFunc) RoundTrip(req *http.Request) (*http.Response, error)

Examples

• Package (Chained)
• RoundTripperFunc
• WithCache
• WithMetrics

Types

type CacheTable

type CacheTable []*CacheTableEntry

CacheTable holds the endpoints that should be cached. If table is empty, all responses will be cached.

var DefaultCacheTable CacheTable

DefaultCacheTable is a CacheTable that caches all requests.

type CacheTableEntry

type CacheTableEntry struct {
	// Path is the URL Path for requests whose responses should be cached.
	// Can be a literal path, or a regular expression. In the latter case, set IsRegExp to true
	Path string
	// Methods is the list of HTTP Methods for which requests the response should be cached.
	// If empty, requests for any method will be cached.
	Methods []string
	// IsRegExp indicates if the Path is a regular expression.
	// CacheTableEntry will panic if Path does not contain a valid regular expression.
	IsRegExp bool
	// Expiry indicates how long a response should be cached.
	Expiry time.Duration
	// contains filtered or unexported fields
}

CacheTableEntry contains a single endpoint that should be cached. If the Path is a regular expression, IsRegExp must be true.

type Option

type Option func(current http.RoundTripper) http.RoundTripper

Option is a function that can be passed to NewRoundTripper to specify the behaviour of the RoundTripper. See WithMetrics, WithCache, etc. for examples.

func WithCache

func WithCache(table CacheTable, defaultExpiry, cleanupInterval time.Duration) Option

WithCache creates a RoundTripper that caches the HTTP responses. table determines the caching behaviour per target path. If table is empty, all responses are cached for defaultExpiry amount of time. If table is not empty, only requests that match an entry in the table are cached, for the amount of time specified in the table's entry.

Expired entries are periodically removed from the cache as per CleanupInterval. If CleanupInterval is zero, expired entries will never be removed.

Example

package main

import (
	"fmt"
	"github.com/clambin/go-common/httpclient"
	"io"
	"net/http"
	"time"
)

func main() {
	cacheTable := []*httpclient.CacheTableEntry{
		{
			Path:     "/foo/.+",
			IsRegExp: true,
			Expiry:   5 * time.Second,
		},
	}
	c := &http.Client{
		Transport: httpclient.NewRoundTripper(httpclient.WithCache(cacheTable, time.Second, time.Minute)),
	}

	if resp, err := c.Get("https://example.com"); err == nil {
		body, _ := io.ReadAll(resp.Body)
		fmt.Print(string(body))
		_ = resp.Body.Close()
	}
}

func WithCustomMetrics

func WithCustomMetrics(o RequestMeasurer) Option

WithCustomMetrics creates a RoundTripper that measures outbound HTTP requests and exposes them to Prometheus. The provided RequestMeasurer determines which characteristics to measure and how to present them to Prometheus. NewDefaultRequestMeasurer measures the request's latency and error count.

func WithInstrumentedCache

func WithInstrumentedCache(table CacheTable, defaultExpiry, cleanupInterval time.Duration, namespace, subsystem, application string) Option

WithInstrumentedCache causes RoundTripper to cache the HTTP responses, as WithCache. Additionally, it measures how often the cache was consulted and hit as Prometheus metrics.

namespace and subsystem are prepended to the metric names, e.g. api_errors_total will be called foo_bar_api_errors_total if namespace and subsystem are set to foo and bar respectively. Application will be set in the metrics' application label.

If namespace, subsystem and application are blank, the call is equivalent to calling WithCache, i.e. no metrics are created.

func WithInstrumentedLimiter

func WithInstrumentedLimiter(maxParallel int64, namespace, subsystem, application string) Option

WithInstrumentedLimiter creates a RoundTripper that limits concurrent http requests, like WithLimiter. Additionally, it measures the current and maximum outstanding requests as Prometheus metrics.

namespace and subsystem are prepended to the metric names, e.g. api_max_inflight will be called foo_bar_api_max_inflight if namespace and subsystem are set to foo and bar respectively. Application will be set in the metrics' application label.

If namespace, subsystem and application are blank, the call is equivalent to calling WithLimiter, i.e. no metrics are created.

func WithLimiter

func WithLimiter(maxParallel int64) Option

WithLimiter creates a RoundTripper that limits the number concurrent http requests to maxParallel.

func WithMetrics

func WithMetrics(namespace, subsystem, application string) Option

WithMetrics creates a RoundTripper that measures outbound HTTP requests and exposes them to Prometheus.

namespace and subsystem are prepended to the metric names, e.g. api_latency will be called foo_bar_api_latency if namespace and subsystem are set to foo and bar respectively. Application will be set in the metrics' application label.

Example

package main

import (
	"fmt"
	"github.com/clambin/go-common/httpclient"
	"github.com/prometheus/client_golang/prometheus"
	"io"
	"net/http"
)

func main() {
	transport := httpclient.NewRoundTripper(
		httpclient.WithMetrics("", "", "example"),
	)
	client := &http.Client{
		Transport: transport,
	}

	// The RoundTripper needs to be registered with a Prometheus registry so Prometheus will collect it.
	prometheus.DefaultRegisterer.MustRegister(transport)

	if resp, err := client.Get("https://example.com"); err == nil {
		body, _ := io.ReadAll(resp.Body)
		fmt.Print(string(body))
		_ = resp.Body.Close()
	}
}

func WithRoundTripper

func WithRoundTripper(roundTripper http.RoundTripper) Option

WithRoundTripper specifies the http.RoundTripper to make the final http client call. If no WithRoundTripper is provided, RoundTripper defaults to http.DefaultTransport. Providing a nil roundTripper causes RoundTripper to panic.

type RequestMeasurer

type RequestMeasurer interface {
	MeasureRequest(*http.Request, *http.Response, error, time.Duration)
	prometheus.Collector
}

A RequestMeasurer measures performance characteristics of an outbound HTTP request.

type RoundTripper

type RoundTripper struct {
	http.RoundTripper

	http.HandlerFunc
	// contains filtered or unexported fields
}

RoundTripper implements http.RoundTripper. It implements a net/http-compatible transport layer so it can be used by http.Client's Transport.

func NewRoundTripper

func NewRoundTripper(options ...Option) *RoundTripper

NewRoundTripper returns a RoundTripper that implements the behaviour as specified by the different Option parameters. NewRoundTripper will construct a cascading roundTripper in the order of the provided options. E.g.

r := NewRoundTripper(WithMetrics(...), WithCache(...))

returns a roundTripper that measures the client metrics, and then attempts to get the response from cache. Metrics are therefor captured for cached and non-cached responses. On the other hand:

r := NewRoundTripper(WithCache(...), WithMetrics(...))

first attempts to get the response from cache, and failing that, performs real call, recording its http metrics. Metrics will therefore only be captured for real http calls.

NewRoundTripper makes no attempt to sanitize the order of the provided options. E.g. WithRoundTripper does not cascade to a next roundTripper; it directly performs the http call using the provided transport. Therefore, if we create the following RoundTripper:

r := NewRoundTripper(WithRoundTripper(...), WithMetrics(...))

the WithMetrics option will not be used at all.

If no options are provided, or the final option isn't WithRoundTripper, the http call is done using http.DefaultTransport.

deprecated: use github.com/clambin/go-common/http/roundtripper instead

func (RoundTripper) Collect

func (r RoundTripper) Collect(metrics chan<- prometheus.Metric)

Collect implements the prometheus.Collector interface. A RoundTripper can therefore be directly registered and collected from a Prometheus registry.

func (RoundTripper) Describe

func (r RoundTripper) Describe(descs chan<- *prometheus.Desc)

Describe implements the prometheus.Collector interface. A RoundTripper can therefore be directly registered and collected from a Prometheus registry.

type RoundTripperFunc

type RoundTripperFunc func(*http.Request) (*http.Response, error)

The RoundTripperFunc type is an adapter to allow the use of ordinary functions as HTTP roundTrippers. If f is a function with the appropriate signature, RoundTripperFunc(f) is a roundTripper that calls f.

Example

package main

import (
	"bytes"
	"fmt"
	"github.com/clambin/go-common/httpclient"
	"io"
	"net/http"
)

func main() {
	stub := func(_ *http.Request) (*http.Response, error) {
		return &http.Response{
			StatusCode: http.StatusOK,
			Body:       io.NopCloser(bytes.NewBufferString(`hello`)),
		}, nil
	}
	tp := httpclient.NewRoundTripper(httpclient.WithRoundTripper(httpclient.RoundTripperFunc(stub)))

	c := http.Client{Transport: tp}

	if resp, err := c.Get("/"); err == nil {
		body, _ := io.ReadAll(resp.Body)
		fmt.Print(string(body))
		_ = resp.Body.Close()
	}
}

func (RoundTripperFunc) RoundTrip

func (f RoundTripperFunc) RoundTrip(req *http.Request) (*http.Response, error)

RoundTrip calls f(req)