rest

README

REST helpers and middleware

Install and update

go get -u github.com/go-pkgz/rest

Middlewares

AppInfo middleware

Adds info to every response header:

• App-Name - application name
• App-Version - application version
• Org - organization
• M-Host - host name from instance-level $MHOST env

Ping-Pong middleware

Responds with pong on GET /ping. Also, responds to anything with /ping suffix, like /v2/ping.

Example for both:

> http GET https://remark42.radio-t.com/ping

HTTP/1.1 200 OK
Date: Sun, 15 Jul 2018 19:40:31 GMT
Content-Type: text/plain
Content-Length: 4
Connection: keep-alive
App-Name: remark42
App-Version: master-ed92a0b-20180630-15:59:56
Org: Umputun

pong

Health middleware

Responds with the status 200 if all health checks passed, 503 if any failed. Both health path and check functions passed by consumer. For production usage this middleware should be used with throttler/limiter and, optionally, with some auth middlewares

Example of usage:

    check1 := func(ctx context.Context) (name string, err error) {
        // do some check, for example check DB connection		
		return "check1", nil // all good, passed
    }
    check2 := func(ctx context.Context) (name string, err error) {
        // do some other check, for example ping an external service		
		return "check2", errors.New("some error") // check failed
    }

    router := chi.NewRouter()
	router.Use(rest.Health("/health", check1, check2))

example of the actual call and response:

> http GET https://example.com/health

HTTP/1.1 503 Service Unavailable
Date: Sun, 15 Jul 2018 19:40:31 GMT
Content-Type: application/json; charset=utf-8
Content-Length: 36

[
    {"name":"check1","status":"ok"},
    {"name":"check2","status":"failed","error":"some error"}
]

this middleware is pretty basic, but can be used for simple health checks. For more complex cases, like async/cached health checks see alexliesenfeld/health

Logger middleware

Logs request, request handling time and response. Log record fields in order of occurrence:

• Request's HTTP method
• Requested URL (with sanitized query)
• Remote IP
• Response's HTTP status code
• Response body size
• Request handling time
• Userinfo associated with the request (optional)
• Request subject (optional)
• Request ID (if X-Request-ID present)
• Request body (optional)

remote IP can be masked with user defined function

example: 019/03/05 17:26:12.976 [INFO] GET - /api/v1/find?site=remark - 8e228e9cfece - 200 (115) - 4.47784618s

Recoverer middleware

Recoverer is a middleware that recovers from panics, logs the panic (and a backtrace), and returns an HTTP 500 (Internal Server Error) status if possible. It prevents server crashes in case of panic in one of the controllers.

OnlyFrom middleware

OnlyFrom middleware allows access from a limited list of source IPs. Such IPs can be defined as complete ip (like 192.168.1.12), prefix (129.168.) or CIDR (192.168.0.0/16). The middleware will respond with StatusForbidden (403) if the request comes from a different IP. It supports both IPv4 and IPv6 and checks the usual headers like X-Forwarded-For and X-Real-IP and the remote address.

Note: headers should be trusted and set by a proxy, otherwise it is possible to spoof them.

Metrics middleware

Metrics middleware responds to GET /metrics with list of expvar. Optionally allows a restricted list of source ips.

BlackWords middleware

BlackWords middleware doesn't allow user-defined words in the request body.

SizeLimit middleware

SizeLimit middleware checks if body size is above the limit and returns StatusRequestEntityTooLarge (413)

Trace middleware

The Trace middleware is designed to add request tracing functionality. It looks for the X-Request-ID header in the incoming HTTP request. If not found, a random ID is generated. This trace ID is then set in the response headers and added to the request's context.

Deprecation middleware

Adds the HTTP Deprecation response header, see draft-dalal-deprecation-header-00

BasicAuth middleware

BasicAuth middleware requires basic auth and matches user & passwd with client-provided checker. In case if no basic auth headers returns StatusUnauthorized, in case if checker failed - StatusForbidden

Rewrite middleware

The Rewrite middleware is designed to rewrite the URL path based on a given rule, similar to how URL rewriting is done in nginx. It supports regular expressions for pattern matching and prevents multiple rewrites.

For example, Rewrite("^/sites/(.*)/settings/$", "/sites/settings/$1") will change request's URL from /sites/id1/settings/ to /sites/settings/id1

NoCache middleware

Sets a number of HTTP headers to prevent a router (handler's) response from being cached by an upstream proxy and/or client.

Headers middleware

Sets headers (passed as key:value) to requests. I.e. rest.Headers("Server:MyServer", "X-Blah:Foo")

Gzip middleware

Compresses response with gzip.

RealIP middleware

RealIP is a middleware that sets a http.Request's RemoteAddr to the results of parsing either the X-Forwarded-For or X-Real-IP headers.

Maybe middleware

Maybe middleware allows changing the flow of the middleware stack execution depending on the return value of maybeFn(request). This is useful, for example, to skip a middleware handler if a request does not satisfy the maybeFn logic.

Reject middleware

Reject is a middleware that rejects requests with a given status code and message based on a user-defined function. This is useful, for example, to reject requests to a particular resource based on a request header, or to implement a conditional request handler based on service parameters.

example with chi router:

    router := chi.NewRouter()
	
	rejectFn := func(r *http.Request) (bool) {
        return r.Header.Get("X-Request-Id") == "" // reject if no X-Request-Id header
    }
	
	router.Use(rest.Reject(http.StatusBadRequest, "X-Request-Id header is required", rejectFn))

Benchmarks middleware

Benchmarks middleware allows measuring the time of request handling, number of requests per second and report aggregated metrics. This middleware keeps track of the request in the memory and keep up to 900 points (15 minutes, data-point per second).

To retrieve the data user should call Stats(d duration) method. The duration is the time window for which the benchmark data should be returned. It can be any duration from 1s to 15m. Note: all the time data is in microseconds.

example with chi router:

    router := chi.NewRouter()
	bench = rest.NewBenchmarks()
	router.Use(bench.Middleware)
	...
	router.Get("/bench", func(w http.ResponseWriter, r *http.Request) {
        resp := struct {
            OneMin     rest.BenchmarkStats `json:"1min"`
            FiveMin    rest.BenchmarkStats `json:"5min"`
            FifteenMin rest.BenchmarkStats `json:"15min"`
        }{
            bench.Stats(time.Minute),
            bench.Stats(time.Minute * 5),
            bench.Stats(time.Minute * 15),
        }
        render.JSON(w, r, resp) 		
    })

Helpers

• rest.Wrap - converts a list of middlewares to nested handlers calls (in reverse order)
• rest.JSON - map alias, just for convenience type JSON map[string]interface{}
• rest.RenderJSON - renders json response from interface{}
• rest.RenderJSONFromBytes - renders json response from []byte
• rest.RenderJSONWithHTML - renders json response with html tags and forced charset=utf-8
• rest.SendErrorJSON - makes {error: blah, details: blah} json body and responds with given error code. Also, adds context to the logged message
• rest.NewErrorLogger - creates a struct providing shorter form of logger call
• rest.FileServer - creates a file server for static assets with directory listing disabled
• realip.Get - returns client's IP address
• rest.ParseFromTo - parses "from" and "to" request's query params with various formats

Profiler

Profiler is a convenient sub-router used for mounting net/http/pprof, i.e.

 func MyService() http.Handler {
   r := chi.NewRouter()
   // ..middlewares
   r.Mount("/debug", middleware.Profiler())
   // ..routes
   return r
 }

It exposes a bunch of /pprof/* endpoints as well as /vars. Builtin support for onlyIps allows restricting access, which is important if it runs on a publicly exposed port. However, counting on IP check only is not that reliable way to limit request and for production use it would be better to add some sort of auth (for example provided BasicAuth middleware) or run with a separate http server, exposed to internal ip/port only.

Documentation

Overview

Package rest provides common middlewares and helpers for rest services

Index

• func AppInfo(app, author, version string) func(http.Handler) http.Handler
• func BasicAuth(checker func(user, passwd string) bool) func(http.Handler) http.Handler
• func BasicAuthWithPrompt(user, passwd string) func(http.Handler) http.Handler
• func BasicAuthWithUserPasswd(user, passwd string) func(http.Handler) http.Handler
• func BlackWords(words ...string) func(http.Handler) http.Handler
• func BlackWordsFn(fn func() []string) func(http.Handler) http.Handler
• func CacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler
• func CacheControlDynamic(expiration time.Duration, versionFn func(r *http.Request) string) func(http.Handler) http.Handler
• func DecodeJSON[T any](r *http.Request, res *T) error
• func Deprecation(version string, date time.Time) func(http.Handler) http.Handler
• func EncodeJSON[T any](w http.ResponseWriter, status int, v T) error
• func FileServer(public, local string, notFound io.Reader) (http.Handler, error)
• func FileServerSPA(public, local string, notFound io.Reader) (http.Handler, error)
• func FsOptListing(fs *FS) error
• func FsOptSPA(fs *FS) error
• func GetTraceID(r *http.Request) string
• func Gzip(contentTypes ...string) func(http.Handler) http.Handler
• func Headers(headers ...string) func(http.Handler) http.Handler
• func Health(path string, checkers ...func(ctx context.Context) (name string, err error)) func(http.Handler) http.Handler
• func IsAuthorized(ctx context.Context) bool
• func Maybe(mw func(http.Handler) http.Handler, maybeFn func(r *http.Request) bool) func(http.Handler) http.Handler
• func Metrics(onlyIps ...string) func(http.Handler) http.Handler
• func NoCache(h http.Handler) http.Handler
• func OnlyFrom(onlyIps ...string) func(http.Handler) http.Handler
• func ParseFromTo(r *http.Request) (from, to time.Time, err error)
• func Ping(next http.Handler) http.Handler
• func Profiler(onlyIps ...string) http.Handler
• func RealIP(h http.Handler) http.Handler
• func Recoverer(l logger.Backend) func(http.Handler) http.Handler
• func Reject(errCode int, errMsg string, rejectFn func(r *http.Request) bool) func(h http.Handler) http.Handler
• func RenderJSON(w http.ResponseWriter, data interface{})
• func RenderJSONFromBytes(w http.ResponseWriter, r *http.Request, data []byte) error
• func RenderJSONWithHTML(w http.ResponseWriter, r *http.Request, v interface{}) error
• func Rewrite(from, to string) func(http.Handler) http.Handler
• func SendErrorJSON(w http.ResponseWriter, r *http.Request, l logger.Backend, code int, err error, ...)
• func SizeLimit(size int64) func(http.Handler) http.Handler
• func Throttle(limit int64) func(http.Handler) http.Handler
• func Trace(next http.Handler) http.Handler
• func Wrap(handler http.Handler, mws ...func(http.Handler) http.Handler) http.Handler
• type BenchmarkStats
• type Benchmarks
  • func NewBenchmarks() *Benchmarks
  • func (b *Benchmarks) Handler(next http.Handler) http.Handler
  • func (b *Benchmarks) Stats(interval time.Duration) BenchmarkStats
  • func (b *Benchmarks) WithTimeRange(max time.Duration) *Benchmarks
• type ErrorLogger
  • func NewErrorLogger(l logger.Backend) *ErrorLogger
  • func (e *ErrorLogger) Log(w http.ResponseWriter, r *http.Request, httpCode int, err error, msg ...string)
• type FS
  • func NewFileServer(public, local string, options ...FsOpt) (*FS, error)
  • func (fs *FS) ServeHTTP(w http.ResponseWriter, r *http.Request)
• type FsOpt
  • func FsOptCustom404(fr io.Reader) FsOpt
• type JSON

Functions

func AppInfo

func AppInfo(app, author, version string) func(http.Handler) http.Handler

AppInfo adds custom app-info to the response header

func BasicAuth

func BasicAuth(checker func(user, passwd string) bool) func(http.Handler) http.Handler

BasicAuth middleware requires basic auth and matches user & passwd with client-provided checker

func BasicAuthWithPrompt

func BasicAuthWithPrompt(user, passwd string) func(http.Handler) http.Handler

BasicAuthWithPrompt middleware requires basic auth and matches user & passwd with client-provided values If the user is not authorized, it will prompt for basic auth

func BasicAuthWithUserPasswd

func BasicAuthWithUserPasswd(user, passwd string) func(http.Handler) http.Handler

BasicAuthWithUserPasswd middleware requires basic auth and matches user & passwd with client-provided values

func BlackWords

func BlackWords(words ...string) func(http.Handler) http.Handler

BlackWords middleware doesn't allow some words in the request body

func BlackWordsFn

func BlackWordsFn(fn func() []string) func(http.Handler) http.Handler

BlackWordsFn middleware uses func to get the list and doesn't allow some words in the request body

func CacheControl

func CacheControl(expiration time.Duration, version string) func(http.Handler) http.Handler

CacheControl is a middleware setting cache expiration. Using url+version for etag

func CacheControlDynamic

func CacheControlDynamic(expiration time.Duration, versionFn func(r *http.Request) string) func(http.Handler) http.Handler

CacheControlDynamic is a middleware setting cache expiration. Using url+ func(r) for etag

func DecodeJSON

func DecodeJSON[T any](r *http.Request, res *T) error

DecodeJSON decodes json request from http.Request to given type

func Deprecation

func Deprecation(version string, date time.Time) func(http.Handler) http.Handler

Deprecation adds a header 'Deprecation: version="version", date="date" header' see https://tools.ietf.org/id/draft-dalal-deprecation-header-00.html

func EncodeJSON

func EncodeJSON[T any](w http.ResponseWriter, status int, v T) error

EncodeJSON encodes given type to http.ResponseWriter and sets status code and content type header

func FileServer

func FileServer(public, local string, notFound io.Reader) (http.Handler, error)

FileServer is a shortcut for making FS with listing disabled and the custom noFound reader (can be nil). Deprecated: the method is for back-compatibility only and user should use the universal NewFileServer instead

func FileServerSPA

func FileServerSPA(public, local string, notFound io.Reader) (http.Handler, error)

FileServerSPA is a shortcut for making FS with SPA-friendly handling of 404, listing disabled and the custom noFound reader (can be nil). Deprecated: the method is for back-compatibility only and user should use the universal NewFileServer instead

func FsOptListing

func FsOptListing(fs *FS) error

FsOptListing turns on directory listing

func FsOptSPA

func FsOptSPA(fs *FS) error

FsOptSPA turns on SPA mode returning "/index.html" on not-found

func GetTraceID

func GetTraceID(r *http.Request) string

GetTraceID returns request id from the context

func Gzip

func Gzip(contentTypes ...string) func(http.Handler) http.Handler

Gzip is a middleware compressing response

func Headers

func Headers(headers ...string) func(http.Handler) http.Handler

Headers middleware adds headers to request

func Health

func Health(path string, checkers ...func(ctx context.Context) (name string, err error)) func(http.Handler) http.Handler

Health middleware response with health info and status (200 if healthy). Stops chain if health request detected passed checkers implements custom health checks and returns error if health check failed. The check has to return name regardless to the error state. For production usage this middleware should be used with throttler and, optionally, with BasicAuth middlewares

func IsAuthorized

func IsAuthorized(ctx context.Context) bool

IsAuthorized returns true is user authorized. it can be used in handlers to check if BasicAuth middleware was applied

func Maybe

func Maybe(mw func(http.Handler) http.Handler, maybeFn func(r *http.Request) bool) func(http.Handler) http.Handler

Maybe middleware will allow you to change the flow of the middleware stack execution depending on return value of maybeFn(request). This is useful for example if you'd like to skip a middleware handler if a request does not satisfy the maybeFn logic. borrowed from https://github.com/go-chi/chi/blob/master/middleware/maybe.go

func Metrics

func Metrics(onlyIps ...string) func(http.Handler) http.Handler

Metrics responds to GET /metrics with list of expvar

func NoCache

func NoCache(h http.Handler) http.Handler

NoCache is a simple piece of middleware that sets a number of HTTP headers to prevent a router (or subrouter) from being cached by an upstream proxy and/or client.

As per http://wiki.nginx.org/HttpProxyModule - NoCache sets:

Expires: Thu, 01 Jan 1970 00:00:00 UTC
Cache-Control: no-cache, private, max-age=0
X-Accel-Expires: 0
Pragma: no-cache (for HTTP/1.0 proxies/clients)

func OnlyFrom

func OnlyFrom(onlyIps ...string) func(http.Handler) http.Handler

OnlyFrom middleware allows access for limited list of source IPs. Such IPs can be defined as complete ip (like 192.168.1.12), prefix (129.168.) or CIDR (192.168.0.0/16)

func ParseFromTo

func ParseFromTo(r *http.Request) (from, to time.Time, err error)

ParseFromTo parses from and to query params of the request

func Ping

func Ping(next http.Handler) http.Handler

Ping middleware response with pong to /ping. Stops chain if ping request detected

func Profiler

func Profiler(onlyIps ...string) http.Handler

Profiler is a convenient subrouter used for mounting net/http/pprof. ie.

func MyService() http.Handler {
  r := chi.NewRouter()
  // ..middlewares
  r.Mount("/debug", middleware.Profiler())
  // ..routes
  return r
}

func RealIP

func RealIP(h http.Handler) http.Handler

RealIP is a middleware that sets a http.Request's RemoteAddr to the results of parsing either the X-Forwarded-For or X-Real-IP headers.

This middleware should only be used if user can trust the headers sent with request. If reverse proxies are configured to pass along arbitrary header values from the client, or if this middleware used without a reverse proxy, malicious clients could set anything as X-Forwarded-For header and attack the server in various ways.

func Recoverer

func Recoverer(l logger.Backend) func(http.Handler) http.Handler

Recoverer is a middleware that recovers from panics, logs the panic and returns a HTTP 500 status if possible.

func Reject

func Reject(errCode int, errMsg string, rejectFn func(r *http.Request) bool) func(h http.Handler) http.Handler

Reject is a middleware that conditionally rejects requests with a given status code and message. user-defined condition function rejectFn is used to determine if the request should be rejected.

func RenderJSON

func RenderJSON(w http.ResponseWriter, data interface{})

RenderJSON sends data as json

func RenderJSONFromBytes

func RenderJSONFromBytes(w http.ResponseWriter, r *http.Request, data []byte) error

RenderJSONFromBytes sends binary data as json

func RenderJSONWithHTML

func RenderJSONWithHTML(w http.ResponseWriter, r *http.Request, v interface{}) error

RenderJSONWithHTML allows html tags and forces charset=utf-8

func Rewrite

func Rewrite(from, to string) func(http.Handler) http.Handler

Rewrite middleware with from->to rule. Supports regex (like nginx) and prevents multiple rewrites example: Rewrite(`^/sites/(.*)/settings/$`, `/sites/settings/$1`

func SendErrorJSON

func SendErrorJSON(w http.ResponseWriter, r *http.Request, l logger.Backend, code int, err error, msg string)

SendErrorJSON sends {error: msg} with error code and logging error and caller

func SizeLimit

func SizeLimit(size int64) func(http.Handler) http.Handler

SizeLimit middleware checks if body size is above the limit and returns StatusRequestEntityTooLarge (413)

func Throttle

func Throttle(limit int64) func(http.Handler) http.Handler

Throttle middleware checks how many request in-fly and rejects with 503 if exceeded

func Trace

func Trace(next http.Handler) http.Handler

Trace looks for header X-Request-ID and makes it as random id if not found, then populates it to the result's header and to request context

func Wrap

func Wrap(handler http.Handler, mws ...func(http.Handler) http.Handler) http.Handler

Wrap converts a list of middlewares to nested calls (in reverse order)

Types

type BenchmarkStats

type BenchmarkStats struct {
	Requests        int     `json:"requests"`
	RequestsSec     float64 `json:"requests_sec"`
	AverageRespTime int64   `json:"average_resp_time"`
	MinRespTime     int64   `json:"min_resp_time"`
	MaxRespTime     int64   `json:"max_resp_time"`
}

BenchmarkStats holds the stats for a given interval

type Benchmarks

type Benchmarks struct {
	// contains filtered or unexported fields
}

Benchmarks is a basic benchmarking middleware collecting and reporting performance metrics It keeps track of the requests speeds and counts in 1s benchData buckets ,limiting the number of buckets to maxTimeRange. User can request the benchmark for any time duration. This is intended to be used for retrieving the benchmark data for the last minute, 5 minutes and up to maxTimeRange.

func NewBenchmarks

func NewBenchmarks() *Benchmarks

NewBenchmarks creates a new benchmark middleware

func (*Benchmarks) Handler

func (b *Benchmarks) Handler(next http.Handler) http.Handler

Handler calculates 1/5/10m request per second and allows to access those values

func (*Benchmarks) Stats

func (b *Benchmarks) Stats(interval time.Duration) BenchmarkStats

Stats returns the current benchmark stats for the given duration

func (*Benchmarks) WithTimeRange

func (b *Benchmarks) WithTimeRange(max time.Duration) *Benchmarks

WithTimeRange sets the maximum time range for the benchmark to keep data for. Default is 15 minutes. The increase of this range will change memory utilization as each second of the range kept as benchData aggregate. The default means 15*60 = 900 seconds of data aggregate. Larger range allows for longer time periods to be benchmarked.

type ErrorLogger

type ErrorLogger struct {
	// contains filtered or unexported fields
}

ErrorLogger wraps logger.Backend

func NewErrorLogger

func NewErrorLogger(l logger.Backend) *ErrorLogger

NewErrorLogger creates ErrorLogger for given Backend

func (*ErrorLogger) Log

func (e *ErrorLogger) Log(w http.ResponseWriter, r *http.Request, httpCode int, err error, msg ...string)

Log sends json error message {error: msg} with error code and logging error and caller

type FS

type FS struct {
	// contains filtered or unexported fields
}

FS provides http.FileServer handler to serve static files from a http.FileSystem, prevents directory listing by default and supports spa-friendly mode (off by default) returning /index.html on 404. - public defines base path of the url, i.e. for http://example.com/static/* it should be /static - local for the local path to the root of the served directory - notFound is the reader for the custom 404 html, can be nil for default

func NewFileServer

func NewFileServer(public, local string, options ...FsOpt) (*FS, error)

NewFileServer creates file server with optional spa mode and optional direcroty listing (disabled by default)

func (*FS) ServeHTTP

func (fs *FS) ServeHTTP(w http.ResponseWriter, r *http.Request)

ServeHTTP makes FileServer compatible with http.Handler interface

type FsOpt

type FsOpt func(fs *FS) error

FsOpt defines functional option type

func FsOptCustom404

func FsOptCustom404(fr io.Reader) FsOpt

FsOptCustom404 sets custom 404 reader

type JSON

type JSON map[string]any

JSON is a map alias, just for convenience