acceptable

README

Acceptable

This is a library that handles Accept headers, which form the basis of content negotiation in HTTP server applications written in Go. It provides an implementation of the proactive server-driven content negotiation algorithm specified in RFC-7231 section 5.3.

There is also support for conditional requests (RFC-7232) using entity tags and last-modified timestamps.

Bring your favourite router and framework - this library can be used with Gin, Echo, etc.

Please see the documentation for more info.

Status

This API is well-tested and known to work but not yet fully released because it may yet require breaking API changes.

Documentation

Overview

Package acceptable is a library that handles headers for content negotiation and conditional requests in web applications written in Go. Content negotiation is specified by RFC (http://tools.ietf.org/html/rfc7231) and, less formally, by Ajax (https://en.wikipedia.org/wiki/XMLHttpRequest).

Subpackages

* contenttype, headername - bundles of useful constants

* data - for holding response data & metadata prior to rendering the response, also allowing lazy evaluation

* header - for parsing and representing certain HTTP headers

* offer - for enumerating offers to be matched against requests

* templates - for rendering Go templates

Easy content negotiation

Server-based content negotiation is essentially simple: the user agent sends a request including some preferences (accept headers), then the server selects one of several possible ways of sending the response. Finding the best match depends on you listing your available response representations. This is all rolled up into a simple-to-use function `acceptable.RenderBestMatch`. What this does is described in detail in [RFC-7231](https://tools.ietf.org/html/rfc7231#section-5.3), but it's easy to use in practice.

For example

en := ... obtain some content in English
fr := ... obtain some content in French

// a JSON offer with pretty output
offer1 := offer.JSON("  ").With(en, "en").With(fr, "fr")

// an XML offer
offer2 := offer.XML().With(en, "en").With(fr, "fr")

// a catch-all offer is optional
catchAll := offer.Of(acceptable.TXTProcessor(), contenttype.Any).With(en, "en").With(fr, "fr")

err := acceptable.RenderBestMatch(response, request, 200, "", offer1, offer2, catchAll)

The RenderBestMatch function searches for the offer that best matches the request headers. If none match, the response will be 406-Not Acceptable. If you need to have a catch-all case, include offer.Of(p, contenttype.TextAny) or offer.Of(p, contenttype.Any) last in the list. Note that contenttype.TextAny is "text/*" and will typically return "text/plain"; contenttype.Any is "*/*" and will likewise return "application/octet-stream".

Each offer will (usually) have a suitable offer.Processor, which is a rendering function. Several are provided (for JSON, XML etc), but you can also provide your own.

Also, the templates sub-package provides Go template support.

Offers are restricted both by content-type matching and by language matching. The `With` method provides data and specifies its content language. Use it as many times as you need to.

The language(s) is matched against the Accept-Language header using the basic prefix algorithm. This means for example that if you specify "en" it will match "en", "en-GB" and everything else beginning with "en-", but if you specify "en-GB", it only matches "en-GB" and "en-GB-*", but won't match "en-US" or even "en". (This implements the basic filtering language matching algorithm defined in https://tools.ietf.org/html/rfc4647.)

If your data doesn't need to specify a language, the With method should simply use the "*" wildcard instead. For example, myOffer.With(data, "*") attaches data to myOffer and doesn't restrict the offer to any particular language.

The language wildcard could also be used as a catch-all case if it comes after one or more With with a specified language. However, the standard (RFC-7231) advises that a response should be returned even when language matching has failed; RenderBestMatch will do this by picking the first language listed as a fallback, so the catch-all case is only necessary if its data is different to that of the first case.

Providing response data

The response data (en and fr above) can be structs, slices, maps, or other values that the rendering processors accept. They will be wrapped as data.Data values, which you can provid explicitly. These allow for lazy evaluation of the content and also support conditional requests. This comes into its own when there are several offers each with their own data model - if these were all to be read from the database before selection of the best match, all but one would be wasted. Lazy evaluation of the selected data easily overcomes this problem.

en := data.Lazy(func(template, language string) (value interface{}, err error) {
    return ...
})

Besides the data and error returned values, some metadata can optionally be returned. This is the basis for easy support for conditional requests (see [RFC-7232](https://tools.ietf.org/html/rfc7232)).

If the metadata is nil, it is simply ignored. However, if it contains a hash of the data (e.g. via MD5) known as the entity tag or etag, then the response will have an ETag header. User agents that recognise this will later repeat the request along with an If-None-Match header. If present, If-None-Match is recognised before rendering starts and a successful match will avoid the need for any rendering. Due to the lazy content fetching, it can reduce unnecessary database traffic etc.

The metadata can also carry the last-modified timestamp of the data, if this is known. When present, this becomes the Last-Modified header and is checked on subsequent requests using the If-Modified-Since.

The template and language parameters are used for templated/web content data; otherwise they are ignored.

Sequences of data can also be produced. This is done with data.Sequence() and this takes the same supplier function as used by data.Lazy(). The difference is that, in a sequence, the supplier function will be called repeatedly until its result value is nil. All the values will be streamed in the response (how this is done depends on the rendering processor.

Character set transcoding

Most responses will be UTF-8, sometimes UTF-16. All other character sets (e.g. Windows-1252) are now strongly deprecated.

However, legacy support for other character sets is provided. Transcoding is implemented by Match.ApplyHeaders so that the Accept-Charset content negotiation can be implemented. This depends on finding an encoder in golang.org/x/text/encoding/htmlindex (this has an extensive list, however no other encoders are supported).

Whenever possible, responses will be UTF-8. Not only is this strongly recommended, it also avoids any transcoding processing overhead. It means for example that "Accept-Charset: iso-8859-1, utf-8" will ignore the iso-8859-1 preference because it can use UTF-8. Conversely, "Accept-Charset: iso-8859-1" will always have to transcode into ISO-8859-1 because there is no UTF-8 option.

Example

package main

import (
	"fmt"
	"log"
	"net/http"
	"net/http/httptest"
	"sort"

	"github.com/rickb777/acceptable"
	"github.com/rickb777/acceptable/contenttype"
	"github.com/rickb777/acceptable/data"
	"github.com/rickb777/acceptable/headername"
	"github.com/rickb777/acceptable/offer"
	"github.com/rickb777/acceptable/templates"
)

func main() {
	// In this example, the same content is available in three languages. Three different
	// approaches can be used.

	// 1. simple values can be used
	en := "Hello!" // get English content

	// 2. values can be wrapped in a data.Data
	fr := data.Of("Bonjour!").ETag("hash1") // get French content and some metadata

	// 3. this uses a lazy evaluation function, wrapped in a data.Data
	es := data.Lazy(func(template string, language string) (interface{}, error) {
		return "Hola!", nil // get Spanish content - eg from database
	}).ETagUsing(func(template, language string) (string, error) {
		// allows us to obtain the etag lazily, should we need to
		return "hash2", nil
	})

	// We're implementing an HTTP handler, so we are given a request and a response.

	req1, _ := http.NewRequest("GET", "/request1", nil) // some incoming request
	req1.Header.Set(headername.Accept, "text/plain, text/html")
	req1.Header.Set(headername.AcceptLanguage, "es, fr;q=0.8, en;q=0.6")

	req2, _ := http.NewRequest("GET", "/request2", nil) // some incoming request
	req2.Header.Set(headername.Accept, "application/json")
	req2.Header.Set(headername.AcceptLanguage, "fr")

	req3, _ := http.NewRequest("GET", "/request3", nil) // some incoming request
	req3.Header.Set(headername.Accept, "text/html")
	req3.Header.Set(headername.AcceptLanguage, "fr")
	req3.Header.Set(headername.IfNoneMatch, `"hash1"`)

	requests := []*http.Request{req1, req2, req3}

	for _, req := range requests {
		res := httptest.NewRecorder() // replace with the server's http.ResponseWriter

		// Now do the content negotiation. This example has six supported content types, all of them
		// able to serve any of the three example languages.
		//
		// The first offer is for JSON - this is often the most widely used because it also supports
		// Ajax requests.

		err := acceptable.RenderBestMatch(res, req, 200, "home.html", offer.JSON("  ").
			With(en, "en").With(fr, "fr").With(es, "es"), offer.XML("xml", "  ").
			With(en, "en").With(fr, "fr").With(es, "es"), offer.CSV().
			With(en, "en").With(fr, "fr").With(es, "es"), offer.Of(offer.TXTProcessor(), contenttype.TextPlain).
			With(en, "en").With(fr, "fr").With(es, "es"), templates.TextHtmlOffer("example/templates/en", ".html", nil).
			With(en, "en").With(fr, "fr").With(es, "es"), templates.ApplicationXhtmlOffer("example/templates/en", ".html", nil).
			With(en, "en").With(fr, "fr").With(es, "es"))

		if err != nil {
			log.Fatal(err) // replace with suitable error handling
		}

		// ----- ignore the following, which is needed only for the example test to run -----
		fmt.Printf("%s %s %d\n", req.Method, req.URL, res.Code)
		fmt.Printf("%d headers\n", len(res.Header()))
		var hdrs []string
		for h, _ := range res.Header() {
			hdrs = append(hdrs, h)
		}
		sort.Strings(hdrs)
		for _, h := range hdrs {
			fmt.Printf("%s: %s\n", h, res.Header().Get(h))
		}
		fmt.Println()
		fmt.Println(res.Body.String())
	}

}

Output:

GET /request1 200
4 headers
Content-Language: es
Content-Type: text/plain;charset=utf-8
Etag: "hash2"
Vary: Accept, Accept-Language

Hola!

GET /request2 200
4 headers
Content-Language: fr
Content-Type: application/json;charset=utf-8
Etag: "hash1"
Vary: Accept, Accept-Language

"Bonjour!"

GET /request3 304
4 headers
Content-Language: fr
Content-Type: text/html;charset=utf-8
Etag: "hash1"
Vary: Accept, Accept-Language

Index

• Variables
• func BestRequestMatch(req *http.Request, available ...offerpkg.Offer) *offerpkg.Match
• func IsAjax(req *http.Request) bool
• func RenderBestMatch(rw http.ResponseWriter, req *http.Request, statusCode int, template string, ...) error

Examples

• Package

Variables

var Debug = func(string, ...interface{}) {}

Debug can be used for observing decisions made by the negotiation algorithm. By default it is no-op.

var NoMatchAccepted = func(rw http.ResponseWriter, _ *http.Request) {
	rw.Header().Set(headername.ContentType, contenttype.TextPlain+";"+contenttype.CharsetUTF8)
	rw.WriteHeader(http.StatusNotAcceptable)
	defaultNotAcceptableMessage := http.StatusText(http.StatusNotAcceptable) + "\n"
	rw.Write([]byte(defaultNotAcceptableMessage))
}

NoMatchAccepted is a function used by RenderBestMatch when no match has been found. Replace this as needed. Note that offer.Offer can also handle 406-Not-Accepted cases, allowing customised error responses.

Functions

func BestRequestMatch

func BestRequestMatch(req *http.Request, available ...offerpkg.Offer) *offerpkg.Match

BestRequestMatch finds the content type and language that best matches the accepted media ranges and languages contained in request headers. The result contains the best match, based on the rules of RFC-7231. On exit, the result will contain the preferred language and charset, if these are known.

Whenever the result is nil, the response should be 406-Not Acceptable.

For all Ajax requests, the available offers are filtered so that only those capable of providing an Ajax response are considered by the content negotiation algorithm. The other offers are discarded.

The order of offers is important. It determines the order they are compared against the request headers, and this determines what defaults will be used when exact matching is not possible.

If no available offers are provided, the response will normally be nil. Note too that Ajax requests will result in nil being returned if no offer is capable of handling JSON, even if other offers are provided.

However, when no available offers are provided, a match will still be returned if any of the offers has its Handle406 set non-zero. This fallback match allows custom error messages to be returned according to the context. The

func IsAjax

func IsAjax(req *http.Request) bool

IsAjax tests whether a request has the Ajax header sent by browsers for XHR requests.

func RenderBestMatch

func RenderBestMatch(rw http.ResponseWriter, req *http.Request, statusCode int, template string, available ...offerpkg.Offer) error

RenderBestMatch uses BestRequestMatch to find the best matching offer for the request, and then renders the response. The returned error, if any, will have arisen from either the content provider (see data.Content) or the response processor (see offer.Processor).

If there are no available offers, the response is simply 204-No Content and the matching algorithm is skipped.

A matching offer is then sought.

If no match is found, a fallback match is sought. If a fallback offer is matched, its Handle406As status code will be used, and its data is rendered using its processor; no further processing follows. Otherwise NoMatchAccepted is called and processing ends.

If a match is found, the following happens.

If the matched offer has empty data, the response will be 204-No Content; no further processing occurs.

A check is then made for a conditional request (ETag, If-Modified-Since etc). If this is successful, the response is 304-Not Modified and no response rendering occurs.

Finally, if statusCode is non-zero it is applied to the response (200-OK otherwise). Then the matched offer's data is rendered using the offer's processor.