httpcompression

README

Golang server middleware for HTTP compression

This is a small Go package which wraps HTTP handlers to transparently compress response bodies using zstd, brotli or gzip - for clients which support them. Although it's usually simpler to leave that to a reverse proxy (like nginx or Varnish), this package is useful when that is undesirable. In addition, this package allows users to extend it by plugging in third-party or custom compression encoders.

Note: This package was recently forked from NYTimes/gziphandler. Maintaining drop-in compatibility is not a goal of this fork, as the scope of this fork is significantly wider than the original package.

⚠ As we have not reached 1.0 yet, API is still subject to changes.

Features

• gzip and brotli compression by default, zstd and alternate (faster) gzip implementations are optional
• Apply compression only if response body size is greater than a threshold
• Apply compression only to a allowlist/denylist of MIME content types
• Define encoding priority (e.g. give brotli a higher priority than gzip)
• Control whether the client or the server defines the encoder priority
• Plug in third-party/custom compression schemes or implementations
• Custom dictionary compression for zstd

Install

go get -u github.com/CAFxX/httpcompression

Usage

Call httpcompression.Handler to get an adapter that can be used to wrap any handler (an object which implements the http.Handler interface), to transparently provide response body compression. Note that, despite the name, httpcompression automatically compresses using Brotli or Gzip, depending on the capabilities of the client (Accept-Encoding) and the configuration of this handler (by default, both Gzip and Brotli are enabled and Brotli is used by default if the client supports both).

As a simple example:

package main

import (
	"io"
	"net/http"
	"github.com/CAFxX/httpcompression"
)

func main() {
	handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Header().Set("Content-Type", "text/plain")
		io.WriteString(w, "Hello, World")
	})
	compress := httpcompression.DefaultAdapter() // Use the default configuration
	http.Handle("/", compress(handler))
	http.ListenAndServe("0.0.0.0:8080", nil)
}

Benchmark

See the benchmark results to get an idea of the relative performance and compression efficiency of gzip, brotli and zstd in the current implementation.

TODO

• Add dictionary support to gzip and brotli (zstd already supports it)
• Allow to choose dictionary based on content-type

License

Apache 2.0.

Documentation

Overview

Example

package main

import (
	"log"
	"net/http"

	"github.com/CAFxX/httpcompression"
)

func main() {
	// Create a compression adapter with default configuration
	compress, err := httpcompression.DefaultAdapter()
	if err != nil {
		log.Fatal(err)
	}
	// Define your handler, and apply the compression adapter.
	http.Handle("/", compress(http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) {
		w.Write([]byte("Hello world!"))
	})))
	// ...
}

Index

• Constants
• func Adapter(opts ...Option) (func(http.Handler) http.Handler, error)
• func DefaultAdapter(opts ...Option) (func(http.Handler) http.Handler, error)
• func NewDefaultGzipCompressor(level int) (*defaultGzipCompressor, error)
• type CompressorProvider
• type Flusher
• type Option
  • func BrotliCompressionLevel(level int) Option
  • func BrotliCompressor(b CompressorProvider) Option
  • func Compressor(contentEncoding string, priority int, compressor CompressorProvider) Option
  • func ContentTypes(types []string, blacklist bool) Option
  • func GzipCompressionLevel(level int) Option
  • func GzipCompressor(g CompressorProvider) Option
  • func MinSize(size int) Option
  • func Prefer(prefer PreferType) Option
• type PreferType

Examples

• Package

Constants

const (
	// DefaultMinSize is the default minimum response body size for which we enable compression.
	//
	// 200 is a somewhat arbitrary number; in experiments compressing short text/markup-like sequences
	// with different compressors we saw that sequences shorter that ~180 the output generated by the
	// compressor would sometime be larger than the input.
	// This default may change between versions.
	// In general there can be no one-size-fits-all value: you will want to measure if a different
	// minimum size improves end-to-end performance for your workloads.
	DefaultMinSize = 200
)

Functions

func Adapter

func Adapter(opts ...Option) (func(http.Handler) http.Handler, error)

Adapter returns a HTTP handler wrapping function (a.k.a. middleware) which can be used to wrap an HTTP handler to transparently compress the response body if the client supports it (via the Accept-Encoding header). It is possible to pass one or more options to modify the middleware configuration. If no options are provided, no compressors are enabled and therefore the adapter is a no-op. An error will be returned if invalid options are given.

func DefaultAdapter

func DefaultAdapter(opts ...Option) (func(http.Handler) http.Handler, error)

DefaultAdapter is like Adapter, but it includes sane defaults for general usage. Currently the defaults enable gzip and brotli compression, and set a minimum body size of 200 bytes. The provided opts override the defaults. The defaults are not guaranteed to remain constant over time: if you want to avoid this use Adapter directly.

func NewDefaultGzipCompressor

func NewDefaultGzipCompressor(level int) (*defaultGzipCompressor, error)

Types

type CompressorProvider

type CompressorProvider interface {
	// Get returns a writer that writes compressed output to the supplied parent io.Writer.
	// When Close() is called on the returned io.WriteCloser, it is guaranteed that it will
	// not be used anymore so implementations can safely recycle the compressor (e.g. put the
	// WriteCloser in a pool to be reused by a later call to Get).
	// The returned io.WriteCloser can optionally implement the Flusher interface if it is
	// able to flush data buffered internally.
	Get(parent io.Writer) (compressor io.WriteCloser)
}

type Flusher

type Flusher interface {
	// Flush flushes the data buffered internally by the Writer.
	// Flush does not need to internally flush the parent Writer.
	Flush() error
}

type Option

type Option func(c *config) error

Option can be passed to Handler to control its configuration.

func BrotliCompressionLevel

func BrotliCompressionLevel(level int) Option

BrotliCompressionLevel is an option that controls the Brotli compression level to be used when compressing payloads. The default is 3 (the same default used in the reference brotli C implementation).

func BrotliCompressor

func BrotliCompressor(b CompressorProvider) Option

BrotliCompressor is an option to specify a custom compressor factory for Brotli.

func Compressor

func Compressor(contentEncoding string, priority int, compressor CompressorProvider) Option

Compressor returns an Option that sets the CompressorProvider for a specific Content-Encoding. If multiple CompressorProviders are set for the same Content-Encoding, the last one is used. If compressor is nil, it disables the specified Content-Encoding. Priority is used to specify the priority of the Content-Encoding. A higher number means higher priority. See PreferType to understand how priority is used to select the Content-Encoding for a specific request.

func ContentTypes

func ContentTypes(types []string, blacklist bool) Option

ContentTypes specifies a list of content types to compare the Content-Type header to before compressing. If none match, and blacklist is false, the response will be returned as-is.

Content types are compared in a case-insensitive, whitespace-ignored manner.

A MIME type without any other directive will match a content type that has the same MIME type, regardless of that content type's other directives. I.e., "text/html" will match both "text/html" and "text/html; charset=utf-8".

A MIME type with any other directive will only match a content type that has the same MIME type and other directives. I.e., "text/html; charset=utf-8" will only match "text/html; charset=utf-8".

If blacklist is true then only content types that do not match the provided list of types are compressed. If blacklist is false, only content types that match the provided list are compressed.

By default, responses are compressed regardless of Content-Type.

func GzipCompressionLevel

func GzipCompressionLevel(level int) Option

GzipCompressionLevel is an option that controls the Gzip compression level to be used when compressing payloads. The default is gzip.DefaultCompression.

func GzipCompressor

func GzipCompressor(g CompressorProvider) Option

GzipCompressor is an option to specify a custom compressor factory for Gzip.

func MinSize

func MinSize(size int) Option

MinSize is an option that controls the minimum size of payloads that should be compressed. The default is DefaultMinSize.

func Prefer

func Prefer(prefer PreferType) Option

Prefer controls the behavior of the middleware in case both Gzip and Brotli can be used to compress a response (i.e. in case the client supports both encodings, and the MIME type of the response is allowed for both encodings). See the comments on the PreferType constants for the supported values.

type PreferType

type PreferType byte

PreferType allows to control the choice of compression algorithm when multiple algorithms are allowed by both client and server.

const (
	// PreferServer prefers compressors in the order specified on the server.
	// If two or more compressors have the same priority on the server, the client preference is taken into consideration.
	// If both server and client do no specify a preference between two or more compressors, the order is determined by the name of the encoding.
	// PreferServer is the default.
	PreferServer PreferType = iota

	// PreferClient prefers compressors in the order specified by the client.
	// If two or more compressors have the same priority according to the client, the server priority is taken into consideration.
	// If both server and client do no specify a preference between two or more compressors, the order is determined by the name of the encoding.
	PreferClient
)