microcache

package module
v0.0.0-...-b20cb0c Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 23, 2019 License: MIT Imports: 14 Imported by: 0

README

microcache is a non-standard HTTP cache implemented as Go middleware.

HTTP Microcaching is a well known strategy for improving the efficiency, availability and response time variability of HTTP web services. These benefits are especially relevant in microservice architectures where a service's synchronous dependencies sometimes become unavailable and it is not always feasible or economical to add a separate caching layer between all services.

To date, very few software packages exist to solve this specific problem. Most microcache deployments make use of existing HTTP caching middleware. This presents a challenge. When an HTTP cache exists for the purpose of microcaching between an origin server and a CDN, the server must choose whether to use standard HTTP caching headers with aggressive short TTLs for the microcache or less aggressive longer TTL headers more suitable to CDNs. The overlap in HTTP header key space prevents these two cache layers from coexisting without some additional customization.

All request specific custom response headers supported by this cache are prefixed with microcache- and scrubbed from the response. Most of the common HTTP caching headers one would expect to see in an http cache are ignored (except Vary). This was intentional and support may change depending on developer feedback. The purpose of this cache is not to act as a substitute for a robust HTTP caching layer but rather to serve as an additional caching layer with separate controls for shorter lived, more aggressive caching measures.

The manner in which this cache operates (writing response bodies to byte buffers) may not be suitable for all applications. Caching should certainly be disabled for any resources serving very large and/or streaming responses. For instance, caching is automatically disabled for all websocket requests.

More info in the docs: https://godoc.org/github.com/kevburnsjr/microcache

Example

package main

import (
	"log"
	"math/rand"
	"net/http"
	"strings"
	"time"

	"github.com/kevburnsjr/microcache"
)

type handler struct {
}

// This example fills up to 1.2GB of memory, so at least 2.0GB of RAM is recommended
func (h handler) ServeHTTP(w http.ResponseWriter, r *http.Request) {

	// Enable cache
	w.Header().Set("microcache-cache", "1")

	randn := rand.Intn(10) + 1

	// Sleep between 10 and 100 ms
	time.Sleep(time.Duration(randn*10) * time.Millisecond)

	// Return a response body of random size between 10 and 100 kilobytes
	// Requests per sec for cache hits is mostly dependent on response size
	// This cache can saturate a gigabit network connection with cache hits
	// containing response bodies as small as 10kb on a dual core 3.3 Ghz i7 VM
	http.Error(w, strings.Repeat("1234567890", randn*1e3), 200)
}

func logStats(stats microcache.Stats) {
	total := stats.Hits + stats.Misses + stats.Stales
	log.Printf("Size: %d, Total: %d, Hits: %d, Misses: %d, Stales: %d, Backend: %d, Errors: %d\n",
		stats.Size,
		total,
		stats.Hits,
		stats.Misses,
		stats.Stales,
		stats.Backend,
		stats.Errors,
	)
}

func main() {
	// - Nocache: true
	// Cache is disabled for all requests by default
	// Cache can be enabled per request hash with response header
	//
	//     microcache-cache: 1
	//
	// - Timeout: 3 * time.Second
	// Requests will be timed out and treated as 503 if they do not return within 35s
	//
	// - TTL: 30 * time.Second
	// Responses which enable cache explicitly will be cached for 30s by default
	// Response cache time can be configured per endpoint with response header
	//
	//     microcache-ttl: 30
	//
	// - StaleIfError: 3600 * time.Second
	// If the request encounters an error (or times out), a stale response will be returned
	// provided that the stale cached response expired less than an hour ago.
	// Can be altered per request with response header
	// More Info: https://tools.ietf.org/html/rfc5861
	//
	//     microcache-stale-if-error: 86400
	//
	// - StaleRecache: true
	// Upon serving a stale response following an error, that stale response will be
	// re-cached for the default ttl (3s)
	// Can be disabled per request with response header
	//
	//     microcache-no-stale-recache: 1
	//
	// - StaleWhileRevalidate: 30 * time.Second
	// If the cache encounters a request for a cached object that has expired in the
	// last 30s, the cache will reply immediately with a stale response and fetch
	// the resource in a background process.
	// More Info: https://tools.ietf.org/html/rfc5861
	//
	//     microcache-stale-while-revalidate: 20
	//
	// - HashQuery: true
	// All query parameters are included in the request hash
	//
	// - QueryIgnore: []string{}
	// A list of query parameters to ignore when hashing the request
	// Add oauth parameters or other unwanted cache busters to this list
	//
	// - Exposed: true
	// Header will be appended to response indicating HIT / MISS / STALE
	//
	//     microcache: ( HIT | MISS | STALE )
	//
	// - SuppressAgeHeader: false
	// Age is a standard HTTP header indicating the age of the cached object in seconds
	// The Age header is added by default to all HIT and MISS responses
	// This parameter prevents the Age header from being set
	//
	//     Age: ( seconds )
	//
	// - Monitor: microcache.MonitorFunc(5 * time.Second, logStats)
	// LogStats will be called every 5s to log stats about the cache
	//
	cache := microcache.New(microcache.Config{
		Nocache:              true,
		Timeout:              3 * time.Second,
		TTL:                  30 * time.Second,
		StaleIfError:         3600 * time.Second,
		StaleRecache:         true,
		StaleWhileRevalidate: 30 * time.Second,
		CollapsedForwarding:  true,
		HashQuery:            true,
		QueryIgnore:          []string{},
		Exposed:              true,
		SuppressAgeHeader:    false,
		Monitor:              microcache.MonitorFunc(5*time.Second, logStats),
		Driver:               microcache.NewDriverLRU(1e4),
		Compressor:           microcache.CompressorSnappy{},
	})

	h := cache.Middleware(handler{})

	http.ListenAndServe(":80", h)
}

Benefits

May improve service efficiency by reducing origin read traffic

  • ttl - response caching with global or request specific ttl
  • collapsed-forwarding - deduplicate requests for cacheable resources

May improve client facing response time variability

  • stale-while-revalidate - serve stale content while fetching cacheable resources in the background

May improve service availability

  • request-timeout - kill long running requests
  • stale-if-error - serve stale responses on error (or request timeout)
  • stale-recache - recache stale responses following stale-if-error

Supports content negotiation with global and request specific cache splintering

  • vary - splinter requests by request header value
  • vary-query - splinter requests by URL query parameter value

Release

Tests have been written to confirm the correct behavior of this cache.

At least one large scale deploy of this library is currently underway in an API serving 20,000 requests per minute at peak. Results pending.

Compression

A Snappy driver has been added for projects who want to trade CPU for memory over gzip

Snappy provides:

  • 14x faster compression over gzip
  • 8x faster expansion over gzip
  • but the result is 1.5 - 2x the size compared to gzip (for specific json examples)

Your mileage may vary. See compare_compression.go to test your specific workloads

> go run tools/compare_compression.go -f large.json
Original: 616,611 bytes of json
zlib   compress 719.853807ms  61,040 bytes (10.1x)
gzip   compress 720.731066ms  61,052 bytes (10.1x)
snappy compress 48.836002ms  106,613 bytes (5.8x)
zlib   expand 211.538416ms
gzip   expand 220.011961ms
snappy expand 26.973263ms

> go run tools/compare_compression.go -f medium.json
Original: 279,368 bytes of json
zlib   compress 282.549098ms 19,825 bytes (14.1x)
gzip   compress 275.961026ms 19,837 bytes (14.1x)
snappy compress 16.452706ms  37,096 bytes (7.5x)
zlib   expand 86.704103ms
gzip   expand 81.188856ms
snappy expand 10.557594ms

> go run tools/compare_compression.go -f small.json
Original: 53,129 bytes of json
zlib   compress 73.204418ms 5,084 bytes (10.5x)
gzip   compress 74.150401ms 5,096 bytes (10.4x)
snappy compress 5.225558ms  8,412 bytes (6.3x)
zlib   expand 18.764693ms
gzip   expand 18.797717ms
snappy expand 2.354814ms

Benchmarks

All benchmarks are lies. Dual core 3.3Ghz i7 DDR4 Centos 7 VM w/ 10KB response (see example above)

> gobench -u http://localhost/ -c 10 -t 10
Dispatching 10 clients
Waiting for results...

Requests:                           110705 hits
Successful requests:                110705 hits
Network failed:                          0 hits
Bad requests failed (!2xx):              0 hits
Successful requests rate:            11070 hits/sec
Read throughput:                1109430818 bytes/sec
Write throughput:                   896791 bytes/sec
Test time:                              10 sec

Notes

Vary query by parameter presence as well as value

Modify Monitor.Error to accept request, response and error
Add Monitor.Timeout accepting request, response and error

Separate middleware:
  Sanitize lang header? (first language)
  Sanitize region? (country code)

etag support?
if-modified-since support?
HTCP?
TCI?
Custom rule handling?
  Passthrough: func(r) bool

Documentation

Overview

microcache is a non-standard HTTP microcache implemented as Go middleware.

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

This section is empty.

Types

type Compressor 

type Compressor interface {

	// Compress compresses a response prior to being saved in the cache and returns a clone
	// usually by compressing the response body
	Compress(Response) Response

	// Expand decompresses a response's body (destructively)
	Expand(Response) Response
}

Compressor is the interface for response compressors

type CompressorGzip 

type CompressorGzip struct {
}

CompressorGzip is a gzip compressor

func (CompressorGzip) Compress 

func (c CompressorGzip) Compress(res Response) Response

func (CompressorGzip) Expand 

func (c CompressorGzip) Expand(res Response) Response

type CompressorSnappy 

type CompressorSnappy struct {
}

CompressorSnappy is a Snappy compressor 14x faster compress than gzip 8x faster expand than gzip ~ 1.5 - 2x larger result (see README)

func (CompressorSnappy) Compress 

func (c CompressorSnappy) Compress(res Response) Response

func (CompressorSnappy) Expand 

func (c CompressorSnappy) Expand(res Response) Response

type Config 

type Config struct {
	// Nocache prevents responses from being cached by default
	// Can be overridden by the microcache-cache and microcache-nocache response headers
	Nocache bool

	// Timeout specifies the maximum execution time for backend responses
	// Example: If the underlying handler takes more than 10s to respond,
	// the request is cancelled and the response is treated as 503
	// Recommended: 10s
	// Default: 0
	Timeout time.Duration

	// TTL specifies a default ttl for cached responses
	// Can be overridden by the microcache-ttl response header
	// Recommended: 10s
	// Default: 0
	TTL time.Duration

	// StaleWhileRevalidate specifies a period during which a stale response may be
	// served immediately while the resource is fetched in the background. This can be
	// useful for ensuring consistent response times at the cost of content freshness.
	// More Info: https://tools.ietf.org/html/rfc5861
	// Recommended: 20s
	// Default: 0
	StaleWhileRevalidate time.Duration

	// StaleIfError specifies a default stale grace period
	// If a request fails and StaleIfError is set, the object will be served as stale
	// and the response will be re-cached for the duration of this grace period
	// Can be overridden by the microcache-ttl-stale response header
	// More Info: https://tools.ietf.org/html/rfc5861
	// Recommended: 20s
	// Default: 0
	StaleIfError time.Duration

	// StaleRecache specifies whether to re-cache the response object for ttl while serving
	// stale response on backend error
	// Recommended: true
	// Default: false
	StaleRecache bool

	// CollapsedForwarding specifies whether to collapse duplicate requests
	// This helps prevent servers with a cold cache from hammering the backend
	// Default: false
	CollapsedForwarding bool

	// HashQuery determines whether all query parameters in the request URI
	// should be hashed to differentiate requests
	// Default: false
	HashQuery bool

	// QueryIgnore is a list of query parameters to ignore when hashing
	// Default: nil
	QueryIgnore []string

	// Vary specifies a list of http request headers by which all requests
	// should be differentiated. When making use of this option, it may be a good idea
	// to normalize these headers first using a separate piece of middleware.
	//
	//   []string{"accept-language", "accept-encoding", "xml-http-request"}
	//
	// Default: []string{}
	Vary []string

	// Driver specifies a cache storage driver
	// Default: lru with 10,000 item capacity
	Driver Driver

	// Compressor specifies a compressor to use for reducing the memory required to cache
	// response bodies
	// Default: nil
	Compressor Compressor

	// Monitor is an optional parameter which will periodically report statistics about
	// the cache to enable monitoring of cache size, cache efficiency and error rate
	// Default: nil
	Monitor Monitor

	// Exposed determines whether to add a header to the response indicating the response state
	// Microcache: ( HIT | MISS | STALE )
	// Default: false
	Exposed bool

	// SuppressAgeHeader determines whether to suppress the age header in responses
	// The age header is added by default to all HIT and STALE responses
	// Age: ( seconds )
	// Default: false
	SuppressAgeHeader bool
}

type Driver 

type Driver interface {

	// SetRequestOpts stores request options in the request cache.
	// Requests contain request-specific cache configuration based on response headers
	SetRequestOpts(string, RequestOpts) error

	// GetRequestOpts retrieves request options from the request cache
	GetRequestOpts(string) RequestOpts

	// Set stores a response object in the response cache.
	// This contains the full response as well as an expiration date.
	Set(string, Response) error

	// Get retrieves a response object from the response cache
	Get(string) Response

	// Remove removes a response object from the response cache.
	// Required by HTTP spec to purge cached responses after successful unsafe request.
	Remove(string) error

	// GetSize returns the number of objects stored in the cache
	GetSize() int
}

Driver is the interface for cache drivers

type DriverARC 

type DriverARC struct {
	RequestCache  *lru.ARCCache
	ResponseCache *lru.ARCCache
}

DriverARC is a driver implementation using github.com/hashicorp/golang-lru ARCCache is a thread-safe fixed size Adaptive Replacement Cache (ARC). It requires more ram and cpu than straight LRU but can be more efficient https://godoc.org/github.com/hashicorp/golang-lru#ARCCache

func NewDriverARC 

func NewDriverARC(size int) DriverARC

NewDriverARC returns an ARC driver. size determines the number of items in the cache. Memory usage should be considered when choosing the appropriate cache size. The amount of memory consumed by the driver will depend upon the response size. Roughly, memory = cacheSize * averageResponseSize / compression ratio ARC caches have additional CPU and memory overhead when compared with LRU ARC does not support eviction monitoring

func (DriverARC) Get 

func (c DriverARC) Get(hash string) (res Response)

func (DriverARC) GetRequestOpts 

func (c DriverARC) GetRequestOpts(hash string) (req RequestOpts)

func (DriverARC) GetSize 

func (c DriverARC) GetSize() int

func (DriverARC) Remove 

func (c DriverARC) Remove(hash string) error

func (DriverARC) Set 

func (c DriverARC) Set(hash string, res Response) error

func (DriverARC) SetRequestOpts 

func (c DriverARC) SetRequestOpts(hash string, req RequestOpts) error

type DriverLRU 

type DriverLRU struct {
	RequestCache  *lru.Cache
	ResponseCache *lru.Cache
}

DriverLRU is a driver implementation using github.com/hashicorp/golang-lru

func NewDriverLRU 

func NewDriverLRU(size int) DriverLRU

NewDriverLRU returns the default LRU driver configuration. size determines the number of items in the cache. Memory usage should be considered when choosing the appropriate cache size. The amount of memory consumed by the driver will depend upon the response size. Roughly, memory = cacheSize * averageResponseSize / compression ratio

func (DriverLRU) Get 

func (c DriverLRU) Get(hash string) (res Response)

func (DriverLRU) GetRequestOpts 

func (c DriverLRU) GetRequestOpts(hash string) (req RequestOpts)

func (DriverLRU) GetSize 

func (c DriverLRU) GetSize() int

func (DriverLRU) Remove 

func (c DriverLRU) Remove(hash string) error

func (DriverLRU) Set 

func (c DriverLRU) Set(hash string, res Response) error

func (DriverLRU) SetRequestOpts 

func (c DriverLRU) SetRequestOpts(hash string, req RequestOpts) error

type Microcache 

type Microcache interface {
	Middleware(http.Handler) http.Handler
	Start()
	Stop()
	// contains filtered or unexported methods
}

func New 

func New(o Config) Microcache

New creates and returns a configured microcache instance

type Monitor 

type Monitor interface {
	GetInterval() time.Duration
	Log(Stats)
	Hit()
	Miss()
	Stale()
	Backend()
	Error()
}

Monitor is an interface for collecting metrics about the microcache

func MonitorFunc 

func MonitorFunc(interval time.Duration, logFunc func(Stats)) Monitor

MonitorFunc turns a function into a Monitor

type RequestOpts 

type RequestOpts struct {
	// contains filtered or unexported fields
}

RequestOpts stores per-request cache options. This is necessary to allow custom response headers to be evaluated, cached and applied prior to response object retrieval (ie. microcache-vary, microcache-nocache, etc)

type Response 

type Response struct {
	// contains filtered or unexported fields
}

Response is used both as a cache object for the response and to wrap http.ResponseWriter for downstream requests.

func (*Response) Header 

func (res *Response) Header() http.Header

func (*Response) Write 

func (res *Response) Write(b []byte) (int, error)

func (*Response) WriteHeader 

func (res *Response) WriteHeader(code int)

type Stats 

type Stats struct {
	Size    int
	Hits    int
	Misses  int
	Stales  int
	Backend int
	Errors  int
}

Source Files

View all Source files

Directories

Path Synopsis
examples
basic
tools
compare_compression
random_url_generator

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.