cache

Documentation

Overview

Package cache provides a http.RoundTripper implementation that works as a mostly RFC-compliant cache for http responses.

It is only suitable for use as a 'private' cache (i.e. for a web-browser or an API-client and not for a shared proxy).

Mostly borrowed from https://github.com/gregjones/httpcache. Customized for different policies.

Index

• Constants
• Variables
• func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)
• func Date(respHeaders http.Header) (date time.Time, err error)
• func PleaseCache(t *testing.T, cache Cache)
• type Cache
• type Policy
• type Transport
  • func NewMemoryCacheTransport() *Transport
  • func NewTransport(c Cache) *Transport
  • func (t *Transport) Client() *http.Client
  • func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)
  • func (t *Transport) RoundTripDummy(req *http.Request) (resp *http.Response, err error)
  • func (t *Transport) RoundTripRFC2616(req *http.Request) (resp *http.Response, err error)

Constants

const (

	// XFromCache is the header added to responses that are returned from the cache
	XFromCache = "X-From-Cache"
)

Variables

var ErrNoDateHeader = errors.New("no Date header")

ErrNoDateHeader indicates that the HTTP headers contained no Date header.

Functions

func CachedResponse

func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)

CachedResponse returns the cached http.Response for req if present, and nil otherwise.

func Date

func Date(respHeaders http.Header) (date time.Time, err error)

Date parses and returns the value of the Date header.

func PleaseCache

func PleaseCache(t *testing.T, cache Cache)

PleaseCache excercises a Cache implementation.

Types

type Cache

type Cache interface {
	// Get returns the []byte representation of a cached response and a bool
	// set to true if the value isn't empty
	Get(key string) (responseBytes []byte, ok bool)
	// Set stores the []byte representation of a response against a key
	Set(key string, responseBytes []byte)
	// Delete removes the value associated with the key
	Delete(key string)
}

A Cache interface is used by the Transport to store and retrieve responses.

type Policy

type Policy int

const (

	// The Dummy policy is useful for testing spiders faster (without having to wait for downloads every time)
	// and for trying your spider offline, when an Internet connection is not available.
	// The goal is to be able to “replay” a spider run exactly as it ran before.
	Dummy Policy = iota

	// This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness,
	// aimed at production and used in continuous runs to avoid downloading unmodified data
	// (to save bandwidth and speed up crawls).
	RFC2616
)

type Transport

type Transport struct {
	Policy Policy
	// The RoundTripper interface actually used to make requests
	// If nil, http.DefaultTransport is used
	Transport http.RoundTripper
	Cache     Cache
	// If true, responses returned from the cache will be given an extra header, X-From-Cache
	MarkCachedResponses bool
}

Transport is an implementation of http.RoundTripper that will return values from a cache where possible (avoiding a network request) and will additionally add validators (etag/if-modified-since) to repeated requests allowing servers to return 304 / Not Modified

func NewMemoryCacheTransport

func NewMemoryCacheTransport() *Transport

NewMemoryCacheTransport returns a new Transport using the in-memory cache implementation

func NewTransport

func NewTransport(c Cache) *Transport

NewTransport returns a new Transport with the provided Cache implementation and MarkCachedResponses set to true

func (*Transport) Client

func (t *Transport) Client() *http.Client

Client returns an *http.Client that caches responses.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip is a wrapper for caching requests. If there is a fresh Response already in cache, then it will be returned without connecting to the server.

func (*Transport) RoundTripDummy

func (t *Transport) RoundTripDummy(req *http.Request) (resp *http.Response, err error)

RoundTripDummy has no awareness of any HTTP Cache-Control directives. Every request and its corresponding response are cached. When the same request is seen again, the response is returned without transferring anything from the Internet.

func (*Transport) RoundTripRFC2616

func (t *Transport) RoundTripRFC2616(req *http.Request) (resp *http.Response, err error)

RoundTripRFC2616 provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness, aimed at production and used in continuous runs to avoid downloading unmodified data (to save bandwidth and speed up crawls).

If there is a stale Response, then any validators it contains will be set on the new request to give the server a chance to respond with NotModified. If this happens, then the cached Response will be returned.