cache

Documentation

Overview

Package cache the data store

Index

• Constants
• Variables
• func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)
• func Date(respHeaders http.Header) (date time.Time, err error)
• func IsFromCache(res *http.Response) bool
• type Cache
• type Cookie
• type Options
• type Policy
• type Transport
  • func NewTransport(c Cache) *Transport
  • func (t *Transport) Client() *http.Client
  • func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)
  • func (t *Transport) RoundTripDummy(req *http.Request) (resp *http.Response, err error)
  • func (t *Transport) RoundTripRFC2616(req *http.Request) (resp *http.Response, err error)
  • func (t *Transport) SetProxy(proxy func(*http.Request) (*url.URL, error))

Constants

const (

	// XFromCache is the header added to responses that are returned from the cache
	XFromCache = "X-From-Cache"
	// DefaultPath the default cache path
	DefaultPath = "cache"
)

Variables

var ErrNoDateHeader = errors.New("no Date header")

ErrNoDateHeader indicates that the HTTP headers contained no Date header.

Functions

func CachedResponse

func CachedResponse(c Cache, req *http.Request) (resp *http.Response, err error)

CachedResponse returns the cached http.Response for req if present, and nil otherwise.

func Date

func Date(respHeaders http.Header) (date time.Time, err error)

Date parses and returns the value of the Date header.

func IsFromCache

func IsFromCache(res *http.Response) bool

IsFromCache returns true if the response is from cache

Types

type Cache

type Cache interface {
	Get(key string) ([]byte, bool)
	Set(key string, value []byte)
	SetWithTimeout(key string, value []byte, timeout time.Duration)
	Del(key string)
}

A Cache interface is used to store bytes.

type Cookie

type Cookie interface {
	http.CookieJar

	// SetCookieString handles the receipt of the cookies string in a reply for the given URL.
	SetCookieString(u *url.URL, cookies string)
	// CookieString returns the cookies string for the given URL.
	CookieString(u *url.URL) string
	// DeleteCookie delete the cookies for the given URL.
	DeleteCookie(u *url.URL)
}

Cookie manages storage and use of cookies in HTTP requests. Implementations of Cookie must be safe for concurrent use by multiple goroutines.

type Options

type Options struct {
	Path                string        `yaml:"path"`
	ExpireCleanInterval time.Duration `yaml:"expire-clean-interval"`
}

Options the cache configuration

type Policy

type Policy string

Policy has no awareness of any HTTP Cache-Control directives.

const (

	// Dummy policy is useful for testing spiders faster (without having to wait for downloads every time)
	// and for trying your spider offline, when an Internet connection is not available.
	// The goal is to be able to “replay” a spider run exactly as it ran before.
	Dummy Policy = "dummy"

	// RFC2616 This policy provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness,
	// aimed at production and used in continuous runs to avoid downloading unmodified data
	// (to save bandwidth and speed up crawls).
	RFC2616 Policy = "rfc2616"
)

type Transport

type Transport struct {
	Policy Policy
	// The RoundTripper interface actually used to make requests
	// If nil, http.DefaultTransport is used
	Transport http.RoundTripper
	Cache     Cache
	// If true, responses returned from the cache will be given an extra header, X-From-Cache
	MarkCachedResponses bool
}

Transport is an implementation of http.RoundTripper that will return values from a cache where possible (avoiding a network request) and will additionally add validators (etag/if-modified-since) to repeated requests allowing servers to return 304 / Not Modified

func NewTransport

func NewTransport(c Cache) *Transport

NewTransport returns new Transport with the provided Cache implementation and MarkCachedResponses set to true

func (*Transport) Client

func (t *Transport) Client() *http.Client

Client returns an *http.Client that caches responses.

func (*Transport) RoundTrip

func (t *Transport) RoundTrip(req *http.Request) (resp *http.Response, err error)

RoundTrip is a wrapper for caching requests. If there is a fresh Response already in cache, then it will be returned without connecting to the server.

func (*Transport) RoundTripDummy

func (t *Transport) RoundTripDummy(req *http.Request) (resp *http.Response, err error)

RoundTripDummy has no awareness of any HTTP Cache-Control directives. Every request and its corresponding response are cached. When the same request is seen again, the response is returned without transferring anything from the Internet.

func (*Transport) RoundTripRFC2616

func (t *Transport) RoundTripRFC2616(req *http.Request) (resp *http.Response, err error)

RoundTripRFC2616 provides a RFC2616 compliant HTTP cache, i.e. with HTTP Cache-Control awareness, aimed at production and used in continuous runs to avoid downloading unmodified data (to save bandwidth and speed up crawls).

If there is a stale Response, then any validators it contains will be set on the new request to give the server a chance to respond with NotModified. If this happens, then the cached Response will be returned.

func (*Transport) SetProxy

func (t *Transport) SetProxy(proxy func(*http.Request) (*url.URL, error))

SetProxy specifies a function to return a proxy for a given Request.