downloader

README

ACKNOWLEDGEMENT

This download package is a heavily-modified fork of gregjones/httpcache.

httpcache

Package httpcache provides a http.RoundTripper implementation that works as a mostly RFC 7234 compliant cache for http responses. This incarnation of the library is an active fork of github.com/gregjones/httpcache which is unmaintained.

It is only suitable for use as a 'private' cache (i.e. for a web-browser or an API-client and not for a shared proxy).

Cache Backends

• The built-in 'memory' cache stores responses in an in-memory map. - github.com/bitcomplete/httpcache/diskcache provides a filesystem-backed cache using the diskv library.
• github.com/bitcomplete/httpcache/memcache provides memcache implementations, for both App Engine and 'normal' memcache servers.
• sourcegraph.com/sourcegraph/s3cache uses Amazon S3 for storage.
• github.com/bitcomplete/httpcache/leveldbcache provides a filesystem-backed cache using leveldb.
• github.com/die-net/lrucache provides an in-memory cache that will evict least-recently used entries.
• github.com/die-net/lrucache/twotier allows caches to be combined, for example to use lrucache above with a persistent disk-cache.
• github.com/birkelund/boltdbcache provides a BoltDB implementation (based on the bbolt fork).

If you implement any other backend and wish it to be linked here, please send a PR editing this file.

License

• MIT License

Documentation

Overview

Package downloader provides a mechanism for getting files from HTTP/S URLs, making use of a mostly RFC-compliant cache.

The entrypoint is downloader.New.

Acknowledgement: This package is a heavily customized fork of https://github.com/gregjones/httpcache, via bitcomplete/download.

Index

• Constants
• Variables
• type Downloader
  • func New(name string, c *http.Client, dlURL, dlDir string) (*Downloader, error)
  • func (dl *Downloader) CacheFile(ctx context.Context) (fp string, err error)
  • func (dl *Downloader) Checksum(ctx context.Context) (sum checksum.Checksum, ok bool)
  • func (dl *Downloader) Clear(ctx context.Context) error
  • func (dl *Downloader) Get(ctx context.Context) (dlFile string, dlStream *streamcache.Stream, err error)
  • func (dl *Downloader) State(ctx context.Context) State
• type Handler
• type SinkHandler
  • func NewSinkHandler(log *slog.Logger) *SinkHandler
  • func (sh *SinkHandler) Reset()
• type State
  • func (s State) String() string

Constants

const XFromCache = "X-From-Stream"

XFromCache is the header added to responses that are returned from the cache.

Variables

var OptCache = options.NewBool(
	"download.cache",
	nil,
	true,
	"Cache downloads",
	`Cache downloaded remote files. When false, the download cache is not used and
the file is re-downloaded on each command.`,
	options.TagSource,
)

var OptContinueOnError = options.NewBool(
	"download.refresh.ok-on-err",
	nil,
	true,
	"Continue with stale download if refresh fails",
	`Continue with stale download if refresh fails. This option applies if a download
is in the cache, but is considered stale, and a refresh attempt fails. If set to
true, the refresh error is logged, and the stale download is returned. This is a
sort of "Airplane Mode" for downloads: when true, sq continues with the cached
download when the network is unavailable. If false, an error is returned instead.`,
	options.TagSource,
)

Types

type Downloader

type Downloader struct {
	// contains filtered or unexported fields
}

Downloader encapsulates downloading a file from a URL, using a local disk cache if possible. Downloader.Get returns either a filepath to the already-downloaded file, or a stream of the download in progress, or an error. If a stream is returned, the Downloader cache is updated when the stream is fully consumed (this can be observed by the closing of the channel returned by streamcache.Stream.Filled).

To be extra clear about that last point: the caller must consume the stream returned by Downloader.Get, or the cache will not be written.

func New

func New(name string, c *http.Client, dlURL, dlDir string) (*Downloader, error)

New returns a new Downloader for url that caches downloads in dlDir. Arg name is a user-friendly label, such as a source handle like @data. The name may show up in logs, or progress indicators etc.

func (*Downloader) CacheFile

func (dl *Downloader) CacheFile(ctx context.Context) (fp string, err error)

CacheFile returns the path to the cached file, if it exists. If there's a download in progress (Downloader.Get returned a stream), then CacheFile may return the filepath to the previously cached file. The caller should wait on any previously returned download stream to complete to ensure that the returned filepath is that of the current download. The caller can also check the cache state via Downloader.State.

func (*Downloader) Checksum

func (dl *Downloader) Checksum(ctx context.Context) (sum checksum.Checksum, ok bool)

Checksum returns the checksum of the cached download, if available.

func (*Downloader) Clear

func (dl *Downloader) Clear(ctx context.Context) error

Clear deletes the cache.

func (*Downloader) Get

func (dl *Downloader) Get(ctx context.Context) (dlFile string, dlStream *streamcache.Stream, err error)

Get attempts to get the remote file, returning either the filepath of the already-cached file in dlFile, or a stream of a newly-started download in dlStream, or an error. Exactly one of the return values will be non-nil.

• If dlFile is non-empty, it is the filepath on disk of the cached download, and dlStream and err are nil. However, depending on OptContinueOnError, dlFile may be the path to a stale download. If the cache is stale and a transport error occurs during refresh, and OptContinueOnError is true, the previous cached download is returned. If OptContinueOnError is false, the transport error is returned, and dlFile is empty. The caller can also check the cache state via Downloader.State.
• If dlStream is non-nil, it is a stream of the download in progress, and dlFile is empty. The cache is updated when the stream has been completely consumed. If the stream is not consumed, the cache is not updated. If an error occurs reading from the stream, the cache is also not updated: this means that the cache may still contain the previous (stale) download.
• If err is non-nil, there was an unrecoverable problem (e.g. a transport error, and there's no previous cache) and the download is unavailable.

Get consults the context for options. In particular, it makes use of OptCache and OptContinueOnError.

func (*Downloader) State

func (dl *Downloader) State(ctx context.Context) State

State returns the Downloader's cache state.

type Handler

type Handler struct {
	// Cached is invoked when the download is already cached on disk. The
	// dlFile arg is the path to the downloaded file.
	Cached func(dlFile string)

	// Uncached is invoked when the download is not cached on disk and
	// downloading has begun. The dlStream arg can be used to read the
	// bytes as would be returned from resp.Body. Downloader.Get will
	// continue the download process after Uncached returns. The caller
	// can wait on the download to complete using the channel returned
	// by streamcache.Stream's Filled method.
	Uncached func(dlStream *streamcache.Stream)

	// Error is invoked by Downloader.Get if an error occurs before Handler.Cached
	// or Handler.Uncached is invoked. If Uncached is invoked, any error from
	// reading the download resp.Body will be returned when reading
	// from the streamcache.Stream provided to Uncached.
	Error func(err error)
}

Handler is a callback invoked by Downloader.Get. Exactly one of the handler functions will be invoked, exactly one time. The handler is called as early as possible, and Downloader.Get may continue afterwards, e.g. to download the file. This mechanism allows the caller to start processing the download stream before the download completes.

type SinkHandler

type SinkHandler struct {
	Handler

	// Errors records the errors received via Handler.Error.
	Errors []error

	// Downloaded records the already-downloaded files received via Handler.Cached.
	Downloaded []string

	// Streams records the streams received via Handler.Uncached.
	Streams []*streamcache.Stream
	// contains filtered or unexported fields
}

SinkHandler is a downloader.Handler that records the results of the callbacks it receives. This is used for testing.

func NewSinkHandler

func NewSinkHandler(log *slog.Logger) *SinkHandler

NewSinkHandler returns a new SinkHandler.

func (*SinkHandler) Reset

func (sh *SinkHandler) Reset()

Reset resets the handler sinks. It also closes the source reader of any streams that were received via Handler.Uncached.

type State

type State int

State is an enumeration of caching states based on the cache-control values of the request and the response.

• Uncached indicates the item is not cached.
• Fresh indicates that the cached item can be returned.
• Stale indicates that the cached item needs validating before it is returned.
• Transparent indicates the cached item should not be used to fulfil the request.

Because this is only a private cache, 'public' and 'private' in cache-control aren't significant. Similarly, smax-age isn't used.

const (
	// Uncached indicates that the item is not cached.
	Uncached State = iota

	// Stale indicates that the cached item needs validating before it is returned.
	Stale

	// Fresh indicates the cached item can be returned.
	Fresh

	// Transparent indicates the cached item should not be used to fulfil the request.
	Transparent
)

func (State) String

func (s State) String() string

String returns a string representation of State.