s3proxy

Documentation

Overview

Package s3proxy implements components of a Go module proxy that caches files locally on disk, backed by objects in an S3 bucket.

Index

• type Cacher
  • func (c *Cacher) Close() error
  • func (c *Cacher) Get(ctx context.Context, name string) (_ io.ReadCloser, oerr error)
  • func (c *Cacher) Metrics() *expvar.Map
  • func (c *Cacher) Put(ctx context.Context, name string, data io.ReadSeeker) (oerr error)

Types

type Cacher

type Cacher struct {
	// Local is the path of a local cache directory where modules are cached.
	// It must be non-empty.
	Local string

	// S3Client is the S3 client used to read and write cache entries to the
	// backing store. It must be non-nil.
	S3Client *s3util.Client

	// KeyPrefix, if non-empty, is prepended to each key stored into S3, with an
	// intervening slash.
	KeyPrefix string

	// MaxTasks, if positive, limits the number of concurrent tasks that may be
	// interacting with S3. If zero or negative, the default is
	// [runtime.NumCPU].
	MaxTasks int

	// Logf, if non-nil, is used to write log messages. If nil, logs are
	// discarded.
	Logf func(string, ...any)

	// LogRequests, if true, enables detailed (but noisy) debug logging of all
	// requests handled by the cache. Logs are written to Logf.
	//
	// Each result is presented in the format:
	//
	//    B <op> "<name>" (<digest>)
	//    E <op> "<name>", err=<error>, <time> elapsed
	//
	// Where the operations are "GET" and "PUT". The "B" line is when the
	// operation began, and "E" when it ended. When a GET operation successfully
	// faults in a result from S3, the log is:
	//
	//    F GET "<name>" hit (<digest>)
	//
	// When a PUT operation finishes writing a value behind to S3, the log is:
	//
	//    W PUT "<name>", err=<error>, <time> elapsed
	//
	LogRequests bool
	// contains filtered or unexported fields
}

Cacher implements the github.com/goproxy/goproxy.Cacher interface using a local disk cache backed by an S3 bucket.

Cache Layout

Module cache files are stored under a SHA256 digest of the filename presented to the cache, encoded as hex and partitioned by the first two bytes of the digest:

For example:

SHA256("fizzlepug") → 160db4d719252162c87a9169e26deda33d2340770d0d540fd4c580c55008b2d6
<cache-dir>/module/16/160db4d719252162c87a9169e26deda33d2340770d0d540fd4c580c55008b2d6

When files are stored in S3, the same naming convention is used, but with the specified key prefix instead:

<key-prefix>/module/16/0db4d719252162c87a9169e26deda33d2340770d0d540fd4c580c55008b2d6

func (*Cacher) Close

func (c *Cacher) Close() error

Close waits until all background updates are complete.

func (*Cacher) Get

func (c *Cacher) Get(ctx context.Context, name string) (_ io.ReadCloser, oerr error)

Get implements a method of the goproxy.Cacher interface. It reports cache hits out of the local directory if available, or faults in from S3.

func (*Cacher) Metrics

func (c *Cacher) Metrics() *expvar.Map

Metrics returns a map of cacher metrics. The caller is responsible for publishing these metrics.

func (*Cacher) Put

func (c *Cacher) Put(ctx context.Context, name string, data io.ReadSeeker) (oerr error)

Put implements a method of the goproxy.Cacher interface. It stores data into the local directory and then writes it back to S3 in the background.