s3cache

Documentation

Overview

Package s3cache implements callbacks for a gocache.Server that store data into an S3 bucket through a local directory.

Index

• type Cache
  • func (s *Cache) Close(ctx context.Context) error
  • func (s *Cache) Get(ctx context.Context, actionID string) (objectID, diskPath string, _ error)
  • func (s *Cache) Put(ctx context.Context, obj gocache.Object) (diskPath string, _ error)
  • func (s *Cache) SetMetrics(_ context.Context, m *expvar.Map)

Types

type Cache

type Cache struct {
	// Local is the local cache directory where actions and objects are staged.
	// It must be non-nil. A local stage is required because the Go toolchain
	// needs direct access to read the files reported by the cache.
	// It is safe to use a tmpfs directory.
	Local *cachedir.Dir

	// S3Client is the S3 client used to read and write cache entries to the
	// backing store. It must be non-nil.
	S3Client *s3util.Client

	// KeyPrefix, if non-empty, is prepended to each key stored into S3, with an
	// intervening slash.
	KeyPrefix string

	// MinUploadSize, if positive, defines a minimum object size in bytes below
	// which the cache will not write the object to S3.
	MinUploadSize int64

	// UploadConcurrency, if positive, defines the maximum number of concurrent
	// tasks for writing cache entries to S3.  If zero or negative, it uses
	// runtime.NumCPU.
	UploadConcurrency int
	// contains filtered or unexported fields
}

Cache implements callbacks for a gocache.Server using an S3 bucket for backing store with a local directory for staging.

Remote Cache Layout

Within the designated S3 bucket, keys are organized into two groups. Each action is stored in a file named:

[<prefix>/]action/<xx>/<action-id>

Each output object is stored in a file named:

[<prefix>/]object/<xx>/<object-id>

The object and action IDs are encoded as lower-case hexadecimal strings, with "<xx>" denoting the first two bytes of the ID to partition the space.

The contents of each action file have the format:

<object-id> <timestamp>

where the object ID is hex encoded and the timestamp is Unix nanoseconds. The object file contains just the binary data of the object.

func (*Cache) Close

func (s *Cache) Close(ctx context.Context) error

Close implements the corresponding callback of the cache protocol.

func (*Cache) Get

func (s *Cache) Get(ctx context.Context, actionID string) (objectID, diskPath string, _ error)

Get implements the corresponding callback of the cache protocol.

func (*Cache) Put

func (s *Cache) Put(ctx context.Context, obj gocache.Object) (diskPath string, _ error)

Put implements the corresponding callback of the cache protocol.

func (*Cache) SetMetrics

func (s *Cache) SetMetrics(_ context.Context, m *expvar.Map)

SetMetrics implements the corresponding server callback.