sharded

Documentation

Overview

Package sharded implements common support for sharded blob providers, such as filesystem or webdav.

Package sharded implements common support for sharded blob providers, such as filesystem or webdav.

Index

• Constants
• type Impl
• type Options
• type Parameters
  • func DefaultParameters(shards []int) *Parameters
  • func (p *Parameters) Clone() *Parameters
  • func (p *Parameters) GetShardDirectoryAndBlob(rootPath string, blobID blob.ID) (string, blob.ID)
  • func (p *Parameters) Load(r io.Reader) error
  • func (p *Parameters) Save(w io.Writer) error
• type PrefixAndShards
• type Storage
  • func New(impl Impl, rootPath string, opt Options, isCreate bool) Storage
  • func (s *Storage) DeleteBlob(ctx context.Context, blobID blob.ID) error
  • func (s *Storage) GetBlob(ctx context.Context, blobID blob.ID, offset, length int64, ...) error
  • func (s *Storage) GetMetadata(ctx context.Context, blobID blob.ID) (blob.Metadata, error)
  • func (s *Storage) GetShardedPathAndFilePath(ctx context.Context, blobID blob.ID) (shardPath, filePath string, err error)
  • func (s *Storage) ListBlobs(ctx context.Context, prefix blob.ID, callback func(blob.Metadata) error) error
  • func (s *Storage) PutBlob(ctx context.Context, blobID blob.ID, data blob.Bytes, opts blob.PutOptions) error

Constants

const CompleteBlobSuffix = ".f"

CompleteBlobSuffix is the extension for sharded blobs that have completed writing.

const ParametersFile = ".shards"

ParametersFile is the name of the hidden parameters file in a sharded storage.

Types

type Impl

type Impl interface {
	GetBlobFromPath(ctx context.Context, dirPath, filePath string, offset, length int64, output blob.OutputBuffer) error
	GetMetadataFromPath(ctx context.Context, dirPath, filePath string) (blob.Metadata, error)
	PutBlobInPath(ctx context.Context, dirPath, filePath string, dataSlices blob.Bytes, opts blob.PutOptions) error
	DeleteBlobInPath(ctx context.Context, dirPath, filePath string) error
	ReadDir(ctx context.Context, path string) ([]os.FileInfo, error)
}

Impl must be implemented by underlying provider.

type Options

type Options struct {
	DirectoryShards []int `json:"dirShards"`
	ListParallelism int   `json:"listParallelism,omitempty"`
}

Options must be anonymously embedded in sharded provider options.

type Parameters

type Parameters struct {
	DefaultShards   []int             `json:"default"`
	UnshardedLength int               `json:"maxNonShardedLength"`
	Overrides       []PrefixAndShards `json:"overrides,omitempty"`
}

Parameters contains sharded storage configuration optionally persisted in the storage itself.

func DefaultParameters

func DefaultParameters(shards []int) *Parameters

DefaultParameters constructs Parameters based on the provided shards specification.

func (*Parameters) Clone

func (p *Parameters) Clone() *Parameters

Clone returns a clone of sharding parameters.

func (*Parameters) GetShardDirectoryAndBlob

func (p *Parameters) GetShardDirectoryAndBlob(rootPath string, blobID blob.ID) (string, blob.ID)

GetShardDirectoryAndBlob gets that sharded directory and blob ID for a provided blob.

func (*Parameters) Load

func (p *Parameters) Load(r io.Reader) error

Load loads the Parameters from the provided reader.

func (*Parameters) Save

func (p *Parameters) Save(w io.Writer) error

Save saves the parameters to the provided writer.

type PrefixAndShards

type PrefixAndShards struct {
	Prefix blob.ID `json:"prefix"`
	Shards []int   `json:"shards"`
}

PrefixAndShards defines shards to use for a particular blob ID prefix.

type Storage

type Storage struct {
	Impl Impl

	RootPath string
	Options
	// contains filtered or unexported fields
}

Storage provides common implementation of sharded storage.

func New

func New(impl Impl, rootPath string, opt Options, isCreate bool) Storage

New returns new sharded.Storage helper.

func (*Storage) DeleteBlob

func (s *Storage) DeleteBlob(ctx context.Context, blobID blob.ID) error

DeleteBlob implements blob.Storage.

func (*Storage) GetBlob

func (s *Storage) GetBlob(ctx context.Context, blobID blob.ID, offset, length int64, output blob.OutputBuffer) error

GetBlob implements blob.Storage.

func (*Storage) GetMetadata

func (s *Storage) GetMetadata(ctx context.Context, blobID blob.ID) (blob.Metadata, error)

GetMetadata implements blob.Storage.

func (*Storage) GetShardedPathAndFilePath

func (s *Storage) GetShardedPathAndFilePath(ctx context.Context, blobID blob.ID) (shardPath, filePath string, err error)

GetShardedPathAndFilePath returns the path of the shard and file name within the shard for a given blob ID.

func (*Storage) ListBlobs

func (s *Storage) ListBlobs(ctx context.Context, prefix blob.ID, callback func(blob.Metadata) error) error

ListBlobs implements blob.Storage.

func (*Storage) PutBlob

func (s *Storage) PutBlob(ctx context.Context, blobID blob.ID, data blob.Bytes, opts blob.PutOptions) error

PutBlob implements blob.Storage.