Back to godoc.org

Package storage

v0.20.5
Latest Go to latest
Published: Jul 24, 2020 | License: Apache-2.0 | Module: github.com/bufbuild/buf

Overview

Package storage implements a simple storage abstraction.

This is meant to abstract filesystem calls, as well as be a wrapper for in-memory or remote storage. It also provides a smaller attack vector as implementations can do verifications as to what is accessed and what is not.

Index

Package Files

Variables

var (
	// ErrIncompleteWrite is the error returned if a write is not complete
	ErrIncompleteWrite = errors.New("incomplete write")
	// ErrClosed is the error returned if a bucket or object is already closed.
	ErrClosed = errors.New("already closed")
	// ErrSetExternalPathUnsupported is the error returned if a bucket does not support SetExternalPath.
	ErrSetExternalPathUnsupported = errors.New("setting the external path is unsupported for this bucket")
)

func AllPaths

func AllPaths(ctx context.Context, readBucket ReadBucket, prefix string) ([]string, error)

AllPaths walks the bucket and gets all the paths.

func Copy

func Copy(
	ctx context.Context,
	from ReadBucket,
	to WriteBucket,
	options ...CopyOption,
) (int, error)

Copy copies the bucket at from to the bucket at to.

Copies done concurrently. Returns the number of files copied.

func Diff

func Diff(
	ctx context.Context,
	one ReadBucket,
	two ReadBucket,
	oneBucketName string,
	twoBucketName string,
) ([]byte, error)

Diff does a diff of the ReadBuckets.

func Exists

func Exists(ctx context.Context, readBucket ReadBucket, path string) (bool, error)

Exists returns true if the path exists, false otherwise.

Returns error on system error.

func IsExistsMultipleLocations

func IsExistsMultipleLocations(err error) bool

IsExistsMultipleLocations returns true if the error is for a path existing in multiple locations.

func IsNotExist

func IsNotExist(err error) bool

IsNotExist returns true for a error that is for a path not existing.

func NewErrExistsMultipleLocations

func NewErrExistsMultipleLocations(path string, externalPaths ...string) error

NewErrExistsMultipleLocations returns a new error if a path exists in multiple locations.

func NewErrNotExist

func NewErrNotExist(path string) error

NewErrNotExist returns a new error for a path not existing.

func PutPath

func PutPath(ctx context.Context, writeBucket WriteBucket, path string, data []byte) (retErr error)

PutPath puts the data at the path.

func ReadPath

func ReadPath(ctx context.Context, readBucket ReadBucket, path string) (_ []byte, retErr error)

ReadPath is analogous to ioutil.ReadFile.

Returns an error that fufills IsNotExist if the path does not exist.

func WalkReadObjects

func WalkReadObjects(
	ctx context.Context,
	readBucket ReadBucket,
	prefix string,
	f func(ReadObject) error,
) error

WalkReadObjects walks the bucket and calls get on each, closing the resulting ReadObjectCloser when done.

type CopyOption

type CopyOption func(*copyOptions)

CopyOption is an option for Copy.

func CopyWithExternalPaths

func CopyWithExternalPaths() CopyOption

CopyWithExternalPaths returns a new CopyOption that says to copy external paths.

The to WriteBucket must support setting external paths.

type Mapper

type Mapper interface {
	// Map maps the path to the full path.
	//
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	// If the path cannot be mapped, this returns false.
	MapPath(path string) (string, bool)
	// Map maps the prefix to the full prefix.
	//
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	// If the path cannot be mapped, this returns false.
	MapPrefix(prefix string) (string, bool)
	// UnmapFullPath maps the full path to the path.
	//
	// Returns false if the full path does not apply.
	// The path is expected to be normalized and validated.
	// The returned path is expected to be normalized and validated.
	UnmapFullPath(fullPath string) (string, bool, error)
	// contains filtered or unexported methods
}

Mapper is a path mapper.

This will cause a Bucket to operate as if the Mapper has all paths mapped.

func MapChain

func MapChain(mappers ...Mapper) Mapper

MapChain chains the mappers.

If any mapper does not match, this stops checking Mappers and returns an empty path and false. This is as opposed to MatchAnd, that runs every Matcher and returns the path regardless.

If the Mappers are empty, a no-op Mapper is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func MapOnPrefix

func MapOnPrefix(prefix string) Mapper

MapOnPrefix returns a Mapper that will map the Bucket as if it was created on the given prefix.

The prefix is expected to be normalized and validated.

type Matcher

type Matcher interface {
	Mapper
	// contains filtered or unexported methods
}

Matcher is a path matcher.

This will cause a Bucket to operate as if it only contains matching paths.

func MatchAnd

func MatchAnd(matchers ...Matcher) Matcher

MatchAnd returns an And of the Matchers.

func MatchNot

func MatchNot(matcher Matcher) Matcher

MatchNot returns an Not of the Matcher.

func MatchOr

func MatchOr(matchers ...Matcher) Matcher

MatchOr returns an Or of the Matchers.

func MatchPathContained

func MatchPathContained(containingDir string) Matcher

MatchPathContained returns a Matcher for the directory that matches on paths by contained by containingDir.

func MatchPathEqual

func MatchPathEqual(equalPath string) Matcher

MatchPathEqual returns a Matcher for the path.

func MatchPathEqualOrContained

func MatchPathEqualOrContained(equalOrContainingPath string) Matcher

MatchPathEqualOrContained returns a Matcher for the path that matches on paths equal or contained by equalOrContainingPath.

func MatchPathExt

func MatchPathExt(ext string) Matcher

MatchPathExt returns a Matcher for the extension.

type ObjectInfo

type ObjectInfo interface {
	// Size is the size of the object.
	//
	// For writes, the write size must sum up to this size when closed, otherwise ErrIncompleteWrite is returned.
	// For writes, any write over this size will return io.EOF.
	Size() uint32
	// Path is the path of the object.
	//
	// This will always correspond to a path within the Bucket. For sub-buckets, this is the sub-path, but the
	// external path will include the sub-bucket path.
	//
	// This path will always be normalized, validated, and non-empty.
	Path() string
	// ExternalPath is the path that identifies the object externally.
	//
	// This path is not necessarily a file path, and should only be used to
	// uniquely identify this file as compared to other assets, to for display
	// to users.
	//
	// The path will be unnormalized, if it is a file path.
	// The path will never be empty. If a given implementation has no external path, this falls back to path.
	//
	// Example:
	//   Directory: /foo/bar
	//   Path: baz/bat.proto
	//   ExternalPath: /foo/bar/baz/bat.proto
	//
	// Example:
	//   Directory: .
	//   Path: baz/bat.proto
	//   ExternalPath: baz/bat.proto
	//
	// Example:
	//   S3 Bucket: https://s3.amazonaws.com/foo
	//   Path: baz/bat.proto
	//   ExternalPath: s3://foo/baz/bat.proto
	ExternalPath() string
}

ObjectInfo contains object info.

type ReadBucket

type ReadBucket interface {
	// Get gets the path.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Get(ctx context.Context, path string) (ReadObjectCloser, error)
	// Stat gets info in the object.
	//
	// Returns ErrNotExist if the path does not exist, other error
	// if there is a system error.
	Stat(ctx context.Context, path string) (ObjectInfo, error)
	// Walk walks the bucket with the prefix, calling f on each path.
	//
	// Note that foo/barbaz will not be called for foo/bar, but will
	// be called for foo/bar/baz.
	//
	// All paths given to f are normalized and validated.
	// If f returns error, Walk will stop short and return this error.
	// Returns other error on system error.
	Walk(ctx context.Context, prefix string, f func(ObjectInfo) error) error
}

ReadBucket is a simple read-only bucket.

All paths are regular files - Buckets do not handle directories. All paths must be relative. All paths are cleaned and ToSlash'ed by each function. Paths must not jump the bucket context, that is after clean, they cannot contain "..".

func Map

func Map(readBucket ReadBucket, mappers ...Mapper) ReadBucket

Map maps the bucket.

If the Mappers are empty, the original ReadBucket is returned. If there is more than one Mapper, the Mappers are called in order for UnmapFullPath, with the order reversed for MapPath and MapPrefix.

That is, order these assuming you are starting with a full path and working to a path.

func Multi

func Multi(readBuckets ...ReadBucket) ReadBucket

Multi takes the union of the ReadBuckets.

If no readBuckets are given, this returns a no-op ReadBucket. If one readBucket is given, this returns the original ReadBucket. Otherwise, this returns a ReadBucket that will get from all buckets.

This expects and validates that no paths overlap between the ReadBuckets. This assumes that buckets are logically unique.

type ReadBucketCloser

type ReadBucketCloser interface {
	io.Closer
	ReadBucket
}

ReadBucketCloser is a read-only bucket that must be closed.

func NopReadBucketCloser

func NopReadBucketCloser(readBucket ReadBucket) ReadBucketCloser

NopReadBucketCloser returns a ReadBucketCloser for the ReadBucket.

type ReadObject

type ReadObject interface {
	ObjectInfo
	io.Reader
}

ReadObject is an object read from a bucket.

type ReadObjectCloser

type ReadObjectCloser interface {
	ReadObject
	io.Closer
}

ReadObjectCloser is a ReadObject with a closer.

It must be closed when done.

type ReadWriteBucket

type ReadWriteBucket interface {
	ReadBucket
	WriteBucket
}

ReadWriteBucket is a simple read/write bucket.

type ReadWriteBucketCloser

type ReadWriteBucketCloser interface {
	io.Closer
	ReadWriteBucket
}

ReadWriteBucketCloser is a read/write bucket that must be closed.

func NopReadWriteBucketCloser

func NopReadWriteBucketCloser(readWriteBucket ReadWriteBucket) ReadWriteBucketCloser

NopReadWriteBucketCloser returns a ReadWriteBucketCloser for the ReadWriteBucket.

type WriteBucket

type WriteBucket interface {
	// Put returns a WriteObjectCloser to write to the path.
	//
	// The path is truncated on close.
	//
	// Returns error on system error.
	Put(ctx context.Context, path string, size uint32) (WriteObjectCloser, error)
	// SetExternalPathSupported returns true if SetExternalPath is supported.
	//
	// For example, in-memory buckets may choose to return true so that object sources
	// are preserved, but filesystem buckets may choose to return false as they have
	// their own external paths.
	SetExternalPathSupported() bool
}

WriteBucket is a write-only bucket.

type WriteBucketCloser

type WriteBucketCloser interface {
	io.Closer
	WriteBucket
}

WriteBucketCloser is a write-only bucket that must be closed.

func NopWriteBucketCloser

func NopWriteBucketCloser(writeBucket WriteBucket) WriteBucketCloser

NopWriteBucketCloser returns a WriteBucketCloser for the WriteBucket.

type WriteObject

type WriteObject interface {
	io.Writer

	// ExternalPath attempts to explicitly set the external path for the new object.
	//
	// If SetExternalPathSupported returns false, this returns error.
	SetExternalPath(externalPath string) error
}

WriteObject object written to a bucket.

type WriteObjectCloser

type WriteObjectCloser interface {
	WriteObject
	io.Closer
}

WriteObjectCloser is a WriteObject with a closer.

It must be closed when done.

Documentation was rendered with GOOS=linux and GOARCH=amd64.

Jump to identifier

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to identifier