storage

package
v0.3.0-pre1 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jan 21, 2019 License: Apache-2.0 Imports: 5 Imported by: 0

Documentation

Overview

Package storage implements simple storage of immutable, unstructured binary large objects (BLOBs).

Index

Constants

This section is empty.

Variables

View Source
var ErrBlockNotFound = errors.New("block not found")

ErrBlockNotFound is returned when a block cannot be found in storage.

Functions

func AddSupportedStorage

func AddSupportedStorage(
	urlScheme string,
	defaultConfigFunc func() interface{},
	createStorageFunc func(context.Context, interface{}) (Storage, error))

AddSupportedStorage registers factory function to create storage with a given type name.

func WithUploadProgressCallback

func WithUploadProgressCallback(ctx context.Context, callback ProgressFunc) context.Context

WithUploadProgressCallback returns a context that passes callback function to be used storage upload progress.

Types

type BlockMetadata

type BlockMetadata struct {
	BlockID   string
	Length    int64
	Timestamp time.Time
}

BlockMetadata represents metadata about a single block in a storage.

func ListAllBlocks

func ListAllBlocks(ctx context.Context, st Storage, prefix string) ([]BlockMetadata, error)

ListAllBlocks returns BlockMetadata for all blocks in a given storage that have the provided name prefix.

func ListAllBlocksConsistent

func ListAllBlocksConsistent(ctx context.Context, st Storage, prefix string, maxAttempts int) ([]BlockMetadata, error)

ListAllBlocksConsistent lists all blocks with given name prefix in the provided storage until the results are consistent. The results are consistent if the list result fetched twice is identical. This guarantees that while the first scan was in progress, no new block was added or removed. maxAttempts specifies maximum number of list attempts (must be >= 2)

type CancelFunc

type CancelFunc func()

CancelFunc requests cancellation of a storage operation.

type ConnectionInfo

type ConnectionInfo struct {
	Type   string
	Config interface{}
}

ConnectionInfo represents JSON-serializable configuration of a blob storage.

func (ConnectionInfo) MarshalJSON

func (c ConnectionInfo) MarshalJSON() ([]byte, error)

MarshalJSON returns JSON-encoded storage configuration.

func (*ConnectionInfo) UnmarshalJSON

func (c *ConnectionInfo) UnmarshalJSON(b []byte) error

UnmarshalJSON parses the JSON-encoded data into ConnectionInfo.

type ProgressFunc

type ProgressFunc func(desc string, completed, total int64)

ProgressFunc is used to report progress of a long-running storage operation.

func ProgressCallback

func ProgressCallback(ctx context.Context) ProgressFunc

ProgressCallback gets the progress callback function from the context.

type Storage

type Storage interface {
	// PutBlock uploads the block with given data to the repository or replaces existing block with the provided
	// id with given contents.
	PutBlock(ctx context.Context, id string, data []byte) error

	// DeleteBlock removes the block from storage. Future GetBlock() operations will fail with ErrBlockNotFound.
	DeleteBlock(ctx context.Context, id string) error

	// GetBlock returns full or partial contents of a block with given ID.
	// If length>0, the the function retrieves a range of bytes [offset,offset+length)
	// If length<0, the entire block must be fetched.
	GetBlock(ctx context.Context, id string, offset, length int64) ([]byte, error)

	// ListBlocks returns a channel of BlockMetadata that describes storage blocks with existing name prefixes.
	// Iteration continues until all blocks have been listed or until client code invokes the returned cancellation function.
	ListBlocks(ctx context.Context, prefix string, cb func(bm BlockMetadata) error) error

	// ConnectionInfo returns JSON-serializable data structure containing information required to
	// connect to storage.
	ConnectionInfo() ConnectionInfo

	// Close releases all resources associated with storage.
	Close(ctx context.Context) error
}

Storage encapsulates API for connecting to blob storage.

The underlying storage system must provide:

* high durability, availability and bit-rot protection * read-after-write - block written using PutBlock() must be immediately readable using GetBlock() and ListBlocks() * atomicity - it mustn't be possible to observe partial results of PutBlock() via either GetBlock() or ListBlocks() * timestamps that don't go back in time (small clock skew up to minutes is allowed) * reasonably low latency for retrievals

The required semantics are provided by existing commercial cloud storage products (Google Cloud, AWS, Azure).

func NewStorage

func NewStorage(ctx context.Context, cfg ConnectionInfo) (Storage, error)

NewStorage creates new storage based on ConnectionInfo. The storage type must be previously registered using AddSupportedStorage.

Directories

Path Synopsis
Package filesystem implements filesystem-based Storage.
Package filesystem implements filesystem-based Storage.
Package gcs implements Storage based on Google Cloud Storage bucket.
Package gcs implements Storage based on Google Cloud Storage bucket.
Package logging implements wrapper around Storage that logs all activity.
Package logging implements wrapper around Storage that logs all activity.
Package providers registers all storage providers that are included as part of Kopia.
Package providers registers all storage providers that are included as part of Kopia.
Package s3 implements Storage based on an S3 bucket.
Package s3 implements Storage based on an S3 bucket.
Package webdav implements WebDAV-based Storage.
Package webdav implements WebDAV-based Storage.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL