Documentation
¶
Index ¶
- func Download(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)
- func FilePathWalkDir(root string) ([]string, error)
- func Upload(ctx context.Context, store Storage, transfers []Transfer, parallelLimit int)
- type AmazonS3
- func (s3b *AmazonS3) Get(ctx context.Context, url, path string) (*Object, error)
- func (s3b *AmazonS3) Join(url, path string) (string, error)
- func (s3b *AmazonS3) List(ctx context.Context, url string) ([]*Object, error)
- func (s3b *AmazonS3) Put(ctx context.Context, url, path string) (*Object, error)
- func (s3b *AmazonS3) Stat(ctx context.Context, url string) (*Object, error)
- func (s3b *AmazonS3) UnsupportedOperations(url string) UnsupportedOperations
- type ErrInvalidURL
- type ErrUnsupportedProtocol
- type FTP
- func (b *FTP) Get(ctx context.Context, url, path string) (*Object, error)
- func (b *FTP) Join(url, path string) (string, error)
- func (b *FTP) List(ctx context.Context, url string) ([]*Object, error)
- func (b *FTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)
- func (b *FTP) Stat(ctx context.Context, url string) (*Object, error)
- func (b *FTP) UnsupportedOperations(url string) UnsupportedOperations
- type Fake
- func (f Fake) Get(ctx context.Context, url, path string) (*Object, error)
- func (f Fake) Join(url, path string) (string, error)
- func (f Fake) List(ctx context.Context, url string) ([]*Object, error)
- func (f Fake) Put(ctx context.Context, url, path string) (*Object, error)
- func (f Fake) Stat(ctx context.Context, url string) (*Object, error)
- func (f Fake) UnsupportedOperations(url string) UnsupportedOperations
- type GenericS3
- func (s3 *GenericS3) Get(ctx context.Context, url, path string) (*Object, error)
- func (s3 *GenericS3) Join(url, path string) (string, error)
- func (s3 *GenericS3) List(ctx context.Context, url string) ([]*Object, error)
- func (s3 *GenericS3) Put(ctx context.Context, url, path string) (*Object, error)
- func (s3 *GenericS3) Stat(ctx context.Context, url string) (*Object, error)
- func (s3 *GenericS3) UnsupportedOperations(url string) UnsupportedOperations
- type GoogleCloud
- func (gs *GoogleCloud) Get(ctx context.Context, url, path string) (*Object, error)
- func (gs *GoogleCloud) Join(url, path string) (string, error)
- func (gs *GoogleCloud) List(ctx context.Context, url string) ([]*Object, error)
- func (gs *GoogleCloud) Put(ctx context.Context, url, path string) (*Object, error)
- func (gs *GoogleCloud) Stat(ctx context.Context, url string) (*Object, error)
- func (gs *GoogleCloud) UnsupportedOperations(url string) UnsupportedOperations
- type HTTP
- func (b *HTTP) Get(ctx context.Context, url, path string) (*Object, error)
- func (b *HTTP) Join(url, path string) (string, error)
- func (b *HTTP) List(ctx context.Context, url string) ([]*Object, error)
- func (b *HTTP) Put(ctx context.Context, url string, hostPath string) (*Object, error)
- func (b *HTTP) Stat(ctx context.Context, url string) (*Object, error)
- func (b *HTTP) UnsupportedOperations(url string) UnsupportedOperations
- type Local
- func (local *Local) Get(ctx context.Context, url, path string) (*Object, error)
- func (local *Local) Join(url, path string) (string, error)
- func (local *Local) List(ctx context.Context, url string) ([]*Object, error)
- func (local *Local) Put(ctx context.Context, url, path string) (*Object, error)
- func (local *Local) Stat(ctx context.Context, url string) (*Object, error)
- func (local *Local) UnsupportedOperations(url string) UnsupportedOperations
- type Mux
- func (mux *Mux) AttachLogger(log *logger.Logger)
- func (mux *Mux) Get(ctx context.Context, url, path string) (*Object, error)
- func (mux *Mux) Join(url, path string) (string, error)
- func (mux *Mux) List(ctx context.Context, url string) ([]*Object, error)
- func (mux *Mux) Put(ctx context.Context, url, path string) (*Object, error)
- func (mux *Mux) Stat(ctx context.Context, url string) (*Object, error)
- func (mux *Mux) UnsupportedOperations(url string) UnsupportedOperations
- type Object
- type Retrier
- func (r *Retrier) Get(ctx context.Context, url, path string) (obj *Object, err error)
- func (r *Retrier) Join(url, path string) (string, error)
- func (r *Retrier) List(ctx context.Context, url string) (objects []*Object, err error)
- func (r *Retrier) Put(ctx context.Context, url, path string) (obj *Object, err error)
- func (r *Retrier) Stat(ctx context.Context, url string) (obj *Object, err error)
- func (r *Retrier) UnsupportedOperations(url string) UnsupportedOperations
- type Storage
- type Swift
- func (sw *Swift) Get(ctx context.Context, url, path string) (*Object, error)
- func (sw *Swift) Join(url, path string) (string, error)
- func (sw *Swift) List(ctx context.Context, url string) ([]*Object, error)
- func (sw *Swift) Put(ctx context.Context, url, path string) (*Object, error)
- func (sw *Swift) Stat(ctx context.Context, url string) (*Object, error)
- func (sw *Swift) UnsupportedOperations(url string) UnsupportedOperations
- type Transfer
- type UnsupportedOperations
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func Download ¶
Download downloads a list of transfers from storage, in parallel.
Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.
func FilePathWalkDir ¶
Types ¶
type AmazonS3 ¶
type AmazonS3 struct {
// contains filtered or unexported fields
}
AmazonS3 provides access to an S3 object store.
func NewAmazonS3 ¶
func NewAmazonS3(conf config.AmazonS3Storage) (*AmazonS3, error)
NewAmazonS3 creates an AmazonS3 session instance
func (*AmazonS3) UnsupportedOperations ¶
func (s3b *AmazonS3) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type ErrInvalidURL ¶
type ErrInvalidURL struct {
// contains filtered or unexported fields
}
ErrInvalidURL is returned by SupportsGet / SupportsPut when a url's format is invalid.
func (*ErrInvalidURL) Error ¶
func (e *ErrInvalidURL) Error() string
type ErrUnsupportedProtocol ¶
type ErrUnsupportedProtocol struct {
// contains filtered or unexported fields
}
ErrUnsupportedProtocol is returned by SupportsGet / SupportsPut when a url's protocol is unsupported by that backend
func (*ErrUnsupportedProtocol) Error ¶
func (e *ErrUnsupportedProtocol) Error() string
type FTP ¶
type FTP struct {
// contains filtered or unexported fields
}
FTP provides read access to public URLs.
func (*FTP) UnsupportedOperations ¶
func (b *FTP) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Fake ¶
type Fake struct{}
Fake implements a the Storage interface with methods that do nothing and return nil. This is a testing utility.
func (Fake) Put ¶
Put a single object to storage URL, from a local file path. Returns the Object that was created in storage.
func (Fake) UnsupportedOperations ¶
func (f Fake) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations are supported by the storage system for the given URL.
type GenericS3 ¶
type GenericS3 struct {
// contains filtered or unexported fields
}
GenericS3 provides access to an S3 object store.
func NewGenericS3 ¶
func NewGenericS3(conf config.GenericS3Storage) (*GenericS3, error)
NewGenericS3 creates a new GenericS3 instance, given an endpoint URL and a set of authentication credentials.
func (*GenericS3) Put ¶
Put copies an object (file) from the host path to S3. Update Put function to be able to upload directories (a la Get() function)
func (*GenericS3) UnsupportedOperations ¶
func (s3 *GenericS3) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type GoogleCloud ¶
type GoogleCloud struct {
// contains filtered or unexported fields
}
GoogleCloud provides access to an GS object store.
func NewGoogleCloud ¶
func NewGoogleCloud(conf config.GoogleCloudStorage) (*GoogleCloud, error)
NewGoogleCloud creates an GoogleCloud client instance, give an endpoint URL and a set of authentication credentials.
func (*GoogleCloud) Join ¶
func (gs *GoogleCloud) Join(url, path string) (string, error)
Join joins the given URL with the given subpath.
func (*GoogleCloud) UnsupportedOperations ¶
func (gs *GoogleCloud) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type HTTP ¶
type HTTP struct {
// contains filtered or unexported fields
}
HTTP provides read access to public URLs.
func NewHTTP ¶
func NewHTTP(conf config.HTTPStorage) (*HTTP, error)
NewHTTP creates a new HTTP instance.
func (*HTTP) UnsupportedOperations ¶
func (b *HTTP) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Local ¶
type Local struct {
// contains filtered or unexported fields
}
Local provides access to a local-disk storage system.
func NewLocal ¶
func NewLocal(conf config.LocalStorage) (*Local, error)
NewLocal returns a Local instance, configured to limit file system access to the given allowed directories.
func (*Local) UnsupportedOperations ¶
func (local *Local) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Mux ¶
type Mux struct {
Backends []Storage
}
Mux provides a client for accessing multiple storage systems, i.e. for downloading/uploading task files from S3, GS, local disk, etc.
For a given storage url, the storage backend is usually determined by the url prefix, e.g. "s3://my-bucket/file" will access the S3 backend.
func (*Mux) AttachLogger ¶
AttachLogger will log information (such as retry warnings) to the given logger.
func (*Mux) Get ¶
Get downloads a file from a storage system at the given "url". The file is downloaded to the given local "path".
func (*Mux) Put ¶
Put uploads a file to a storage system at the given "url". The file is uploaded from the given local "path".
func (*Mux) UnsupportedOperations ¶
func (mux *Mux) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Object ¶
type Object struct { // The storage-specific full URL of the object. // e.g. for S3 this might be "s3://my-bucket/dir1/obj.txt" URL string // The name of the object in the storage system. // e.g. for S3 this might be "dir/object.txt" Name string // ETag is an identifier for a specific version of the object. // This is an opaque string. Each system has a different representation: // md5, sha1, crc32, etc. This field may be empty, if the system can't provide // a unique ID (for example the local filesystem). ETag string LastModified time.Time // Size of the object, in bytes. Size int64 }
Object represents metadata about an object in storage.
type Retrier ¶
Retrier wraps a storage backend with logic which will retry on error, with a configurable backoff strategy.
func NewSwiftRetrier ¶
func NewSwiftRetrier(conf config.SwiftStorage) (*Retrier, error)
NewSwiftRetrier returns a Swift storage client that retries operations on error.
func (*Retrier) UnsupportedOperations ¶
func (r *Retrier) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Storage ¶
type Storage interface { // Stat returns information about the object at the given storage URL. Stat(ctx context.Context, url string) (*Object, error) // List a directory. Calling List on a File is an error. List(ctx context.Context, url string) ([]*Object, error) // Get a single object from storage URL, written to a local file path. Get(ctx context.Context, url, path string) (*Object, error) // Put a single object to storage URL, from a local file path. // Returns the Object that was created in storage. Put(ctx context.Context, url, path string) (*Object, error) // Join a directory URL with a subpath. Join(url, path string) (string, error) // UnsupportedOperations describes which operations are supported by the storage system // for the given URL. // // A backend normally uses this to match the url prefix (e.g. "s3://"), // and some backends (generic s3) might check for the existence of the bucket. UnsupportedOperations(url string) UnsupportedOperations }
Storage provides an interface for a storage backend, providing access to concrete storage systems such as Google Storage, local filesystem, etc.
Storage backends must be safe for concurrent use.
type Swift ¶
type Swift struct {
// contains filtered or unexported fields
}
Swift provides access to an sw object store.
func NewSwift ¶
func NewSwift(conf config.SwiftStorage) (*Swift, error)
NewSwift creates an Swift client instance, give an endpoint URL and a set of authentication credentials.
func (*Swift) UnsupportedOperations ¶
func (sw *Swift) UnsupportedOperations(url string) UnsupportedOperations
UnsupportedOperations describes which operations (Get, Put, etc) are not supported for the given URL.
type Transfer ¶
type Transfer interface { URL() string Path() string Started() Finished(obj *Object) Failed(err error) }
Transfer defines the interface of a single storage upload or download request.
Transfer events (started, failed, finished, etc) are communicated via the Transfer interface.
type UnsupportedOperations ¶
type UnsupportedOperations struct {
Get, Put, List, Stat, Join error
}
UnsupportedOperations describes any operations that are not supported by a storage backend.
func AllSupported ¶
func AllSupported() UnsupportedOperations
AllSupported returns an UnsupportedOperations indicating that all operations are supported.
func AllUnsupported ¶
func AllUnsupported(err error) UnsupportedOperations
AllUnsupported returns an UnsupportedOperations indicating that no operations are supported.