backends

package
v0.0.0-...-0ac389e Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Feb 13, 2023 License: MIT Imports: 40 Imported by: 0

Documentation

Overview

Package backends implements the Backend interface for various targets.

Index

Constants

View Source 
const AWSS3BackendPrefix = "s3"

AWSS3BackendPrefix is the URI prefix used for the AWSS3Backend.

View Source 
const (
	AzureBackendPrefix = "azure"
)

AzureBackendPrefix is the URI prefix used for the AzureBackend.

View Source 
const B2BackendPrefix = "b2"

B2BackendPrefix is the URI prefix used for the B2Backend.

View Source 
const DeleteBackendPrefix = "delete"

DeleteBackendPrefix is the URI prefix used for the DeleteBackend.

View Source 
const FileBackendPrefix = "file"

FileBackendPrefix is the URI prefix used for the FileBackend.

View Source 
const GoogleCloudStorageBackendPrefix = "gs"

GoogleCloudStorageBackendPrefix is the URI prefix used for the GoogleCloudStorageBackend.

View Source 
const SSHBackendPrefix = "ssh"

SSHBackendPrefix is the URI prefix used for the SSHBackend.

Variables

View Source 
var (
	// ErrInvalidURI is returned when a backend determines that the provided URI is malformed/invalid.
	ErrInvalidURI = errors.New("backends: invalid URI provided to backend")
	// ErrInvalidPrefix is returned when a backend destination is provided with a URI prefix that isn't registered.
	ErrInvalidPrefix = errors.New("backends: the provided prefix does not exist")
)

Functions

This section is empty.

Types

type AWSS3Backend 

type AWSS3Backend struct {
	// contains filtered or unexported fields
}

AWSS3Backend integrates with Amazon Web Services' S3.

func (*AWSS3Backend) Close 

func (a *AWSS3Backend) Close() error

Close will release any resources used by the AWS S3 backend.

func (*AWSS3Backend) Delete 

func (a *AWSS3Backend) Delete(ctx context.Context, key string) error

Delete will delete the given object from the configured bucket

func (*AWSS3Backend) Download 

func (a *AWSS3Backend) Download(ctx context.Context, key string) (io.ReadCloser, error)

Download will download the requseted object which can be read from the returned io.ReadCloser

func (*AWSS3Backend) Init 

func (a *AWSS3Backend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the AWSS3Backend and verify the provided URI is valid/exists.

func (*AWSS3Backend) List 

func (a *AWSS3Backend) List(ctx context.Context, prefix string) ([]string, error)

List will iterate through all objects in the configured AWS S3 bucket and return a list of keys, filtering by the provided prefix.

func (*AWSS3Backend) PreDownload 

func (a *AWSS3Backend) PreDownload(ctx context.Context, keys []string) error

PreDownload will restore objects from Glacier as required.

func (*AWSS3Backend) Upload 

func (a *AWSS3Backend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will upload the provided volume to this AWSS3Backend's configured bucket+prefix It utilizes multipart uploads to upload a single file in chunks concurrently. Impartial uploads are cleaned up by the s3manager provided by the AWS Go SDK, but users can also implement lifecycle rules, see: https://docs.aws.amazon.com/AmazonS3/latest/dev/mpuoverview.html#mpu-abort-incomplete-mpu-lifecycle-config

type AzureBackend 

type AzureBackend struct {
	// contains filtered or unexported fields
}

AzureBackend integrates with Microsoft's Azure Storage Services.

func (*AzureBackend) Close 

func (a *AzureBackend) Close() error

Close will release any resources used by the Azure backend.

func (*AzureBackend) Delete 

func (a *AzureBackend) Delete(ctx context.Context, name string) error

Delete will delete the given object from the configured container

func (*AzureBackend) Download 

func (a *AzureBackend) Download(ctx context.Context, name string) (io.ReadCloser, error)

Download will download the requseted object which can be read from the returned io.ReadCloser

func (*AzureBackend) Init 

func (a *AzureBackend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the AzureBackend and verify the provided URI is valid/exists.

func (*AzureBackend) List 

func (a *AzureBackend) List(ctx context.Context, prefix string) ([]string, error)

List will iterate through all objects in the configured Azure Storage Container and return a list of blob names, filtering by the provided prefix.

func (*AzureBackend) PreDownload 

func (a *AzureBackend) PreDownload(ctx context.Context, keys []string) error

PreDownload will do nothing for this backend.

func (*AzureBackend) Upload 

func (a *AzureBackend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will upload the provided volume to this AzureBackend's configured container+prefix It utilizes Azure Block Blob uploads to upload chunks of a single file concurrently. Partial uploads are automatically garbage collected after one week. See: https://docs.microsoft.com/en-us/rest/api/storageservices/put-block-list

type B2Backend 

type B2Backend struct {
	// contains filtered or unexported fields
}

B2Backend integrates with BackBlaze's B2 storage service.

func (*B2Backend) Close 

func (b *B2Backend) Close() error

Close will release any resources used by the B2 backend.

func (*B2Backend) Delete 

func (b *B2Backend) Delete(ctx context.Context, name string) error

Delete will delete the object with the given name from the configured bucket

func (*B2Backend) Download 

func (b *B2Backend) Download(ctx context.Context, name string) (io.ReadCloser, error)

Download will download the requseted object which can be read from the returned io.ReadCloser

func (*B2Backend) Init 

func (b *B2Backend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the B2Backend and verify the provided URI is valid/exists.

func (*B2Backend) List 

func (b *B2Backend) List(ctx context.Context, prefix string) ([]string, error)

List will iterate through all objects in the configured B2 bucket and return a list of object names, filtering by the provided prefix.

func (*B2Backend) PreDownload 

func (b *B2Backend) PreDownload(ctx context.Context, keys []string) error

PreDownload will do nothing for this backend.

func (*B2Backend) Upload 

func (b *B2Backend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will upload the provided volume to this B2Backend's configured bucket+prefix It utilizes B2 Large File Upload API to upload chunks of a single file concurrently. Partial uploads need to be manually cleaned, which the b2 Go client should take care of for us.

type Backend 

type Backend interface {
	Init(ctx context.Context, conf *BackendConfig, opts ...Option) error  // Verifies settings required for backend are present and valid, does basic initialization of backend
	Upload(ctx context.Context, vol *files.VolumeInfo) error              // Upload the volume provided
	List(ctx context.Context, prefix string) ([]string, error)            // Lists all files in the backend, filtering by the provided prefix.
	Close() error                                                         // Release any resources in use
	PreDownload(ctx context.Context, objects []string) error              // PreDownload will prepare the provided files for download (think restoring from Glacier to S3)
	Download(ctx context.Context, filename string) (io.ReadCloser, error) // Download the requested file that can be read from the returned io.ReaderCloser
	Delete(ctx context.Context, filename string) error                    // Delete the file specified on the configured backend
}

Backend is an interface type that defines the functions and functionality required for different backend implementations. nolint:lll // It's neater this way

func GetBackendForURI 

func GetBackendForURI(uri string) (Backend, error)

GetBackendForURI will try and parse the URI for a matching backend to use.

type BackendConfig 

type BackendConfig struct {
	MaxParallelUploadBuffer chan bool
	MaxParallelUploads      int
	MaxBackoffTime          time.Duration
	MaxRetryTime            time.Duration
	TargetURI               string
	UploadChunkSize         int
}

BackendConfig holds values that relate to backend configurations

type DeleteBackend 

type DeleteBackend struct {
	// contains filtered or unexported fields
}

DeleteBackend is a special Backend used to delete the files after they've been uploaded

func (*DeleteBackend) Close 

func (d *DeleteBackend) Close() error

Close does nothing for this backend.

func (*DeleteBackend) Delete 

func (d *DeleteBackend) Delete(ctx context.Context, filename string) error

Delete should not be used with this backend

func (*DeleteBackend) Download 

func (d *DeleteBackend) Download(ctx context.Context, filename string) (io.ReadCloser, error)

Download should not be used with this backend

func (*DeleteBackend) Init 

func (d *DeleteBackend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the DeleteBackend (aka do nothing)

func (*DeleteBackend) List 

func (d *DeleteBackend) List(ctx context.Context, prefix string) ([]string, error)

List should not be used with this backend

func (*DeleteBackend) PreDownload 

func (d *DeleteBackend) PreDownload(ctx context.Context, objects []string) error

PreDownload should not be used with this backend

func (*DeleteBackend) Upload 

func (d *DeleteBackend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will delete the provided volume, usually found in a temporary folder

type FileBackend 

type FileBackend struct {
	// contains filtered or unexported fields
}

FileBackend provides a local destination storage option.

func (*FileBackend) Close 

func (f *FileBackend) Close() error

Close does nothing for this backend.

func (*FileBackend) Delete 

func (f *FileBackend) Delete(ctx context.Context, filename string) error

Delete will delete the given object from the provided path

func (*FileBackend) Download 

func (f *FileBackend) Download(ctx context.Context, filename string) (io.ReadCloser, error)

Download will open the file for reading

func (*FileBackend) Init 

func (f *FileBackend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the FileBackend and verify the provided URI is valid/exists.

func (*FileBackend) List 

func (f *FileBackend) List(ctx context.Context, prefix string) ([]string, error)

List will return a list of all files matching the provided prefix

func (*FileBackend) PreDownload 

func (f *FileBackend) PreDownload(ctx context.Context, objects []string) error

PreDownload does nothing on this backend.

func (*FileBackend) Upload 

func (f *FileBackend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will copy the provided VolumeInfo to the backend's configured local destination

type GoogleCloudStorageBackend 

type GoogleCloudStorageBackend struct {
	// contains filtered or unexported fields
}

GoogleCloudStorageBackend integrates with Google Cloud Storage.

func (*GoogleCloudStorageBackend) Close 

func (g *GoogleCloudStorageBackend) Close() error

Close will release any resources used by the GCS backend.

func (*GoogleCloudStorageBackend) Delete 

func (g *GoogleCloudStorageBackend) Delete(ctx context.Context, filename string) error

Delete will delete the given object from the configured bucket

func (*GoogleCloudStorageBackend) Download 

func (g *GoogleCloudStorageBackend) Download(ctx context.Context, filename string) (io.ReadCloser, error)

Download will download the requseted object which can be read from the return io.ReadCloser.

func (*GoogleCloudStorageBackend) Init 

func (g *GoogleCloudStorageBackend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) error

Init will initialize the GoogleCloudStorageBackend and verify the provided URI is valid/exists.

func (*GoogleCloudStorageBackend) List 

func (g *GoogleCloudStorageBackend) List(ctx context.Context, prefix string) ([]string, error)

List will iterate through all objects in the configured GCS bucket and return a list of object names, filtering by the prefix provided.

func (*GoogleCloudStorageBackend) PreDownload 

func (g *GoogleCloudStorageBackend) PreDownload(ctx context.Context, objects []string) error

PreDownload does nothing on this backend.

func (*GoogleCloudStorageBackend) Upload 

func (g *GoogleCloudStorageBackend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will upload the provided VolumeInfo to Google's Cloud Storage It utilizes Resumeable Uploads to upload one file at a time serially. Incomplete uploads auto-expire after one week. See: https://cloud.google.com/storage/docs/json_api/v1/how-tos/resumable-upload

type Option 

type Option interface {
	Apply(Backend)
}

Option lets users inject functionality to specific backends

func WithGoogleCloudStorageClient 

func WithGoogleCloudStorageClient(c *storage.Client) Option

WithGoogleCloudStorageClient will override an GCS backend's underlying API client with the one provided.

func WithS3Client 

func WithS3Client(c s3iface.S3API) Option

WithS3Client will override an S3 backend's underlying API client with the one provided. Primarily used to inject mock clients for testing.

func WithS3Uploader 

func WithS3Uploader(c s3manageriface.UploaderAPI) Option

WithS3Uploader will override an S3 backend's underlying uploader client with the one provided. Primarily used to inject mock clients for testing.

type SSHBackend 

type SSHBackend struct {
	// contains filtered or unexported fields
}

SSHBackend provides a ssh/sftp storage option.

func (*SSHBackend) Close 

func (s *SSHBackend) Close() (err error)

Close will release any resources used by SSHBackend.

func (*SSHBackend) Delete 

func (s *SSHBackend) Delete(ctx context.Context, filename string) error

Delete will delete the given object from the provided path.

func (*SSHBackend) Download 

func (s *SSHBackend) Download(ctx context.Context, filename string) (io.ReadCloser, error)

Download will open the remote file for reading.

func (*SSHBackend) Init 

func (s *SSHBackend) Init(ctx context.Context, conf *BackendConfig, opts ...Option) (err error)

Init will initialize the SSHBackend and verify the provided URI is valid and the target directory exists.

func (*SSHBackend) List 

func (s *SSHBackend) List(ctx context.Context, prefix string) ([]string, error)

List will return a list of all files matching the provided prefix.

func (*SSHBackend) PreDownload 

func (s *SSHBackend) PreDownload(ctx context.Context, objects []string) error

PreDownload does nothing on this backend.

func (*SSHBackend) Upload 

func (s *SSHBackend) Upload(ctx context.Context, vol *files.VolumeInfo) error

Upload will upload the provided VolumeInfo to the remote sftp server.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.