storage

package module
v0.16.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: May 31, 2024 License: Apache-2.0 Imports: 39 Imported by: 17

README

chartmuseum/storage

GitHub Actions status Go Report Card GoDoc

Go library providing a common interface for working across multiple storage backends.

Supported storage backends:

This code was originally part of the Helm project: ChartMuseum, but has since been released as a standalone package for others to use in their own projects.

Primary Components

Backend (interface)

Backend is a common interface that is implemented by all the supported storage backends and their associated types:

type Backend interface {
    ListObjects(prefix string) ([]Object, error)
    GetObject(path string) (Object, error)
    PutObject(path string, content []byte) error
    DeleteObject(path string) error
}
Object (struct)

Object is a struct that represents a single storage object:

type Object struct {
    Path         string
    Content      []byte
    LastModified time.Time
}
ObjectSliceDiff (struct)

ObjectSliceDiff is a struct that represents overall changes between two Object slices:

type ObjectSliceDiff struct {
    Change  bool
    Removed []Object
    Added   []Object
    Updated []Object
}
GetObjectSliceDiff (function)

GetObjectSliceDiff is a function that takes two Object slices, compares them, and returns an ObjectSliceDiff:

func GetObjectSliceDiff(prev []Object, curr []Object, timestampTolerance time.Duration) ObjectSliceDiff

Usage

Simple example

The following is a simple program that will upload a file either to an Azure Blob Storage bucket (container) or a Google Cloud Storage bucket based on the command line options provided:

// Usage: go run example.go <cloud> <bucket> <file>

package main

import (
	"fmt"
	"io/ioutil"
	"os"
	"path/filepath"

	"github.com/chartmuseum/storage"
)

type (
	Uploader struct {
		Backend storage.Backend
	}
)

func NewUploader(cloud string, bucket string) *Uploader {
	var backend storage.Backend
	switch cloud {
	case "azure":
		backend = storage.NewMicrosoftBlobBackend(bucket, "")
	case "google":
		backend = storage.NewGoogleCSBackend(bucket, "")
	default:
		panic("cloud provider " + cloud + " not supported")
	}
	uploader := Uploader{Backend: backend}
	fmt.Printf("uploader created (cloud: %s, bucket: %s)\n", cloud, bucket)
	return &uploader
}

func (uploader *Uploader) Upload(filename string) {
	basename := filepath.Base(filename)
	content, err := ioutil.ReadFile(filename)
	if err != nil {
		panic(err)
	}
	err = uploader.Backend.PutObject(basename, content)
	if err != nil {
		panic(err)
	}
	fmt.Printf("%s successfully uploaded\n", basename)
}

func main() {
	args := os.Args[1:]
	uploader := NewUploader(args[0], args[1])
	uploader.Upload(args[2])
}

Example of using to upload the file index.html to an Azure bucket:

go run example.go azure mycontainer index.html

Example of using to upload the file index.html to a Google Cloud bucket:

go run example.go google mybucket index.html
Per backend

Each supported storage backend has its own type that implements the Backend interface. All available types are described in detail on GoDoc.

In addition, authentication methods are based on the runtime environment and vary from cloud to cloud.

Documentation

Index

Constants

View Source 
const DefaultPrefix = "/chart_backend_bucket"
View Source 
const (
	HTTPHeaderLastModified = "Last-Modified"
)

Variables

View Source 
var (
	DefileDialTimeOut    = "5s"
	TimeStampKey         = "timestamp"
	ErrNotExistEndpoints = fmt.Errorf("endpoints cannot connect !")
	ErrNotExist          = fmt.Errorf("not exist!")
)

Functions

This section is empty.

Types

type AlibabaCloudOSSBackend 

type AlibabaCloudOSSBackend struct {
	Bucket *oss.Bucket
	Client *oss.Client
	Prefix string
	SSE    string
}

AlibabaCloudOSSBackend is a storage backend for Alibaba Cloud OSS

func NewAlibabaCloudOSSBackend 

func NewAlibabaCloudOSSBackend(bucket string, prefix string, endpoint string, sse string) *AlibabaCloudOSSBackend

NewAlibabaCloudOSSBackend creates a new instance of AlibabaCloudOSSBackend

func (AlibabaCloudOSSBackend) DeleteObject 

func (b AlibabaCloudOSSBackend) DeleteObject(path string) error

DeleteObject removes an object from Alibaba Cloud OSS bucket, at prefix

func (AlibabaCloudOSSBackend) GetObject 

func (b AlibabaCloudOSSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from Alibaba Cloud OSS bucket, at prefix

func (AlibabaCloudOSSBackend) ListObjects 

func (b AlibabaCloudOSSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Alibaba Cloud OSS bucket, at prefix

func (AlibabaCloudOSSBackend) PutObject 

func (b AlibabaCloudOSSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Alibaba Cloud OSS bucket, at prefix

type AmazonS3Backend 

type AmazonS3Backend struct {
	Bucket     string
	Client     *s3.S3
	Downloader *s3manager.Downloader
	Prefix     string
	Uploader   *s3manager.Uploader
	SSE        string
}

AmazonS3Backend is a storage backend for Amazon S3

func NewAmazonS3Backend 

func NewAmazonS3Backend(bucket string, prefix string, region string, endpoint string, sse string) *AmazonS3Backend

NewAmazonS3Backend creates a new instance of AmazonS3Backend

func NewAmazonS3BackendWithCredentials added in v0.6.0 

func NewAmazonS3BackendWithCredentials(bucket string, prefix string, region string, endpoint string, sse string, credentials *credentials.Credentials) *AmazonS3Backend

NewAmazonS3BackendWithCredentials creates a new instance of AmazonS3Backend with credentials

func NewAmazonS3BackendWithOptions added in v0.14.1 

func NewAmazonS3BackendWithOptions(bucket string, prefix string, region string, endpoint string, sse string, options *AmazonS3Options) *AmazonS3Backend

func (AmazonS3Backend) DeleteObject 

func (b AmazonS3Backend) DeleteObject(path string) error

DeleteObject removes an object from Amazon S3 bucket, at prefix

func (AmazonS3Backend) GetObject 

func (b AmazonS3Backend) GetObject(path string) (Object, error)

GetObject retrieves an object from Amazon S3 bucket, at prefix

func (AmazonS3Backend) ListObjects 

func (b AmazonS3Backend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Amazon S3 bucket, at prefix

func (AmazonS3Backend) PutObject 

func (b AmazonS3Backend) PutObject(path string, content []byte) error

PutObject uploads an object to Amazon S3 bucket, at prefix

type AmazonS3Options added in v0.14.1 

type AmazonS3Options struct {
	S3ForcePathStyle *bool
}

type Backend 

type Backend interface {
	ListObjects(prefix string) ([]Object, error)
	GetObject(path string) (Object, error)
	PutObject(path string, content []byte) error
	DeleteObject(path string) error
}

Backend is a generic interface for storage backends

func NewEtcdCSBackend added in v0.4.0 

func NewEtcdCSBackend(endpoints string, cafile, certfile, keyfile string, prefix string) Backend

type BaiduBOSBackend added in v0.3.0 

type BaiduBOSBackend struct {
	Client *bos.Client
	Bucket string
	Prefix string
}

BaiduBOSBackend is a storage backend for Baidu Cloud BOS

func NewBaiDuBOSBackend added in v0.3.0 

func NewBaiDuBOSBackend(bucket string, prefix string, endpoint string) *BaiduBOSBackend

NewBaiduBOSBackend creates a new instance of BaiduBOSBackend

func (BaiduBOSBackend) DeleteObject added in v0.3.0 

func (b BaiduBOSBackend) DeleteObject(path string) error

DeleteObject removes an object from Baidu Cloud BOS bucket, at prefix

func (BaiduBOSBackend) GetObject added in v0.3.0 

func (b BaiduBOSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from Baidu Cloud BOS bucket, at prefix

func (BaiduBOSBackend) ListObjects added in v0.3.0 

func (b BaiduBOSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Baidu Cloud BOS bucket, at prefix

func (BaiduBOSBackend) PutObject added in v0.3.0 

func (b BaiduBOSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Baidu Cloud BOS bucket, at prefix

type GoogleCSBackend 

type GoogleCSBackend struct {
	Prefix  string
	Client  *storage.BucketHandle
	Context context.Context
}

GoogleCSBackend is a storage backend for Google Cloud Storage

func NewGoogleCSBackend 

func NewGoogleCSBackend(bucket string, prefix string) *GoogleCSBackend

NewGoogleCSBackend creates a new instance of GoogleCSBackend

func (GoogleCSBackend) DeleteObject 

func (b GoogleCSBackend) DeleteObject(path string) error

DeleteObject removes an object from Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) GetObject 

func (b GoogleCSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) ListObjects 

func (b GoogleCSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Google Cloud Storage bucket, at prefix

func (GoogleCSBackend) PutObject 

func (b GoogleCSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Google Cloud Storage bucket, at prefix

type LocalFilesystemBackend 

type LocalFilesystemBackend struct {
	RootDirectory string
}

LocalFilesystemBackend is a storage backend for local filesystem storage

func NewLocalFilesystemBackend 

func NewLocalFilesystemBackend(rootDirectory string) *LocalFilesystemBackend

NewLocalFilesystemBackend creates a new instance of LocalFilesystemBackend

func (LocalFilesystemBackend) DeleteObject 

func (b LocalFilesystemBackend) DeleteObject(path string) error

DeleteObject removes an object from root directory

func (LocalFilesystemBackend) GetObject 

func (b LocalFilesystemBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from root directory

func (LocalFilesystemBackend) ListObjects 

func (b LocalFilesystemBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in root directory (depth 1)

func (LocalFilesystemBackend) PutObject 

func (b LocalFilesystemBackend) PutObject(path string, content []byte) error

PutObject puts an object in root directory

type Metadata added in v0.10.0 

type Metadata struct {
	Name    string
	Version string
}

Metadata represents the meta information of the object includes object name , object version , etc...

type MicrosoftBlobBackend 

type MicrosoftBlobBackend struct {
	Prefix    string
	Container *microsoft_storage.Container
}

MicrosoftBlobBackend is a storage backend for Microsoft Azure Blob Storage

func NewMicrosoftBlobBackend 

func NewMicrosoftBlobBackend(container string, prefix string) *MicrosoftBlobBackend

NewMicrosoftBlobBackend creates a new instance of MicrosoftBlobBackend

func (MicrosoftBlobBackend) DeleteObject 

func (b MicrosoftBlobBackend) DeleteObject(path string) error

DeleteObject removes an object from Microsoft Azure Blob Storage container, at path

func (MicrosoftBlobBackend) GetObject 

func (b MicrosoftBlobBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from Microsoft Azure Blob Storage, at path

func (MicrosoftBlobBackend) ListObjects 

func (b MicrosoftBlobBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Microsoft Azure Blob Storage container

func (MicrosoftBlobBackend) PutObject 

func (b MicrosoftBlobBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Microsoft Azure Blob Storage container, at path

type Object 

type Object struct {
	Meta         Metadata
	Path         string
	Content      []byte
	LastModified time.Time
}

Object is a generic representation of a storage object

func (Object) HasExtension 

func (object Object) HasExtension(extension string) bool

HasExtension determines whether or not an object contains a file extension

type ObjectSliceDiff 

type ObjectSliceDiff struct {
	Change  bool
	Removed []Object
	Added   []Object
	Updated []Object
}

ObjectSliceDiff provides information on what has changed since last calling ListObjects

func GetObjectSliceDiff 

func GetObjectSliceDiff(prev []Object, curr []Object, timestampTolerance time.Duration) ObjectSliceDiff

GetObjectSliceDiff takes two objects slices and returns an ObjectSliceDiff

type OpenstackOSBackend 

type OpenstackOSBackend struct {
	Container string
	Prefix    string
	Region    string
	CACert    string
	Client    *gophercloud.ServiceClient
}

OpenstackOSBackend is a storage backend for Openstack Object Storage

func NewOpenstackOSBackend 

func NewOpenstackOSBackend(container string, prefix string, region string, caCert string) *OpenstackOSBackend

NewOpenstackOSBackend creates a new instance of OpenstackOSBackend

func NewOpenstackOSBackendV1Auth added in v0.9.0 

func NewOpenstackOSBackendV1Auth(container string, prefix string, caCert string) *OpenstackOSBackend

NewOpenstackOSBackendV1Auth creates a new instance of OpenstackOSBackend using Swift V1 Auth

func (OpenstackOSBackend) DeleteObject 

func (b OpenstackOSBackend) DeleteObject(path string) error

DeleteObject removes an object from an Openstack container, at prefix

func (OpenstackOSBackend) GetObject 

func (b OpenstackOSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from an Openstack container, at prefix

func (OpenstackOSBackend) ListObjects 

func (b OpenstackOSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in an Openstack container, at prefix

func (OpenstackOSBackend) PutObject 

func (b OpenstackOSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Openstack container, at prefix

type OracleCSBackend 

type OracleCSBackend struct {
	Bucket        string
	Prefix        string
	Namespace     string
	CompartmentId string
	Client        objectstorage.ObjectStorageClient
	Context       context.Context
}

OracleCSBackend is a storage backend for Oracle Cloud Infrastructure Object Storage

func NewOracleCSBackend 

func NewOracleCSBackend(bucket string, prefix string, region string, compartmentId string) *OracleCSBackend

NewOracleCSBackend creates a new instance of OracleCSBackend

func (OracleCSBackend) DeleteObject 

func (b OracleCSBackend) DeleteObject(path string) error

DeleteObject removes an object from OCI Object Storage bucket, at prefix

func (OracleCSBackend) GetObject 

func (b OracleCSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from OCI Object Storage bucket, at prefix

func (OracleCSBackend) ListObjects 

func (b OracleCSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in OCI Object Storage bucket, at prefix

func (OracleCSBackend) PutObject 

func (b OracleCSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to OCI Object Storage bucket, at prefix

type ReauthRoundTripper 

type ReauthRoundTripper struct {
	// contains filtered or unexported fields
}

ReauthRoundTripper satisfies the http.RoundTripper interface and is used to limit the number of consecutive re-auth attempts (infinite by default)

func (*ReauthRoundTripper) RoundTrip 

func (rrt *ReauthRoundTripper) RoundTrip(request *http.Request) (*http.Response, error)

RoundTrip performs a round-trip HTTP request and logs relevant information about it.

type TencentCloudCOSBackend added in v0.4.0 

type TencentCloudCOSBackend struct {
	Bucket *cos.BucketService
	Object *cos.ObjectService
	Client *cos.Client
	Prefix string
}

TencentCloudCOSBackend is a storage backend for Tencent Cloud COS

func NewTencentCloudCOSBackend added in v0.4.0 

func NewTencentCloudCOSBackend(bucket string, prefix string, endpoint string) *TencentCloudCOSBackend

NewTencentCloudCOSBackend creates a new instance of TencentCloudCOSBackend

func (TencentCloudCOSBackend) DeleteObject added in v0.4.0 

func (t TencentCloudCOSBackend) DeleteObject(path string) error

DeleteObject removes an object from Tencent Cloud COS bucket, at prefix

func (TencentCloudCOSBackend) GetObject added in v0.4.0 

func (t TencentCloudCOSBackend) GetObject(path string) (Object, error)

GetObject retrieves an object from Tencent Cloud COS bucket, at prefix

func (TencentCloudCOSBackend) ListObjects added in v0.4.0 

func (t TencentCloudCOSBackend) ListObjects(prefix string) ([]Object, error)

ListObjects lists all objects in Tencent Cloud COS bucket, at prefix

func (TencentCloudCOSBackend) PutObject added in v0.4.0 

func (t TencentCloudCOSBackend) PutObject(path string, content []byte) error

PutObject uploads an object to Tencent Cloud COS bucket, at prefix

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.