README
¶
chartmuseum/storage
Go library providing a common interface for working across multiple storage backends.
Supported storage backends:
- Local filesystem
- Amazon S3
- Google Cloud Storage
- Microsoft Azure Blob Storage
- Alibaba Cloud OSS Storage
- Openstack Object Storage
- Oracle Cloud Infrastructure Object Storage
This code was originally part of the Helm project, ChartMuseum, but has since been released as a standalone package for others to use in their own projects.
Primary Components
Backend (interface)
Backend is a common interface that is implemented by all the supported storage backends and their associated types:
type Backend interface {
ListObjects(prefix string) ([]Object, error)
GetObject(path string) (Object, error)
PutObject(path string, content []byte) error
DeleteObject(path string) error
}
Object (struct)
Object is a struct that represents a single storage object:
type Object struct {
Path string
Content []byte
LastModified time.Time
}
ObjectSliceDiff (struct)
ObjectSliceDiff is a struct that represents overall changes between two Object slices:
type ObjectSliceDiff struct {
Change bool
Removed []Object
Added []Object
Updated []Object
}
GetObjectSliceDiff (function)
GetObjectSliceDiff is a function that takes two Object slices, compares them, and returns an ObjectSliceDiff:
func GetObjectSliceDiff(os1 []Object, os2 []Object) ObjectSliceDiff
Usage
Simple example
The following is a simple program that will upload a file either to an Azure Blob Storage bucket (container) or a Google Cloud Storage bucket based on the command line options provided:
// Usage: go run example.go <cloud> <bucket> <file>
package main
import (
"fmt"
"github.com/chartmuseum/storage"
"io/ioutil"
"os"
"path/filepath"
)
type (
Uploader struct {
Backend storage.Backend
}
)
func NewUploader(cloud string, bucket string) *Uploader {
var backend storage.Backend
switch cloud {
case "azure":
backend = storage.Backend(storage.NewMicrosoftBlobBackend(bucket, ""))
case "google":
backend = storage.Backend(storage.NewGoogleCSBackend(bucket, ""))
default:
panic("cloud provider " + cloud + " not supported")
}
uploader := Uploader{Backend: backend}
fmt.Printf("uploader created (cloud: %s, bucket: %s)\n", cloud, bucket)
return &uploader
}
func (uploader *Uploader) Upload(filename string) {
basename := filepath.Base(filename)
content, err := ioutil.ReadFile(filename)
if err != nil {
panic(err)
}
err = uploader.Backend.PutObject(basename, content)
if err != nil {
panic(err)
}
fmt.Printf("%s successfully uploaded\n", basename)
}
func main() {
args := os.Args[1:]
uploader := NewUploader(args[0], args[1])
uploader.Upload(args[2])
}
Example of using to upload the file index.html to an Azure bucket:
go run example.go azure mycontainer index.html
Example of using to upload the file index.html to a Google Cloud bucket:
go run example.go google mybucket index.html
Per backend
Each supported storage backend has its own type that implements the Backend interface.
All available types are described in detail on GoDoc.
In addition, authentication methods are based on the runtime environment and vary from cloud to cloud.
Documentation
¶
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
This section is empty.
Types ¶
type AlibabaCloudOSSBackend ¶
type AlibabaCloudOSSBackend struct {
Bucket *oss.Bucket
Client *oss.Client
Prefix string
SSE string
}
AlibabaCloudOSSBackend is a storage backend for Alibaba Cloud OSS
func NewAlibabaCloudOSSBackend ¶
func NewAlibabaCloudOSSBackend(bucket string, prefix string, endpoint string, sse string) *AlibabaCloudOSSBackend
NewAlibabaCloudOSSBackend creates a new instance of AlibabaCloudOSSBackend
func (AlibabaCloudOSSBackend) DeleteObject ¶
func (b AlibabaCloudOSSBackend) DeleteObject(path string) error
DeleteObject removes an object from Alibaba Cloud OSS bucket, at prefix
func (AlibabaCloudOSSBackend) GetObject ¶
func (b AlibabaCloudOSSBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from Alibaba Cloud OSS bucket, at prefix
func (AlibabaCloudOSSBackend) ListObjects ¶
func (b AlibabaCloudOSSBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in Alibaba Cloud OSS bucket, at prefix
type AmazonS3Backend ¶
type AmazonS3Backend struct {
Bucket string
Client *s3.S3
Downloader *s3manager.Downloader
Prefix string
Uploader *s3manager.Uploader
SSE string
}
AmazonS3Backend is a storage backend for Amazon S3
func NewAmazonS3Backend ¶
func NewAmazonS3Backend(bucket string, prefix string, region string, endpoint string, sse string) *AmazonS3Backend
NewAmazonS3Backend creates a new instance of AmazonS3Backend
func (AmazonS3Backend) DeleteObject ¶
func (b AmazonS3Backend) DeleteObject(path string) error
DeleteObject removes an object from Amazon S3 bucket, at prefix
func (AmazonS3Backend) GetObject ¶
func (b AmazonS3Backend) GetObject(path string) (Object, error)
GetObject retrieves an object from Amazon S3 bucket, at prefix
func (AmazonS3Backend) ListObjects ¶
func (b AmazonS3Backend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in Amazon S3 bucket, at prefix
type Backend ¶
type Backend interface {
ListObjects(prefix string) ([]Object, error)
GetObject(path string) (Object, error)
PutObject(path string, content []byte) error
DeleteObject(path string) error
}
Backend is a generic interface for storage backends
type BaiduBOSBackend ¶ added in v0.3.0
BaiduBOSBackend is a storage backend for Baidu Cloud BOS
func NewBaiDuBOSBackend ¶ added in v0.3.0
func NewBaiDuBOSBackend(bucket string, prefix string, endpoint string) *BaiduBOSBackend
NewBaiduBOSBackend creates a new instance of BaiduBOSBackend
func (BaiduBOSBackend) DeleteObject ¶ added in v0.3.0
func (b BaiduBOSBackend) DeleteObject(path string) error
DeleteObject removes an object from Baidu Cloud BOS bucket, at prefix
func (BaiduBOSBackend) GetObject ¶ added in v0.3.0
func (b BaiduBOSBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from Baidu Cloud BOS bucket, at prefix
func (BaiduBOSBackend) ListObjects ¶ added in v0.3.0
func (b BaiduBOSBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in Baidu Cloud BOS bucket, at prefix
type GoogleCSBackend ¶
type GoogleCSBackend struct {
Prefix string
Client *storage.BucketHandle
Context context.Context
}
GoogleCSBackend is a storage backend for Google Cloud Storage
func NewGoogleCSBackend ¶
func NewGoogleCSBackend(bucket string, prefix string) *GoogleCSBackend
NewGoogleCSBackend creates a new instance of GoogleCSBackend
func (GoogleCSBackend) DeleteObject ¶
func (b GoogleCSBackend) DeleteObject(path string) error
DeleteObject removes an object from Google Cloud Storage bucket, at prefix
func (GoogleCSBackend) GetObject ¶
func (b GoogleCSBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from Google Cloud Storage bucket, at prefix
func (GoogleCSBackend) ListObjects ¶
func (b GoogleCSBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in Google Cloud Storage bucket, at prefix
type LocalFilesystemBackend ¶
type LocalFilesystemBackend struct {
RootDirectory string
}
LocalFilesystemBackend is a storage backend for local filesystem storage
func NewLocalFilesystemBackend ¶
func NewLocalFilesystemBackend(rootDirectory string) *LocalFilesystemBackend
NewLocalFilesystemBackend creates a new instance of LocalFilesystemBackend
func (LocalFilesystemBackend) DeleteObject ¶
func (b LocalFilesystemBackend) DeleteObject(path string) error
DeleteObject removes an object from root directory
func (LocalFilesystemBackend) GetObject ¶
func (b LocalFilesystemBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from root directory
func (LocalFilesystemBackend) ListObjects ¶
func (b LocalFilesystemBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in root directory (depth 1)
type MicrosoftBlobBackend ¶
type MicrosoftBlobBackend struct {
Prefix string
Container *microsoft_storage.Container
}
MicrosoftBlobBackend is a storage backend for Microsoft Azure Blob Storage
func NewMicrosoftBlobBackend ¶
func NewMicrosoftBlobBackend(container string, prefix string) *MicrosoftBlobBackend
NewMicrosoftBlobBackend creates a new instance of MicrosoftBlobBackend
func (MicrosoftBlobBackend) DeleteObject ¶
func (b MicrosoftBlobBackend) DeleteObject(path string) error
DeleteObject removes an object from Microsoft Azure Blob Storage container, at path
func (MicrosoftBlobBackend) GetObject ¶
func (b MicrosoftBlobBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from Microsoft Azure Blob Storage, at path
func (MicrosoftBlobBackend) ListObjects ¶
func (b MicrosoftBlobBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in Microsoft Azure Blob Storage container
type Object ¶
Object is a generic representation of a storage object
func (Object) HasExtension ¶
HasExtension determines whether or not an object contains a file extension
type ObjectSliceDiff ¶
ObjectSliceDiff provides information on what has changed since last calling ListObjects
func GetObjectSliceDiff ¶
func GetObjectSliceDiff(os1 []Object, os2 []Object) ObjectSliceDiff
GetObjectSliceDiff takes two objects slices and returns an ObjectSliceDiff
type OpenstackOSBackend ¶
type OpenstackOSBackend struct {
Container string
Prefix string
Region string
CACert string
Client *gophercloud.ServiceClient
}
OpenstackOSBackend is a storage backend for Openstack Object Storage
func NewOpenstackOSBackend ¶
func NewOpenstackOSBackend(container string, prefix string, region string, caCert string) *OpenstackOSBackend
NewOpenstackOSBackend creates a new instance of OpenstackOSBackend
func (OpenstackOSBackend) DeleteObject ¶
func (b OpenstackOSBackend) DeleteObject(path string) error
DeleteObject removes an object from an Openstack container, at prefix
func (OpenstackOSBackend) GetObject ¶
func (b OpenstackOSBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from an Openstack container, at prefix
func (OpenstackOSBackend) ListObjects ¶
func (b OpenstackOSBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in an Openstack container, at prefix
type OracleCSBackend ¶
type OracleCSBackend struct {
Bucket string
Prefix string
Namespace string
CompartmentId string
Client objectstorage.ObjectStorageClient
Context context.Context
}
OracleCSBackend is a storage backend for Oracle Cloud Infrastructure Object Storage
func NewOracleCSBackend ¶
func NewOracleCSBackend(bucket string, prefix string, region string, compartmentId string) *OracleCSBackend
NewOracleCSBackend creates a new instance of OracleCSBackend
func (OracleCSBackend) DeleteObject ¶
func (b OracleCSBackend) DeleteObject(path string) error
DeleteObject removes an object from OCI Object Storage bucket, at prefix
func (OracleCSBackend) GetObject ¶
func (b OracleCSBackend) GetObject(path string) (Object, error)
GetObject retrieves an object from OCI Object Storage bucket, at prefix
func (OracleCSBackend) ListObjects ¶
func (b OracleCSBackend) ListObjects(prefix string) ([]Object, error)
ListObjects lists all objects in OCI Object Storage bucket, at prefix
type ReauthRoundTripper ¶
type ReauthRoundTripper struct {
// contains filtered or unexported fields
}
ReauthRoundTripper satisfies the http.RoundTripper interface and is used to limit the number of consecutive re-auth attempts (infinite by default)
Source Files