ots3

Documentation

Overview

Package ots3 provides a s3 uploader with opentracing capabilities.

Introduction

S3 is the de facto standard for cloud file systems.The transport of S3 is HTTP(s) which is pleasantly simple to trace. This package also features a go kit server and client for the upload service.

Simple Usage

Creating a s3 manager:

var manager = NewManager(accessKey, accessSecret, endpoint, region, bucket)
url, err := manager.Upload(context.Background(), "myfile", file)

Integration

Package ots3 exports the following configuration:

s3:
  default:
    accessKey:
    accessSecret:
    bucket:
    endpoint:
    region:
    cdnUrl:

To use package ots3 with package core:

var c *core.C = core.New()
c.Provide(ots3.Providers())
c.AddModuleFunc(ots3.New)
c.Invoke(function(manager *ots3.Manager) {
	// do something with manager
})

Adding the module created by mods3.New is optional. This module provides an "/upload" path for the http router. If this is not relevant, just leave it factoryOut.

Sometimes there are valid reasons to connect to more than one s3 server. Inject mods3.Maker to factory a *ots3.Manager with a specific configuration entry.

c.Invoke(function(maker ots3.Maker) {
	manager, err := maker.Make("default")
})

Future scope

Currently this package only focus on the file upload aspect of s3. Other s3 features can be incrementally implemented.

Example

if os.Getenv("S3_ENDPOINT") == "" {
	fmt.Println("set S3_ENDPOINT to run this exmaple")
	return
}
if os.Getenv("S3_ACCESSKEY") == "" {
	fmt.Println("set S3_ACCESSKEY to run this exmaple")
	return
}
if os.Getenv("S3_ACCESSSECRET") == "" {
	fmt.Println("set S3_ACCESSSECRET to run this exmaple")
	return
}
if os.Getenv("S3_BUCKET") == "" {
	fmt.Println("set S3_BUCKET to run this exmaple")
	return
}
if os.Getenv("S3_REGION") == "" {
	fmt.Println("set S3_REGION to run this exmaple")
	return
}
file, err := os.Open("./testdata/basn0g01-30.png")
if err != nil {
	panic(err)
}
defer file.Close()
uploader := NewManager(os.Getenv("S3_ACCESSKEY"), os.Getenv("S3_ACCESSSECRET"), os.Getenv("S3_ENDPOINT"), os.Getenv("S3_REGION"), os.Getenv("S3_BUCKET"))
_ = uploader.CreateBucket(context.Background(), os.Getenv("S3_BUCKET"))
url, _ := uploader.Upload(context.Background(), "foo", file)
url = strings.Replace(url, os.Getenv("S3_ENDPOINT"), "http://example.org", 1)
fmt.Println(url)

Output:

http://example.org/mybucket/foo.png

Index

• func Providers(optionFunc ...ProvidersOptionFunc) di.Deps
• type Factory
  • func (s Factory) Make(name string) (*Manager, error)
• type Maker
• type Manager
  • func NewManager(accessKey, accessSecret, endpoint, region, bucket string, ...) *Manager
  • func (m *Manager) CreateBucket(ctx context.Context, name string) error
  • func (m *Manager) Upload(ctx context.Context, name string, reader io.Reader) (newUrl string, err error)
  • func (m *Manager) UploadFromUrl(ctx context.Context, url string) (newUrl string, err error)
• type ManagerArgs
• type ManagerConfig
• type ManagerConstructor
• type ManagerOptionFunc
  • func WithAutoExtension(auto bool) ManagerOptionFunc
  • func WithHTTPClient(client contract.HttpDoer) ManagerOptionFunc
  • func WithKeyer(keyer contract.Keyer) ManagerOptionFunc
  • func WithLocationFunc(f func(location string) (url string)) ManagerOptionFunc
  • func WithPathPrefix(pathPrefix string) ManagerOptionFunc
  • func WithTracer(tracer opentracing.Tracer) ManagerOptionFunc
• type ProvidersOptionFunc
  • func WithManagerConstructor(ctor ManagerConstructor) ProvidersOptionFunc
  • func WithReload(shouldReload bool) ProvidersOptionFunc
• type S3Config
• type Uploader

Examples

• Package

Functions

func Providers

func Providers(optionFunc ...ProvidersOptionFunc) di.Deps

Providers returns a set of dependencies providers related to S3. It includes the s3 Manager, the Maker and exported configurations.

Depends On:
	log.Logger
	contract.ConfigAccessor
	opentracing.Tracer `optional:"true"`
	contract.DIPopulator `optional:"true"`
Provide:
	Factory
	Maker
	*Manager
	Uploader

Types

type Factory

type Factory struct {
	*di.Factory
}

Factory can be used to connect to multiple s3 servers.

func (Factory) Make

func (s Factory) Make(name string) (*Manager, error)

Make creates a s3 manager under the given name.

type Maker

type Maker interface {
	Make(name string) (*Manager, error)
}

Maker is an interface for *Factory. Used as a type hint for injection.

type Manager

type Manager struct {
	// contains filtered or unexported fields
}

Manager manages S3 uploads.

func NewManager

func NewManager(accessKey, accessSecret, endpoint, region, bucket string, opts ...ManagerOptionFunc) *Manager

NewManager creates a new S3 manager

func (*Manager) CreateBucket

func (m *Manager) CreateBucket(ctx context.Context, name string) error

CreateBucket create a buckets factoryIn s3 server. TODO: handle acl

func (*Manager) Upload

func (m *Manager) Upload(ctx context.Context, name string, reader io.Reader) (newUrl string, err error)

Upload uploads an io.reader to the S3 server, and returns the url on S3. The extension of the uploaded file is auto detected.

func (*Manager) UploadFromUrl

func (m *Manager) UploadFromUrl(ctx context.Context, url string) (newUrl string, err error)

UploadFromUrl fetches a file from an external url, copy them to the S3 server, and generate a new, local url. It uses streams to relay files (instead of buffering the entire file factoryIn memory). it gives the file a random name using the global seed.

type ManagerArgs

type ManagerArgs struct {
	Name      string
	Conf      S3Config
	Populator contract.DIPopulator
}

ManagerArgs are arguments for constructing the s3 manager. When providing custom constructors, take this as input.

type ManagerConfig

type ManagerConfig struct {
	// contains filtered or unexported fields
}

ManagerConfig contains a various configurations for Manager. It means to be modified by ManagerOptionFunc.

type ManagerConstructor

type ManagerConstructor func(args ManagerArgs) (*Manager, error)

ManagerConstructor constructs a manager

type ManagerOptionFunc

type ManagerOptionFunc func(*ManagerConfig)

ManagerOptionFunc is the type of functional options to alter ManagerConfig.

func WithAutoExtension

func WithAutoExtension(auto bool) ManagerOptionFunc

WithAutoExtension is an option that auto splice extension, default is true.

func WithHTTPClient

func WithHTTPClient(client contract.HttpDoer) ManagerOptionFunc

WithHTTPClient is an option that replaces the default http client. Useful for interceptors like tracing and metrics.

func WithKeyer

func WithKeyer(keyer contract.Keyer) ManagerOptionFunc

WithKeyer is an option that changes the path of the uploaded file.

func WithLocationFunc

func WithLocationFunc(f func(location string) (url string)) ManagerOptionFunc

WithLocationFunc is an option that decides how url is mapped to S3 bucket and path. Useful when not serving file directly from S3, but from a CDN.

func WithPathPrefix

func WithPathPrefix(pathPrefix string) ManagerOptionFunc

WithPathPrefix is an option that changes the path prefix of uploaded file.

func WithTracer

func WithTracer(tracer opentracing.Tracer) ManagerOptionFunc

WithTracer is an option that add opentracing.Tracer via the hook of S3 client.

type ProvidersOptionFunc

type ProvidersOptionFunc func(options *providersOption)

ProvidersOptionFunc is the type of functional providersOption for Providers. Use this type to change how Providers work.

func WithManagerConstructor

func WithManagerConstructor(ctor ManagerConstructor) ProvidersOptionFunc

WithManagerConstructor is a provider option to override how s3 manager are constructed.

func WithReload

func WithReload(shouldReload bool) ProvidersOptionFunc

WithReload toggles whether the factory should reload cached instances upon OnReload event.

type S3Config

type S3Config struct {
	AccessKey    string `json:"accessKey" yaml:"accessKey"`
	AccessSecret string `json:"accessSecret" yaml:"accessSecret"`
	Endpoint     string `json:"endpoint" yaml:"endpoint"`
	Region       string `json:"region" yaml:"region"`
	Bucket       string `json:"bucket" yaml:"bucket"`
	CdnUrl       string `json:"cdnUrl" yaml:"cdnUrl"`
}

S3Config contains credentials of S3 server

type Uploader

type Uploader interface {
	// Upload the bytes from io.Reader with a given filename to a server, and returns the url and error.
	Upload(ctx context.Context, name string, reader io.Reader) (string, error)
}

Uploader models UploadService