upload

Documentation

Overview

Package upload provides upload components

Index

• Variables
• func NewS3(cfg baker.UploadParams) (baker.Upload, error)
• type S3
  • func (u *S3) Run(upch <-chan string) error
  • func (u *S3) Stats() baker.UploadStats
  • func (u *S3) Stop()
• type S3Config

Variables

var All = []baker.UploadDesc{
	S3Desc,
}

All is the list of all aws uploads.

var S3Desc = baker.UploadDesc{
	Name:   "S3",
	New:    NewS3,
	Config: &S3Config{},
	Help:   "S3Uploader uploads files to a destination on S3 that is relative to SourceBasePath",
}

Functions

func NewS3

func NewS3(cfg baker.UploadParams) (baker.Upload, error)

Types

type S3

type S3 struct {
	Cfg *S3Config
	// contains filtered or unexported fields
}

func (*S3) Run

func (u *S3) Run(upch <-chan string) error

func (*S3) Stats

func (u *S3) Stats() baker.UploadStats

func (*S3) Stop

func (u *S3) Stop()

type S3Config

type S3Config struct {
	SourceBasePath string        `help:"Base path used to consider the final S3 path." default:"/tmp/baker/ologs/"`
	Region         string        `help:"S3 region to upload to" default:"us-east-1"`
	Bucket         string        `help:"S3 bucket to upload to"  required:"true"`
	Prefix         string        `help:"Prefix on the destination bucket" default:"/"`
	StagingPath    string        `help:"Local staging area to copy files to before upload." default:"/tmp/baker/ologs/staging/"`
	Retries        int           `help:"Number of retries before a failed upload" default:"3"`
	Concurrency    int           `help:"Number of concurrent workers" default:"5"`
	Interval       time.Duration `help:"Period at which the source path is scanned" default:"15s"`
	ExitOnError    bool          `help:"Exit at first error, instead of logging all errors" default:"false"`
}

S3Config holds the configuration for the S3 uploader.

Each local path sent to the uploader channel in Run is sent to S3. The S3 destination of such files is determined by: - Bucket: S3 bucket name - Prefix: s3://bucket/prefix - the final component is the path of the file to upload that is relative to SourceBasePath. For example, if SourceBasePath is "/tmp/out", and the file to upload is "/tmp/out/foo/bar/file.gz", the final S3 path is: s3://bucket/prefix/foo/bar/file.gz.

All files received by the uploader should be absolute and rooted at SourceBasePath.