upload

package
Version: v0.1.0-alpha Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Mar 19, 2021 License: MIT Imports: 13 Imported by: 0

Documentation

Overview

Package upload provides upload components

Index

Constants

This section is empty.

Variables

All is the list of all aws uploads.

View Source
var S3Desc = baker.UploadDesc{
	Name:   "S3",
	New:    NewS3,
	Config: &S3Config{},
	Help:   "S3Uploader uploads files to a destination on S3 that is relative to SourceBasePath",
}

Functions

func NewS3

func NewS3(cfg baker.UploadParams) (baker.Upload, error)

Types

type S3

type S3 struct {
	Cfg *S3Config
	// contains filtered or unexported fields
}

func (*S3) Run

func (u *S3) Run(upch <-chan string) error

func (*S3) Stats

func (u *S3) Stats() baker.UploadStats

func (*S3) Stop

func (u *S3) Stop()

type S3Config

type S3Config struct {
	SourceBasePath string        `help:"Base path used to consider the final S3 path." default:"/tmp/baker/ologs/"`
	Region         string        `help:"S3 region to upload to" default:"us-east-1"`
	Bucket         string        `help:"S3 bucket to upload to"  required:"true"`
	Prefix         string        `help:"Prefix on the destination bucket" default:"/"`
	StagingPath    string        `help:"Local staging area to copy files to before upload." default:"/tmp/baker/ologs/staging/"`
	Retries        int           `help:"Number of retries before a failed upload" default:"3"`
	Concurrency    int           `help:"Number of concurrent workers" default:"5"`
	Interval       time.Duration `help:"Period at which the source path is scanned" default:"15s"`
	ExitOnError    bool          `help:"Exit at first error, instead of logging all errors" default:"false"`
}

S3Config holds the configuration for the S3 uploader.

Each local path sent to the uploader channel in Run is sent to S3. The S3 destination of such files is determined by: - Bucket: S3 bucket name - Prefix: s3://bucket/prefix - the final component is the path of the file to upload that is relative to SourceBasePath. For example, if SourceBasePath is "/tmp/out", and the file to upload is "/tmp/out/foo/bar/file.gz", the final S3 path is: s3://bucket/prefix/foo/bar/file.gz.

All files received by the uploader should be absolute and rooted at SourceBasePath.

Source Files

Directories

Path Synopsis

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
t or T : Toggle theme light dark auto
y or Y : Canonical URL