Documentation

Overview

    Package upload provides upload components

    Index

    Constants

    This section is empty.

    Variables

      All is the list of all aws uploads.

      View Source
      var S3Desc = baker.UploadDesc{
      	Name:   "S3",
      	New:    NewS3,
      	Config: &S3Config{},
      	Help:   "S3Uploader uploads files to a destination on S3 that is relative to SourceBasePath",
      }

      Functions

      func NewS3

      func NewS3(cfg baker.UploadParams) (baker.Upload, error)

      Types

      type S3

      type S3 struct {
      	Cfg *S3Config
      	// contains filtered or unexported fields
      }

      func (*S3) Run

      func (u *S3) Run(upch <-chan string) error

      func (*S3) Stats

      func (u *S3) Stats() baker.UploadStats

      func (*S3) Stop

      func (u *S3) Stop()

      type S3Config

      type S3Config struct {
      	SourceBasePath string        `help:"Base path used to consider the final S3 path." default:"/tmp/baker/ologs/"`
      	Region         string        `help:"S3 region to upload to" default:"us-east-1"`
      	Bucket         string        `help:"S3 bucket to upload to"  required:"true"`
      	Prefix         string        `help:"Prefix on the destination bucket" default:"/"`
      	StagingPath    string        `help:"Local staging area to copy files to before upload." default:"/tmp/baker/ologs/staging/"`
      	Retries        int           `help:"Number of retries before a failed upload" default:"3"`
      	Concurrency    int           `help:"Number of concurrent workers" default:"5"`
      	Interval       time.Duration `help:"Period at which the source path is scanned" default:"15s"`
      	ExitOnError    bool          `help:"Exit at first error, instead of logging all errors" default:"false"`
      }

        S3Config holds the configuration for the S3 uploader.

        Each local path sent to the uploader channel in Run is sent to S3. The S3 destination of such files is determined by: - Bucket: S3 bucket name - Prefix: s3://bucket/prefix - the final component is the path of the file to upload that is relative to SourceBasePath. For example, if SourceBasePath is "/tmp/out", and the file to upload is "/tmp/out/foo/bar/file.gz", the final S3 path is: s3://bucket/prefix/foo/bar/file.gz.

        All files received by the uploader should be absolute and rooted at SourceBasePath.

        Source Files

        Directories

        Path Synopsis