manager

Documentation

Overview

This file was copied exactly from https://github.com/aws/aws-sdk-go-v2/blob/b7d8e15425d2f86a0596e8d7db2e33bf382a21dd/feature/s3/manager/arn.go#L8

This file was copied exactly from https://github.com/aws/aws-sdk-go-v2/blob/b7d8e15425d2f86a0596e8d7db2e33bf382a21dd/internal/awsutil/copy.go

This file was copied and then modified from: https://github.com/aws/aws-sdk-go-v2/blob/b7d8e15425d2f86a0596e8d7db2e33bf382a21dd/feature/s3/manager/download.go This version of the manager is built to work with io.Writer instead of io.WriterAt. It achieves this by recycling some larger internal buffers and stalling out downloads that are too far ahead in the stream until the ones at the start catching up.

Index

• Constants
• func Copy(dst, src interface{})
• func CopyOf(src interface{}) (dst interface{})
• func WithDownloaderClientOptions(opts ...func(*s3.Options)) func(*Downloader)
• type Downloader
  • func NewDownloader(c manager.DownloadAPIClient, options ...func(*Downloader)) *Downloader
  • func (d Downloader) Download(ctx context.Context, w io.Writer, input *s3.GetObjectInput, ...) (n int64, err error)

Constants

const DefaultDownloadConcurrency = 5

DefaultDownloadConcurrency is the default number of goroutines to spin up when using Download().

const DefaultDownloadPartSize = 1024 * 1024 * 5

DefaultDownloadPartSize is the default range of bytes to get at a time when using Download().

const DefaultMaxBufferedParts = 8

DefaultMaxBufferedParts is the default number of buffered parts internal to the downloader

const DefaultPartBodyMaxRetries = 3

DefaultPartBodyMaxRetries is the default number of retries to make when a part fails to upload.

Functions

func Copy

func Copy(dst, src interface{})

Copy deeply copies a src structure to dst. Useful for copying request and response structures.

Can copy between structs of different type, but will only copy fields which are assignable, and exist in both structs. Fields which are not assignable, or do not exist in both structs are ignored.

func CopyOf

func CopyOf(src interface{}) (dst interface{})

CopyOf returns a copy of src while also allocating the memory for dst. src must be a pointer type or this operation will fail.

func WithDownloaderClientOptions

func WithDownloaderClientOptions(opts ...func(*s3.Options)) func(*Downloader)

WithDownloaderClientOptions appends to the Downloader's API request options.

Types

type Downloader

type Downloader struct {
	// The size (in bytes) to request from S3 for each part.
	// The minimum allowed part size is 5MB, and  if this value is set to zero,
	// the DefaultDownloadPartSize value will be used.
	//
	// PartSize is ignored if the Range input parameter is provided.
	PartSize int64

	// PartBodyMaxRetries is the number of retry attempts to make for failed part uploads
	PartBodyMaxRetries int

	// Enable Logging of part download retry attempts
	LogInterruptedDownloads bool

	// The number of goroutines to spin up in parallel when sending parts.
	// If this is set to zero, the DefaultDownloadConcurrency value will be used.
	//
	// Concurrency of 1 will download the parts sequentially.
	Concurrency int

	// MaxBufferedParts should be greater than or equal to Concurrency
	// It represents how many PartSize buffers could potentially be buffered while
	// waiting on the next bytes to write to the writer.
	MaxBufferedParts int

	// An S3 client to use when performing downloads.
	S3 manager.DownloadAPIClient

	// List of client options that will be passed down to individual API
	// operation requests made by the downloader.
	ClientOptions []func(*s3.Options)

	// buffer pool containing bytes.Buffers
	BufPool iostream.BufPool
}

The Downloader structure that calls Download(). It is safe to call Download() on this structure for multiple objects and across concurrent goroutines. Mutating the Downloader's properties is not safe to be done concurrently.

func NewDownloader

func NewDownloader(c manager.DownloadAPIClient, options ...func(*Downloader)) *Downloader

NewDownloader creates a new Downloader instance to downloads objects from S3 in concurrent chunks. Pass in additional functional options to customize the downloader behavior. Requires a client.ConfigProvider in order to create a S3 service client. The session.Session satisfies the client.ConfigProvider interface.

Example:

// Load AWS Config
cfg, err := config.LoadDefaultConfig(context.TODO())
if err != nil {
	panic(err)
}

// Create an S3 client using the loaded configuration
s3.NewFromConfig(cfg)

// Create a downloader passing it the S3 client
downloader := manager.NewDownloader(s3.NewFromConfig(cfg))

// Create a downloader with the client and custom downloader options
downloader := manager.NewDownloader(client, func(d *manager.Downloader) {
	d.PartSize = 64 * 1024 * 1024 // 64MB per part
})

func (Downloader) Download

func (d Downloader) Download(ctx context.Context, w io.Writer, input *s3.GetObjectInput, options ...func(*Downloader)) (n int64, err error)

Download downloads an object in S3 and writes the payload into w using concurrent GET requests. The n int64 returned is the size of the object downloaded/ in bytes.

It is safe to call this method concurrently across goroutines.

GetObjectInputs with the "Range" value provided are not allowed and will return an error.