pacer

package
v1.48.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jun 15, 2019 License: MIT Imports: 5 Imported by: 0

Documentation

Overview

Package pacer makes pacing and retrying API calls easy

Index

Constants

This section is empty.

Variables

This section is empty.

Functions

func IsRetryAfter added in v1.47.0 

func IsRetryAfter(err error) (retryAfter time.Duration, isRetryAfter bool)

IsRetryAfter returns true if the the error or any of it's Cause's is an error returned by RetryAfterError. It also returns the associated Duration if possible.

func RetryAfterError added in v1.47.0 

func RetryAfterError(err error, retryAfter time.Duration) error

RetryAfterError returns a wrapped error that can be used by Calculator implementations

Types

type AmazonCloudDrive added in v1.47.0 

type AmazonCloudDrive struct {
	// contains filtered or unexported fields
}

AmazonCloudDrive is a specialized pacer for Amazon Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to 0..2**retries seconds.

See https://developer.amazon.com/public/apis/experience/cloud-drive/content/restful-api-best-practices

func NewAmazonCloudDrive added in v1.47.0 

func NewAmazonCloudDrive(opts ...AmazonCloudDriveOption) *AmazonCloudDrive

NewAmazonCloudDrive returns a new AmazonCloudDrive Calculator with default values

func (*AmazonCloudDrive) Calculate added in v1.47.0 

func (c *AmazonCloudDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*AmazonCloudDrive) Update added in v1.47.0 

func (c *AmazonCloudDrive) Update(opts ...AmazonCloudDriveOption)

Update applies the Calculator options.

type AmazonCloudDriveOption added in v1.47.0 

type AmazonCloudDriveOption interface {
	ApplyAmazonCloudDrive(*AmazonCloudDrive)
}

AmazonCloudDriveOption is the interface implemented by all options for the AmazonCloudDrive Calculator

type AttackConstant added in v1.47.0 

type AttackConstant uint

AttackConstant configures the attack constant of a Calculator

func (AttackConstant) ApplyDefault added in v1.47.0 

func (o AttackConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (AttackConstant) ApplyS3 added in v1.47.0 

func (o AttackConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Burst added in v1.47.0 

type Burst int

Burst configures the number of API calls to allow without sleeping

func (Burst) ApplyGoogleDrive added in v1.47.0 

func (o Burst) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

type Calculator added in v1.47.0 

type Calculator interface {
	// Calculate takes the current Pacer state and returns the sleep time after which
	// the next Pacer call will be done.
	Calculate(state State) time.Duration
}

Calculator is a generic calculation function for a Pacer.

type DecayConstant added in v1.47.0 

type DecayConstant uint

DecayConstant configures the decay constant time of a Calculator

func (DecayConstant) ApplyDefault added in v1.47.0 

func (o DecayConstant) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (DecayConstant) ApplyS3 added in v1.47.0 

func (o DecayConstant) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Default added in v1.47.0 

type Default struct {
	// contains filtered or unexported fields
}

Default is a truncated exponential attack and decay.

On retries the sleep time is doubled, on non errors then sleeptime decays according to the decay constant as set with SetDecayConstant.

The sleep never goes below that set with SetMinSleep or above that set with SetMaxSleep.

func NewDefault added in v1.47.0 

func NewDefault(opts ...DefaultOption) *Default

NewDefault creates a Calculator used by Pacer as the default.

func (*Default) Calculate added in v1.47.0 

func (c *Default) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*Default) Update added in v1.47.0 

func (c *Default) Update(opts ...DefaultOption)

Update applies the Calculator options.

type DefaultOption added in v1.47.0 

type DefaultOption interface {
	ApplyDefault(*Default)
}

DefaultOption is the interface implemented by all options for the Default Calculator

type GoogleDrive added in v1.47.0 

type GoogleDrive struct {
	// contains filtered or unexported fields
}

GoogleDrive is a specialized pacer for Google Drive

It implements a truncated exponential backoff strategy with randomization. Normally operations are paced at the interval set with SetMinSleep. On errors the sleep timer is set to (2 ^ n) + random_number_milliseconds seconds.

See https://developers.google.com/drive/v2/web/handle-errors#exponential-backoff

func NewGoogleDrive added in v1.47.0 

func NewGoogleDrive(opts ...GoogleDriveOption) *GoogleDrive

NewGoogleDrive returns a new GoogleDrive Calculator with default values

func (*GoogleDrive) Calculate added in v1.47.0 

func (c *GoogleDrive) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*GoogleDrive) Update added in v1.47.0 

func (c *GoogleDrive) Update(opts ...GoogleDriveOption)

Update applies the Calculator options.

type GoogleDriveOption added in v1.47.0 

type GoogleDriveOption interface {
	ApplyGoogleDrive(*GoogleDrive)
}

GoogleDriveOption is the interface implemented by all options for the GoogleDrive Calculator

type InvokerFunc added in v1.47.0 

type InvokerFunc func(try, tries int, f Paced) (bool, error)

InvokerFunc is the signature of the wrapper function used to invoke the target function in Pacer.

type MaxSleep added in v1.47.0 

type MaxSleep time.Duration

MaxSleep configures the maximum sleep time of a Calculator

func (MaxSleep) ApplyDefault added in v1.47.0 

func (o MaxSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MaxSleep) ApplyS3 added in v1.47.0 

func (o MaxSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type MinSleep added in v1.47.0 

type MinSleep time.Duration

MinSleep configures the minimum sleep time of a Calculator

func (MinSleep) ApplyAmazonCloudDrive added in v1.47.0 

func (o MinSleep) ApplyAmazonCloudDrive(c *AmazonCloudDrive)

ApplyAmazonCloudDrive updates the value on the Calculator

func (MinSleep) ApplyDefault added in v1.47.0 

func (o MinSleep) ApplyDefault(c *Default)

ApplyDefault updates the value on the Calculator

func (MinSleep) ApplyGoogleDrive added in v1.47.0 

func (o MinSleep) ApplyGoogleDrive(c *GoogleDrive)

ApplyGoogleDrive updates the value on the Calculator

func (MinSleep) ApplyS3 added in v1.47.0 

func (o MinSleep) ApplyS3(c *S3)

ApplyS3 updates the value on the Calculator

type Option added in v1.47.0 

type Option func(*pacerOptions)

Option can be used in New to configure the Pacer.

func CalculatorOption added in v1.47.0 

func CalculatorOption(c Calculator) Option

CalculatorOption sets a Calculator for the new Pacer.

func InvokerOption added in v1.47.0 

func InvokerOption(invoker InvokerFunc) Option

InvokerOption sets a InvokerFunc for the new Pacer.

func MaxConnectionsOption added in v1.47.0 

func MaxConnectionsOption(maxConnections int) Option

MaxConnectionsOption sets the maximum connections number for the new Pacer.

func RetriesOption added in v1.47.0 

func RetriesOption(retries int) Option

RetriesOption sets the retries number for the new Pacer.

type Paced 

type Paced func() (bool, error)

Paced is a function which is called by the Call and CallNoRetry methods. It should return a boolean, true if it would like to be retried, and an error. This error may be returned or returned wrapped in a RetryError.

type Pacer 

type Pacer struct {
	// contains filtered or unexported fields
}

Pacer is the primary type of the pacer package. It allows to retry calls with a configurable delay in between.

func New 

func New(options ...Option) *Pacer

New returns a Pacer with sensible defaults.

func (*Pacer) Call 

func (p *Pacer) Call(fn Paced) (err error)

Call paces the remote operations to not exceed the limits and retry on rate limit exceeded

This calls fn, expecting it to return a retry flag and an error. This error may be returned wrapped in a RetryError if the number of retries is exceeded.

func (*Pacer) CallNoRetry 

func (p *Pacer) CallNoRetry(fn Paced) error

CallNoRetry paces the remote operations to not exceed the limits and return a retry error on rate limit exceeded

This calls fn and wraps the output in a RetryError if it would like it to be retried

func (*Pacer) ModifyCalculator added in v1.47.0 

func (p *Pacer) ModifyCalculator(f func(Calculator))

ModifyCalculator calls the given function with the currently configured Calculator and the Pacer lock held.

func (*Pacer) SetCalculator added in v1.47.0 

func (p *Pacer) SetCalculator(c Calculator)

SetCalculator sets the pacing algorithm. Don't modify the Calculator object afterwards, use the ModifyCalculator method when needed.

It will choose the default algorithm if nil is passed in.

func (*Pacer) SetMaxConnections 

func (p *Pacer) SetMaxConnections(n int)

SetMaxConnections sets the maximum number of concurrent connections. Setting the value to 0 will allow unlimited number of connections. Should not be changed once you have started calling the pacer. By default this will be set to fs.Config.Checkers.

func (*Pacer) SetRetries 

func (p *Pacer) SetRetries(retries int)

SetRetries sets the max number of retries for Call

type S3 added in v1.47.0 

type S3 struct {
	// contains filtered or unexported fields
}

S3 implements a pacer compatible with our expectations of S3, where it tries to not delay at all between successful calls, but backs off in the default fashion in response to any errors. The assumption is that errors should be exceedingly rare (S3 seems to have largely solved the sort of stability questions rclone is likely to run into), and in the happy case it can handle calls with no delays between them.

Basically defaultPacer, but with some handling of sleepTime going to/from 0ms

func NewS3 added in v1.47.0 

func NewS3(opts ...S3Option) *S3

NewS3 returns a new S3 Calculator with default values

func (*S3) Calculate added in v1.47.0 

func (c *S3) Calculate(state State) time.Duration

Calculate takes the current Pacer state and return the wait time until the next try.

func (*S3) Update added in v1.47.0 

func (c *S3) Update(opts ...S3Option)

Update applies the Calculator options.

type S3Option added in v1.47.0 

type S3Option interface {
	ApplyS3(*S3)
}

S3Option is the interface implemented by all options for the S3 Calculator

type State added in v1.47.0 

type State struct {
	SleepTime          time.Duration // current time to sleep before adding the pacer token back
	ConsecutiveRetries int           // number of consecutive retries, will be 0 when the last invoker call returned false
	LastError          error         // the error returned by the last invoker call or nil
}

State represents the public Pacer state that will be passed to the configured Calculator

type TokenDispenser 

type TokenDispenser struct {
	// contains filtered or unexported fields
}

TokenDispenser is for controlling concurrency

func NewTokenDispenser 

func NewTokenDispenser(n int) *TokenDispenser

NewTokenDispenser makes a pool of n tokens

func (*TokenDispenser) Get 

func (td *TokenDispenser) Get()

Get gets a token from the pool - don't forget to return it with Put

func (*TokenDispenser) Put 

func (td *TokenDispenser) Put()

Put returns a token

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.