pace

Documentation

Overview

Package pace decides when the next request may go: a token bucket per host, jittered exponential backoff between attempts, a retry policy, and a robots.txt check.

Index

• func RetryAfter(h http.Header, now time.Time) (time.Duration, bool)
• func Sleep(ctx context.Context, d time.Duration) error
• type Backoff
  • func (b Backoff) Delay(attempt int) time.Duration
• type Limiter
  • func NewLimiter(perSecond float64, burst int) *Limiter
  • func (l *Limiter) SetHost(host string, perSecond float64, burst int)
  • func (l *Limiter) Wait(ctx context.Context, host string) error
• type Policy
  • func (p Policy) Attempts() int
  • func (p Policy) ShouldRetry(status int, outcome detect.Outcome, err error) bool
  • func (p Policy) Wait(attempt int, h http.Header, now time.Time) time.Duration
• type Robots
  • func NewRobots(f fetch.Fetcher, ttl time.Duration) *Robots
  • func (r *Robots) Allowed(ctx context.Context, target, agent string, base fetch.Request) (bool, error)

Functions

func RetryAfter

func RetryAfter(h http.Header, now time.Time) (time.Duration, bool)

RetryAfter reads a Retry-After header, in seconds or as an HTTP date.

func Sleep

func Sleep(ctx context.Context, d time.Duration) error

Sleep waits for d or until ctx is done, whichever is first.

Types

type Backoff

type Backoff struct {
	Base   time.Duration // default 500ms
	Max    time.Duration // default 30s
	Factor float64       // default 2
	Jitter float64       // 0 to 1, default 0.5; negative disables jitter
	// Rand returns a number in [0, 1). Nil uses math/rand/v2.
	Rand func() float64
}

Backoff computes the delay before a retry. The delay grows as Base * Factor^attempt, is capped at Max, and is then spread by Jitter: with Jitter 0.5 the delay lands anywhere from half to one and a half times the computed value (still capped at Max), so retries from many workers do not arrive together.

func (Backoff) Delay

func (b Backoff) Delay(attempt int) time.Duration

Delay returns the delay before retry number attempt, counting from 0.

type Limiter

type Limiter struct {
	// contains filtered or unexported fields
}

Limiter is a token bucket per host. Hosts without their own setting share the default rate, each with a bucket of its own. It is safe for concurrent use.

func NewLimiter

func NewLimiter(perSecond float64, burst int) *Limiter

NewLimiter returns a Limiter allowing perSecond requests per second to each host, with bursts of up to burst. A burst below 1 is raised to 1.

func (*Limiter) SetHost

func (l *Limiter) SetHost(host string, perSecond float64, burst int)

SetHost gives one host its own rate. It takes effect at once, replacing the host's bucket.

func (*Limiter) Wait

func (l *Limiter) Wait(ctx context.Context, host string) error

Wait blocks until host may be sent another request, or ctx is done.

type Policy

type Policy struct {
	// MaxAttempts is the total number of attempts, including the first.
	// Default 3.
	MaxAttempts int
	Backoff     Backoff
	// MaxRetryAfter caps how long a server's Retry-After is honored.
	// Default 60s.
	MaxRetryAfter time.Duration
}

Policy decides whether an attempt is retried and how long to wait first.

func (Policy) Attempts

func (p Policy) Attempts() int

Attempts returns MaxAttempts, or its default.

func (Policy) ShouldRetry

func (p Policy) ShouldRetry(status int, outcome detect.Outcome, err error) bool

ShouldRetry reports whether an attempt that ended this way is worth repeating: on a network error (but not a canceled context), or a 429 or 5xx that was not detected as a block. A block or captcha is never retried: the site has said no, and asking again is not the answer. A 404 is never retried either.

func (Policy) Wait

func (p Policy) Wait(attempt int, h http.Header, now time.Time) time.Duration

Wait returns how long to wait before retry number attempt (from 0): the backoff delay, or the server's Retry-After when that is longer, capped at MaxRetryAfter.

type Robots

type Robots struct {
	// contains filtered or unexported fields
}

Robots answers whether robots.txt allows a URL, fetching each site's file once and caching it.

A missing robots.txt (any 4xx) allows everything; a 5xx disallows everything until the entry expires, following Google's reading of the standard. A file that cannot be parsed counts as missing. A network error is returned and not cached.

func NewRobots

func NewRobots(f fetch.Fetcher, ttl time.Duration) *Robots

NewRobots returns a Robots that fetches through f and keeps each file for ttl (24 hours when ttl is zero or negative).

func (*Robots) Allowed

func (r *Robots) Allowed(ctx context.Context, target, agent string, base fetch.Request) (bool, error)

Allowed reports whether agent may fetch target. The robots.txt request is built from base, so it goes out through the same proxy, profile and cookie jar as the page it guards.