retry

README

Retry

Retry is a Go library for facilitating retry logic and backoff. It's highly extensible with full control over how and when retries occur. You can also write your own custom backoff functions by implementing the Backoff interface.

Features

• Extensible - Inspired by Go's built-in HTTP package, this Go backoff and retry library is extensible via middleware. You can write custom backoff functions or use a provided filter.

• Independent - No external dependencies besides the Go standard library, meaning it won't bloat your project.

• Concurrent - Unless otherwise specified, everything is safe for concurrent use.

• Context-aware - Use native Go contexts to control cancellation.

Usage

Here is an example use for connecting to a database using Go's database/sql package:

package main

import (
  "context"
  "database/sql"
  "log"
  "time"

  "github.com/sethvargo/go-retry"
)

func main() {
  db, err := sql.Open("mysql", "...")
  if err != nil {
    log.Fatal(err)
  }

  ctx := context.Background()
  if err := retry.Fibonacci(ctx, 1*time.Second, func(ctx context.Context) error {
    if err := db.PingContext(ctx); err != nil {
      // This marks the error as retryable
      return retry.RetryableError(err)
    }
    return nil
  }); err != nil {
    log.Fatal(err)
  }
}

Backoffs

In addition to your own custom algorithms, there are built-in algorithms for backoff in the library.

Constant

A very rudimentary backoff, just returns a constant value. Here is an example:

1s -> 1s -> 1s -> 1s -> 1s -> 1s

Usage:

NewConstant(1 * time.Second)

Exponential

Arguably the most common backoff, the next value is double the previous value. Here is an example:

1s -> 2s -> 4s -> 8s -> 16s -> 32s -> 64s

Usage:

NewExponential(1 * time.Second)

Fibonacci

The Fibonacci backoff uses the Fibonacci sequence to calculate the backoff. The next value is the sum of the current value and the previous value. This means retires happen quickly at first, but then gradually take slower, ideal for network-type issues. Here is an example:

1s -> 1s -> 2s -> 3s -> 5s -> 8s -> 13s

Usage:

NewFibonacci(1 * time.Second)

Modifiers (Middleware)

The built-in backoff algorithms never terminate and have no caps or limits - you control their behavior with middleware. There's built-in middleware, but you can also write custom middleware.

Jitter

To reduce the changes of a thundering herd, add random jitter to the returned value.

b := NewFibonacci(1 * time.Second)

// Return the next value, +/- 500ms
b = WithJitter(500*time.Millisecond, b)

// Return the next value, +/- 5% of the result
b = WithJitterPercent(5, b)

MaxRetries

To terminate a retry, specify the maximum number of retries. Note this is retries, not attempts. Attempts is retries + 1.

b := NewFibonacci(1 * time.Second)

// Stop after 4 retries, when the 5th attempt has failed. In this example, the worst case elapsed
// time would be 1s + 1s + 2s + 3s = 7s.
b = WithMaxRetries(4, b)

CappedDuration

To ensure an individual calculated duration never exceeds a value, use a cap:

b := NewFibonacci(1 * time.Second)

// Ensure the maximum value is 2s. In this example, the sleep values would be
// 1s, 1s, 2s, 2s, 2s, 2s...
b = WithCappedDuration(2 * time.Second, b)

WithMaxDuration

For a best-effort limit on the total execution time, specify a max duration:

b := NewFibonacci(1 * time.Second)

// Ensure the maximum total retry time is 5s.
b = WithMaxDuration(5 * time.Second, b)

Benchmarks

Here are benchmarks against some other popular Go backoff and retry libraries. You can run these benchmarks yourself via the benchmark/ folder. Commas and spacing fixed for clarity.

Benchmark/cenkalti-7      13,052,668     87.3 ns/op
Benchmark/lestrrat-7         902,044    1,355 ns/op
Benchmark/sethvargo-7    203,914,245     5.73 ns/op

Notes and Caveats

• Randomization uses math/rand seeded with the Unix timestamp instead of crypto/rand.
• Ordering of addition of multiple modifiers will make a difference. For example; ensure you add CappedDuration before WithMaxDuration, otherwise it may early out too early. Another example is you could add Jitter before or after capping depending on your desired outcome.

Documentation

Overview

Package retry provides helpers for retrying.

This package defines flexible interfaces for retrying Go functions that may be flakey or eventually consistent. It abstracts the "backoff" (how long to wait between tries) and "retry" (execute the function again) mechanisms for maximum flexibility. Furthermore, everything is an interface, so you can define your own implementations.

The package is modeled after Go's built-in HTTP package, making it easy to customize the built-in backoff with your own custom logic. Additionally, callers specify which errors are retryable by wrapping them. This is helpful with complex operations where only certain results should retry.

Index

• func Constant(ctx context.Context, t time.Duration, f RetryFunc) error
• func Do(ctx context.Context, b Backoff, f RetryFunc) error
• func Exponential(ctx context.Context, base time.Duration, f RetryFunc) error
• func Fibonacci(ctx context.Context, base time.Duration, f RetryFunc) error
• func RetryableError(err error) error
• type Backoff
  • func NewConstant(t time.Duration) Backoff
  • func NewExponential(base time.Duration) Backoff
  • func NewFibonacci(base time.Duration) Backoff
  • func WithCappedDuration(cap time.Duration, next Backoff) Backoff
  • func WithJitter(j time.Duration, next Backoff) Backoff
  • func WithJitterPercent(j uint64, next Backoff) Backoff
  • func WithMaxDuration(timeout time.Duration, next Backoff) Backoff
  • func WithMaxRetries(max uint64, next Backoff) Backoff
• type BackoffFunc
  • func (b BackoffFunc) Next() (time.Duration, bool)
• type RetryFunc

Examples

• BackoffFunc
• Do (CustomRetry)
• Do (Simple)
• NewConstant
• NewExponential
• NewFibonacci
• WithCappedDuration
• WithJitter
• WithJitterPercent
• WithMaxDuration
• WithMaxRetries

Functions

func Constant

func Constant(ctx context.Context, t time.Duration, f RetryFunc) error

Constant is a wrapper around Retry that uses a constant backoff. It panics if the given base is less than zero.

func Do

func Do(ctx context.Context, b Backoff, f RetryFunc) error

Do wraps a function with a backoff to retry. The provided context is the same context passed to the RetryFunc.

Example (CustomRetry)

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Nanosecond)

// This example demonstrates selectively retrying specific errors. Only errors
// wrapped with RetryableError are eligible to be retried.
if err := retry.Do(ctx, retry.WithMaxRetries(3, b), func(ctx context.Context) error {
	resp, err := http.Get("https://google.com/")
	if err != nil {
		return err
	}
	defer resp.Body.Close()

	switch resp.StatusCode / 100 {
	case 4:
		return fmt.Errorf("bad response: %v", resp.StatusCode)
	case 5:
		return retry.RetryableError(fmt.Errorf("bad response: %v", resp.StatusCode))
	default:
		return nil
	}
}); err != nil {
	// handle error
}

Example (Simple)

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Nanosecond)

i := 0
if err := retry.Do(ctx, retry.WithMaxRetries(3, b), func(ctx context.Context) error {
	fmt.Printf("%d\n", i)
	i++
	return retry.RetryableError(fmt.Errorf("oops"))
}); err != nil {
	// handle error
}

Output:

0
1
2
3

func Exponential

func Exponential(ctx context.Context, base time.Duration, f RetryFunc) error

Exponential is a wrapper around Retry that uses an exponential backoff. See NewExponential.

func Fibonacci

func Fibonacci(ctx context.Context, base time.Duration, f RetryFunc) error

Fibonacci is a wrapper around Retry that uses a Fibonacci backoff. See NewFibonacci.

func RetryableError

func RetryableError(err error) error

RetryableError marks an error as retryable.

Types

type Backoff

type Backoff interface {
	// Next returns the time duration to wait and whether to stop.
	Next() (next time.Duration, stop bool)
}

Backoff is an interface that backs off.

func NewConstant

func NewConstant(t time.Duration) Backoff

NewConstant creates a new constant backoff using the value t. The wait time is the provided constant value. It panics if the given base is less than zero.

Example

b := retry.NewConstant(1 * time.Second)

for i := 0; i < 5; i++ {
	val, _ := b.Next()
	fmt.Printf("%v\n", val)
}

Output:

1s
1s
1s
1s
1s

func NewExponential

func NewExponential(base time.Duration) Backoff

NewExponential creates a new exponential backoff using the starting value of base and doubling on each failure (1, 2, 4, 8, 16, 32, 64...), up to max.

Once it overflows, the function constantly returns the maximum time.Duration for a 64-bit integer.

It panics if the given base is less than zero.

Example

b := retry.NewExponential(1 * time.Second)

for i := 0; i < 5; i++ {
	val, _ := b.Next()
	fmt.Printf("%v\n", val)
}

Output:

1s
2s
4s
8s
16s

func NewFibonacci

func NewFibonacci(base time.Duration) Backoff

NewFibonacci creates a new Fibonacci backoff using the starting value of base. The wait time is the sum of the previous two wait times on each failed attempt (1, 1, 2, 3, 5, 8, 13...).

Once it overflows, the function constantly returns the maximum time.Duration for a 64-bit integer.

It panics if the given base is less than zero.

Example

b := retry.NewFibonacci(1 * time.Second)

for i := 0; i < 5; i++ {
	val, _ := b.Next()
	fmt.Printf("%v\n", val)
}

Output:

1s
2s
3s
5s
8s

func WithCappedDuration

func WithCappedDuration(cap time.Duration, next Backoff) Backoff

WithCappedDuration sets a maximum on the duration returned from the next backoff. This is NOT a total backoff time, but rather a cap on the maximum value a backoff can return. Without another middleware, the backoff will continue infinitely.

Example

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Second)
b = retry.WithCappedDuration(3*time.Second, b)

if err := retry.Do(ctx, b, func(_ context.Context) error {
	// TODO: logic here
	return nil
}); err != nil {
	// handle error
}

func WithJitter

func WithJitter(j time.Duration, next Backoff) Backoff

WithJitter wraps a backoff function and adds the specified jitter. j can be interpreted as "+/- j". For example, if j were 5 seconds and the backoff returned 20s, the value could be between 15 and 25 seconds. The value can never be less than 0.

Example

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Second)
b = retry.WithJitter(1*time.Second, b)

if err := retry.Do(ctx, b, func(_ context.Context) error {
	// TODO: logic here
	return nil
}); err != nil {
	// handle error
}

func WithJitterPercent

func WithJitterPercent(j uint64, next Backoff) Backoff

WithJitterPercent wraps a backoff function and adds the specified jitter percentage. j can be interpreted as "+/- j%". For example, if j were 5 and the backoff returned 20s, the value could be between 19 and 21 seconds. The value can never be less than 0 or greater than 100.

Example

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Second)
b = retry.WithJitterPercent(5, b)

if err := retry.Do(ctx, b, func(_ context.Context) error {
	// TODO: logic here
	return nil
}); err != nil {
	// handle error
}

func WithMaxDuration

func WithMaxDuration(timeout time.Duration, next Backoff) Backoff

WithMaxDuration sets a maximum on the total amount of time a backoff should execute. It's best-effort, and should not be used to guarantee an exact amount of time.

Example

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Second)
b = retry.WithMaxDuration(5*time.Second, b)

if err := retry.Do(ctx, b, func(_ context.Context) error {
	// TODO: logic here
	return nil
}); err != nil {
	// handle error
}

func WithMaxRetries

func WithMaxRetries(max uint64, next Backoff) Backoff

WithMaxRetries executes the backoff function up until the maximum attempts.

Example

ctx := context.Background()

b := retry.NewFibonacci(1 * time.Second)
b = retry.WithMaxRetries(3, b)

if err := retry.Do(ctx, b, func(_ context.Context) error {
	// TODO: logic here
	return nil
}); err != nil {
	// handle error
}

type BackoffFunc

type BackoffFunc func() (time.Duration, bool)

BackoffFunc is a backoff expressed as a function.

Example

ctx := context.Background()

// Example backoff middleware that adds the provided duration t to the result.
withShift := func(t time.Duration, next retry.Backoff) retry.BackoffFunc {
	return func() (time.Duration, bool) {
		val, stop := next.Next()
		if stop {
			return 0, true
		}
		return val + t, false
	}
}

// Middlewrap wrap another backoff:
b := retry.NewFibonacci(1 * time.Second)
b = withShift(5*time.Second, b)

if err := retry.Do(ctx, b, func(ctx context.Context) error {
	// Actual retry logic here
	return nil
}); err != nil {
	// handle error
}

func (BackoffFunc) Next

func (b BackoffFunc) Next() (time.Duration, bool)

Next implements Backoff.

type RetryFunc

type RetryFunc func(ctx context.Context) error

RetryFunc is a function passed to retry.