ctxaws

README

ctxaws - Context-Aware AWS SDK Operations

When using the net/context package, one should respect context expiration when performing requests to backing services. Typically, a context.Context is created for every incoming request and all calls to other services that happen during this request have to be cancelled when the request's context expires.

The ctxaws package includes a helper function ctxaws.InContext that performs AWS API requests while respecting a context's cancellation.

Support for the context pattern in the SDK directly was discussed but not implemented.

Usage

In Go projects using the AWS SDK, requests or most typically performed like this:

func countItems() (int, error) {
    out, err := client.Scan(&dynamodb.ScanInput{
        TableName: aws.String("my-table"),
    })
    if err != nil {
        return 0, err
    }
    return len(out.Items), nil
}

DynamoDB is just used as an example here.

The Problem with such SDK operations is that they lead to HTTP requests that cannot be cancelled and are retried for a (configurable) number of times until they succeed. In most cases, these retries are desired behavior, however it cannot be controlled how long the whole operation takes.

In order to make the same API call while respecting a context's cancellation, one can use the ctxaws package like this:

func countItems(ctx context.Context) (int, error) {
    req, out := client.ScanRequest(&dynamodb.ScanInput{
        TableName: aws.String("my-table"),
    })
    if err := ctxaws.InContext(ctx, req); err != nil {
        return 0, err
    }
    return len(out.Items), nil
}

By using the special *Request methods of the AWS service clients, we can delegate the processing of the request to ctxhttp which makes sure that the context cancellation is respected while performing the request.

Besides cancelling requests that exceed the context's deadline, ctxaws also uses a custom retry mechanism that stops retrying as soon as the context expires. This way, the context ist honored during HTTP requests and during backoff periods.

Pagination

When using SDK operations that return a list of resources, it is almost always a good idea to use the SDKs *Pages functions. With this in mind, the example from above becomes slightly more complex and could be implemented like this:

func countItems() (int, error) {
    count := 0
    err := client.ScanPages(&dynamodb.ScanInput{
        TableName: aws.String("my-table"),
    }, func(out *dynamodb.ScanOutput, last bool) bool {
        count += len(out.Items)
        return !last
    })
    if err != nil {
        return 0, err
    }
    return count, nil
}

With a context, one can use PaginageInContext which works very similar to the *Pages methods. The only difference is the type of the passed closure's first argument. Instead of a reqular *Output type, the closure takes an interface{} which can safely be cased to the respective *Output type (*dynamodb.ScanOutput in the example):

func countItems(ctx context.Context) (int, error) {
    count := 0
    req, _ := client.ScanRequest(&dynamodb.ScanInput{
        TableName: aws.String("my-table"),
    })
    err := ctxaws.PaginateInContext(ctx, req, func(out interface{}, last bool) bool {
        count += len(out.(*dynamodb.ScanOutput).Items)
        return !last
    })
    if err != nil {
        return 0, err
    }
    return count, nil
}

Development

There is a cmd/main package with an example program that can be used for testing. Please note that it requires a DynamoDB table named test-table that needs to be created before running it. When run it uses the AWS credentials exported on the command line to connect to DynamoDB.

$ go run cmd/main/main.go

Besides that, there are a couple of tests that are run automatically by Travis. To run those tests locally, execute

$ go test

Documentation

Index

• Variables
• func InContext(ctx context.Context, req *request.Request) error
• func PaginateInContext(ctx context.Context, req *request.Request, ...) error
• type Retryer
  • func NewContextAwareRetryer(ctx context.Context) *Retryer
  • func (r *Retryer) ShouldRetry(req *request.Request) bool

Variables

var ErrDeadlineWouldExceedBeforeRetry = errors.New("deadline would exceed before next retry")

ErrDeadlineWouldExceedBeforeRetry is set as a request's error when a retry is cancelled because the backoff interval would exceed the context's deadline.

Functions

func InContext

func InContext(ctx context.Context, req *request.Request) error

InContext performs an AWS request and takes care that it does not exceed the context's deadline. It replaces the request's send handler with an adapted version. The original was taken from https://github.com/aws/aws-sdk-go/blob/7761b2cb776ce24b934a6cf24d9f8dc018563abb/aws/corehandlers/handlers.go#L68-L108 The customization is that instead of using the request's HTTP client directly to perform the HTTP request, the send handler uses the x/net/context/ctxhttp package to do this while respecting the context's deadline.

func PaginateInContext

func PaginateInContext(ctx context.Context, req *request.Request, handlePage func(interface{}, bool) bool) error

Types

type Retryer

type Retryer struct {
	client.DefaultRetryer
	// contains filtered or unexported fields
}

Retryer is an implementation of `request.Retryer` that honors its context's deadline. While the delay mechanism is exactly the same as in the `client.DefaultRetryer`, it instructs its clients not to retry requests when the context would expire while waiting for the next try. Also requests won't be retried when the context already has expired.

func NewContextAwareRetryer

func NewContextAwareRetryer(ctx context.Context) *Retryer

NewContextAwareRetryer creates a Retryer that honors the given context's deadline.

func (*Retryer) ShouldRetry

func (r *Retryer) ShouldRetry(req *request.Request) bool

ShouldRetry compares the next retry time with the context's deadline and returns false if the deadline would occur prior to the next try. If not, the call is passed through to the default retryer.