s3file

Documentation

Overview

Package s3file implements grail file interface for S3.

Index

• Variables
• func GetBucketRegion(ctx context.Context, client s3iface.S3API, bucket string) (string, error)
• func InvalidateBucketRegion(bucket string)
• func NewImplementation(provider ClientProvider, opts Options) file.Implementation
• func ParseURL(url string) (scheme, bucket, key string, err error)
• func SetBucketRegion(bucket, region string)
• type ClientProvider
  • func NewDefaultProvider(opts session.Options) ClientProvider
• type Options

Examples

• ParseURL

Variables

var (
	// BackoffPolicy defines backoff timing parameters. It is exposed publicly only
	// for unittests.
	// TODO(josh): Rename to `RetryPolicy`.
	// TODO(josh): Create `retry.ThrottlePolicy` and `retry.AIMDPolicy` and use here.
	BackoffPolicy = retry.Jitter(retry.Backoff(500*time.Millisecond, time.Minute, 1.2), 0.2)

	// MaxRetryDuration defines the max amount of time a request can spend
	// retrying on errors.
	//
	// Requirements:
	//
	// - The value must be >5 minutes. 5 min is the S3 negative-cache TTL.  If
	//   less than 5 minutes, an Open() call w/ RetryWhenNotFound may fail.
	//
	// - It must be long enough to allow CompleteMultiPartUpload to finish after a
	//   retry. The doc says it may take a few minutes even in a successful case.
	MaxRetryDuration = 60 * time.Minute
)

var UploadPartSize = 16 << 20

UploadPartSize is the size of a chunk during multi-part uploads. It is exposed only for unittests.

Functions

func GetBucketRegion

func GetBucketRegion(ctx context.Context, client s3iface.S3API, bucket string) (string, error)

GetBucketRegion finds the AWS region for the S3 bucket and inserts it in the cache. "client" is used to issue the GetBucketRegion S3 call. It doesn't need to be in the region for the "bucket".

func InvalidateBucketRegion

func InvalidateBucketRegion(bucket string)

InvalidateBucketRegion removes the cache entry for bucket, if it exists.

func NewImplementation

func NewImplementation(provider ClientProvider, opts Options) file.Implementation

NewImplementation creates a new file.Implementation for S3. The provider is called to create s3 client objects.

func ParseURL

func ParseURL(url string) (scheme, bucket, key string, err error)

ParseURL parses a path of form "s3://grail-bucket/dir/file" and returns ("s3", "grail-bucket", "dir/file", nil).

Example

package main

import (
	"fmt"

	"github.com/grailbio/base/file/s3file"
)

func main() {
	scheme, bucket, key, err := s3file.ParseURL("s3://grail-bucket/dir/file")
	fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
	scheme, bucket, key, err = s3file.ParseURL("s3://grail-bucket/dir/")
	fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
	scheme, bucket, key, err = s3file.ParseURL("s3://grail-bucket")
	fmt.Printf("scheme: %s, bucket: %s, key: %s, err: %v\n", scheme, bucket, key, err)
}

Output:

scheme: s3, bucket: grail-bucket, key: dir/file, err: <nil>
scheme: s3, bucket: grail-bucket, key: dir/, err: <nil>
scheme: s3, bucket: grail-bucket, key: , err: <nil>

func SetBucketRegion

func SetBucketRegion(bucket, region string)

SetBucketRegion sets a bucket's region, overriding region discovery and defaults.

Types

type ClientProvider

type ClientProvider interface {
	// Get returns S3 clients that can be used to perform "op" on "path".
	//
	// "op" is an S3 operation name, without the "s3:" prefix; for example
	// "PutObject" or "ListBucket". The full list of operations is defined in
	// https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
	//
	// Path is a full URL of form "s3://bucket/key".  This method may be called
	// concurrently from multiple threads.
	//
	// Usually Get() returns one S3 client object on success. If it returns
	// multiple clients, the s3 file implementation will try each client in order,
	// until the operation succeeds.
	//
	// REQUIRES: Get returns either >=1 clients, or a non-nil error.
	Get(ctx context.Context, op, path string) ([]s3iface.S3API, error)

	// NotifyResult is called to inform that using "client" to perform "op" on
	// "path" resulted in the given error (err is nil if the op succeeded). The
	// provider should use it to optimize the list of clients to return in Get in
	// a future.
	//
	// Parameter "client" is one of the clients returned by the Get call.
	NotifyResult(ctx context.Context, op, path string, client s3iface.S3API, err error)
}

ClientProvider is responsible for creating an S3 client object. Get() is called whenever s3File needs to access a file. The provider should cache and reuse the client objects, if needed. The implementation must be thread safe.

func NewDefaultProvider

func NewDefaultProvider(opts session.Options) ClientProvider

NewDefaultProvider creates a trivial ClientProvider that uses AWS session.NewSession() (https://docs.aws.amazon.com/sdk-for-go/api/aws/session/).

opts is passed to NewSession. The exception is opts.Config.Region, which will be be overwritten to point to the actual bucket location.

type Options

type Options struct {
	// ServerSideEncryption allows you to set the `ServerSideEncryption` value to use when
	// uploading files (e.g.  "AES256")
	ServerSideEncryption string
}

Options defines options that can be given when creating an s3Impl