Documentation

Overview

    Package s3file implements grail file interface for S3.

    Index

    Examples

    Constants

    This section is empty.

    Variables

    View Source
    var (
    	// BackoffPolicy defines backoff timing parameters. It is exposed publicly only
    	// for unittests.
    	// TODO(josh): Rename to `RetryPolicy`.
    	// TODO(josh): Create `retry.ThrottlePolicy` and `retry.AIMDPolicy` and use here.
    	BackoffPolicy = retry.Jitter(retry.Backoff(500*time.Millisecond, time.Minute, 1.2), 0.2)
    
    	// MaxRetryDuration defines the max amount of time a request can spend
    	// retrying on errors.
    	//
    	// Requirements:
    	//
    	// - The value must be >5 minutes. 5 min is the S3 negative-cache TTL.  If
    	//   less than 5 minutes, an Open() call w/ RetryWhenNotFound may fail.
    	//
    	// - It must be long enough to allow CompleteMultiPartUpload to finish after a
    	//   retry. The doc says it may take a few minutes even in a successful case.
    	MaxRetryDuration = 60 * time.Minute
    )
    View Source
    var UploadPartSize = 16 << 20

      UploadPartSize is the size of a chunk during multi-part uploads. It is exposed only for unittests.

      Functions

      func GetBucketRegion

      func GetBucketRegion(ctx context.Context, client s3iface.S3API, bucket string) (string, error)

        GetBucketRegion finds the AWS region for the S3 bucket and inserts it in the cache. "client" is used to issue the GetBucketRegion S3 call. It doesn't need to be in the region for the "bucket".

        func InvalidateBucketRegion

        func InvalidateBucketRegion(bucket string)

          InvalidateBucketRegion removes the cache entry for bucket, if it exists.

          func NewImplementation

          func NewImplementation(provider ClientProvider, opts Options) file.Implementation

            NewImplementation creates a new file.Implementation for S3. The provider is called to create s3 client objects.

            func ParseURL

            func ParseURL(url string) (scheme, bucket, key string, err error)

              ParseURL parses a path of form "s3://grail-bucket/dir/file" and returns ("s3", "grail-bucket", "dir/file", nil).

              Example
              Output:
              
              scheme: s3, bucket: grail-bucket, key: dir/file, err: <nil>
              scheme: s3, bucket: grail-bucket, key: dir/, err: <nil>
              scheme: s3, bucket: grail-bucket, key: , err: <nil>
              

              func SetBucketRegion

              func SetBucketRegion(bucket, region string)

                SetBucketRegion sets a bucket's region, overriding region discovery and defaults.

                Types

                type ClientProvider

                type ClientProvider interface {
                	// Get returns S3 clients that can be used to perform "op" on "path".
                	//
                	// "op" is an S3 operation name, without the "s3:" prefix; for example
                	// "PutObject" or "ListBucket". The full list of operations is defined in
                	// https://docs.aws.amazon.com/AmazonS3/latest/dev/using-with-s3-actions.html
                	//
                	// Path is a full URL of form "s3://bucket/key".  This method may be called
                	// concurrently from multiple threads.
                	//
                	// Usually Get() returns one S3 client object on success. If it returns
                	// multiple clients, the s3 file implementation will try each client in order,
                	// until the operation succeeds.
                	//
                	// REQUIRES: Get returns either >=1 clients, or a non-nil error.
                	Get(ctx context.Context, op, path string) ([]s3iface.S3API, error)
                
                	// NotifyResult is called to inform that using "client" to perform "op" on
                	// "path" resulted in the given error (err is nil if the op succeeded). The
                	// provider should use it to optimize the list of clients to return in Get in
                	// a future.
                	//
                	// Parameter "client" is one of the clients returned by the Get call.
                	NotifyResult(ctx context.Context, op, path string, client s3iface.S3API, err error)
                }

                  ClientProvider is responsible for creating an S3 client object. Get() is called whenever s3File needs to access a file. The provider should cache and reuse the client objects, if needed. The implementation must be thread safe.

                  func NewDefaultProvider

                  func NewDefaultProvider(opts session.Options) ClientProvider

                    NewDefaultProvider creates a trivial ClientProvider that uses AWS session.NewSession() (https://docs.aws.amazon.com/sdk-for-go/api/aws/session/).

                    opts is passed to NewSession. The exception is opts.Config.Region, which will be be overwritten to point to the actual bucket location.

                    type Options

                    type Options struct {
                    	// ServerSideEncryption allows you to set the `ServerSideEncryption` value to use when
                    	// uploading files (e.g.  "AES256")
                    	ServerSideEncryption string
                    }

                      Options defines options that can be given when creating an s3Impl

                      Directories

                      Path Synopsis
                      internal
                      cmd/resolvetest
                      resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host.
                      resolvetest simply resolves a hostname at an increasing time interval to observe the diversity in DNS lookup addresses for the host.