s3

package
v5.10.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Sep 16, 2021 License: MIT Imports: 27 Imported by: 0

Documentation

Overview

Package s3 AWS S3 VFS implementation.

Usage

Rely on github.com/c2fo/vfs/v5/backend 

import(
    "github.com/c2fo/vfs/v5/backend"
    "github.com/c2fo/vfs/v5/backend/s3"
)

func UseFs() error {
    fs := backend.Backend(s3.Scheme)
    ...
}

Or call directly: 

import "github.com/c2fo/vfs/v5/backend/s3"

func DoSomething() {
    fs := s3.NewFileSystem()
    ...
}

s3 can be augmented with the following implementation-specific methods. Backend returns vfs.FileSystem interface so it would have to be cast as s3.FileSystem to use the following: 

func DoSomething() {

    ...

    // cast if fs was created using backend.Backend().  Not necessary if created directly from s3.NewFileSystem().
    fs = fs.(s3.FileSystem)

    // to pass in client options
    fs = fs.WithOptions(
        s3.Options{
            AccessKeyID:     "AKIAIOSFODNN7EXAMPLE",
            SecretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
            Region:          "us-west-2",
            ACL: 			   "bucket-owner-full-control",
        },
    )

    // to pass specific client, for instance a mock client
    s3apiMock := &mocks.S3API{}
    s3apiMock.On("GetObject", mock.AnythingOfType("*s3.GetObjectInput")).
        Return(&s3.GetObjectOutput{
            Body: nopCloser{bytes.NewBufferString("Hello world!")},
            }, nil)
    fs = fs.WithClient(s3apiMock)
}

Object ACL

Canned ACL's can be passed in as an Option. This string will be applied to all writes, moves, and copies. See https://docs.aws.amazon.com/AmazonS3/latest/dev/acl-overview.html#canned-acl for values.

Authentication

Authentication, by default, occurs automatically when Client() is called. It looks for credentials in the following places, preferring the first location found:

  1. StaticProvider - set of credentials which are set programmatically, and will never expire.

  2. EnvProvider - credentials from the environment variables of the running process. Environment credentials never expire. Environment variables used:

    * Access Key ID: AWS_ACCESS_KEY_ID or AWS_ACCESS_KEY * Secret Access Key: AWS_SECRET_ACCESS_KEY or AWS_SECRET_KEY

  3. SharedCredentialsProvider - looks for "AWS_SHARED_CREDENTIALS_FILE" env variable. If the env value is empty will default to current user's home directory.

    * Linux/OSX: "$HOME/.aws/credentials" * Windows: "%USERPROFILE%\.aws\credentials"

  4. RemoteCredProvider - default remote endpoints such as EC2 or ECS IAM Roles

  5. EC2RoleProvider - credentials from the EC2 service, and keeps track if those credentials are expired

See the following for more auth info: https://docs.aws.amazon.com/cli/latest/userguide/cli-configure-envvars.html and https://docs.aws.amazon.com/sdk-for-go/v1/developer-guide/configuring-sdk.html

See Also

See: https://github.com/aws/aws-sdk-go/tree/master/service/s3

Index

Constants

View Source 
const Scheme = "s3"

Scheme defines the file system type.

Variables

This section is empty.

Functions

This section is empty.

Types

type Downloader added in v5.10.0 

type Downloader interface {
	Download(w io.WriterAt, input *s3.GetObjectInput, options ...func(downloader *s3manager.Downloader)) (n int64, err error)

	DownloadWithContext(ctx aws.Context, w io.WriterAt, input *s3.GetObjectInput, options ...func(*s3manager.Downloader)) (n int64, err error)

	DownloadWithIterator(ctx aws.Context, iter s3manager.BatchDownloadIterator, opts ...func(*s3manager.Downloader)) error
}

Downloader interface needed to mock S3 Downloader data access object in tests

type File 

type File struct {
	// contains filtered or unexported fields
}

File implements vfs.File interface for S3 fs.

func (*File) Close 

func (f *File) Close() error

Close cleans up underlying mechanisms for reading from and writing to the file. Closes and removes the local temp file, and triggers a write to s3 of anything in the f.writeBuffer if it has been created.

func (*File) CopyToFile 

func (f *File) CopyToFile(file vfs.File) error

CopyToFile puts the contents of File into the targetFile passed. Uses the S3 CopyObject method if the target file is also on S3, otherwise uses io.CopyBuffer.

func (*File) CopyToLocation 

func (f *File) CopyToLocation(location vfs.Location) (vfs.File, error)

CopyToLocation creates a copy of *File, using the file's current name as the new file's name at the given location. If the given location is also s3, the AWS API for copying files will be utilized, otherwise, standard io.Copy will be done to the new file.

func (*File) Delete 

func (f *File) Delete() error

Delete clears any local temp file, or write buffer from read/writes to the file, then makes a DeleteObject call to s3 for the file. Returns any error returned by the API.

func (*File) Exists 

func (f *File) Exists() (bool, error)

Exists returns a boolean of whether or not the object exists on s3, based on a call for the object's HEAD through the s3 API.

func (*File) LastModified 

func (f *File) LastModified() (*time.Time, error)

LastModified returns the LastModified property of a HEAD request to the s3 object.

func (*File) Location 

func (f *File) Location() vfs.Location

Location returns a vfs.Location at the location of the object. IE: if file is at s3://bucket/here/is/the/file.txt the location points to s3://bucket/here/is/the/

func (*File) MoveToFile 

func (f *File) MoveToFile(file vfs.File) error

MoveToFile puts the contents of File into the targetFile passed using File.CopyToFile. If the copy succeeds, the source file is deleted. Any errors from the copy or delete are returned.

func (*File) MoveToLocation 

func (f *File) MoveToLocation(location vfs.Location) (vfs.File, error)

MoveToLocation works by first calling File.CopyToLocation(vfs.Location) then, if that succeeds, it deletes the original file, returning the new file. If the copy process fails the error is returned, and the Delete isn't called. If the call to Delete fails, the error and the file generated by the copy are both returned.

func (*File) Name 

func (f *File) Name() string

Name returns the name portion of the file's key property. IE: "file.txt" of "s3://some/path/to/file.txt

func (*File) Path 

func (f *File) Path() string

Path return the directory portion of the file's key. IE: "path/to" of "s3://some/path/to/file.txt

func (*File) Read 

func (f *File) Read(p []byte) (n int, err error)

Read implements the standard for io.Reader. For this to work with an s3 file, a temporary local copy of the file is created, and reads work on that. This file is closed and removed upon calling f.Close()

func (*File) Seek 

func (f *File) Seek(offset int64, whence int) (int64, error)

Seek implements the standard for io.Seeker. A temporary local copy of the s3 file is created (the same one used for Reads) which Seek() acts on. This file is closed and removed upon calling f.Close()

func (*File) Size 

func (f *File) Size() (uint64, error)

Size returns the ContentLength value from an s3 HEAD request on the file's object.

func (*File) String 

func (f *File) String() string

String implement fmt.Stringer, returning the file's URI as the default string.

func (*File) Touch 

func (f *File) Touch() error

Touch creates a zero-length file on the vfs.File if no File exists. Update File's last modified timestamp. Returns error if unable to touch File.

func (*File) URI 

func (f *File) URI() string

URI returns the File's URI as a string.

func (*File) Write 

func (f *File) Write(data []byte) (res int, err error)

Write implements the standard for io.Writer. A buffer is added to with each subsequent write. When f.Close() is called, the contents of the buffer are used to initiate the PutObject to s3. The underlying implementation uses s3manager which will determine whether it is appropriate to call PutObject, or initiate a multi-part upload.

type FileSystem 

type FileSystem struct {
	// contains filtered or unexported fields
}

FileSystem implements vfs.FileSystem for the S3 file system.

func NewFileSystem 

func NewFileSystem() *FileSystem

NewFileSystem initializer for FileSystem struct accepts aws-sdk s3iface.S3API client and returns Filesystem or error.

func (*FileSystem) Client 

func (fs *FileSystem) Client() (s3iface.S3API, error)

Client returns the underlying aws s3 client, creating it, if necessary See Overview for authentication resolution

func (*FileSystem) Name 

func (fs *FileSystem) Name() string

Name returns "AWS S3"

func (*FileSystem) NewFile 

func (fs *FileSystem) NewFile(volume, name string) (vfs.File, error)

NewFile function returns the s3 implementation of vfs.File.

func (*FileSystem) NewLocation 

func (fs *FileSystem) NewLocation(volume, name string) (vfs.Location, error)

NewLocation function returns the s3 implementation of vfs.Location.

func (*FileSystem) Retry 

func (fs *FileSystem) Retry() vfs.Retry

Retry will return the default no-op retrier. The S3 client provides its own retryer interface, and is available to override via the s3.FileSystem Options type.

func (*FileSystem) Scheme 

func (fs *FileSystem) Scheme() string

Scheme return "s3" as the initial part of a file URI ie: s3://

func (*FileSystem) WithClient 

func (fs *FileSystem) WithClient(client interface{}) *FileSystem

WithClient passes in an s3 client and returns the file system (chainable)

func (*FileSystem) WithOptions 

func (fs *FileSystem) WithOptions(opts vfs.Options) *FileSystem

WithOptions sets options for client and returns the file system (chainable)

type Location 

type Location struct {
	// contains filtered or unexported fields
}

Location implements the vfs.Location interface specific to S3 fs.

func (*Location) ChangeDir 

func (l *Location) ChangeDir(relativePath string) error

ChangeDir takes a relative path, and modifies the underlying Location's path. The caller is modified by this so the only return is any error. For this implementation there are no errors.

func (*Location) DeleteFile 

func (l *Location) DeleteFile(fileName string) error

DeleteFile removes the file at fileName path.

func (*Location) Exists 

func (l *Location) Exists() (bool, error)

Exists returns true if the bucket exists, and the user in the underlying s3.fileSystem.Client() has the appropriate permissions. Will receive false without an error if the bucket simply doesn't exist. Otherwise could receive false and any errors passed back from the API.

func (*Location) FileSystem 

func (l *Location) FileSystem() vfs.FileSystem

FileSystem returns a vfs.FileSystem interface of the location's underlying file system.

func (*Location) List 

func (l *Location) List() ([]string, error)

List calls the s3 API to list all objects in the location's bucket, with a prefix automatically set to the location's path. This will make a call to the s3 API for every 1000 keys to return. If you have many thousands of keys at the given location, this could become quite expensive.

func (*Location) ListByPrefix 

func (l *Location) ListByPrefix(prefix string) ([]string, error)

ListByPrefix calls the s3 API with the location's prefix modified relatively by the prefix arg passed to the function. The resource considerations of List() apply to this function as well.

func (*Location) ListByRegex 

func (l *Location) ListByRegex(regex *regexp.Regexp) ([]string, error)

ListByRegex retrieves the keys of all the files at the location's current path, then filters out all those that don't match the given regex. The resource considerations of List() apply here as well.

func (*Location) NewFile 

func (l *Location) NewFile(filePath string) (vfs.File, error)

NewFile uses the properties of the calling location to generate a vfs.File (backed by an s3.File). The filePath argument is expected to be a relative path to the location's current path.

func (*Location) NewLocation 

func (l *Location) NewLocation(relativePath string) (vfs.Location, error)

NewLocation makes a copy of the underlying Location, then modifies its path by calling ChangeDir with the relativePath argument, returning the resulting location. The only possible errors come from the call to ChangeDir, which, for the s3 implementation doesn't ever result in an error.

func (*Location) Path 

func (l *Location) Path() string

Path returns the prefix the location references in most s3 calls.

func (*Location) String 

func (l *Location) String() string

String implement fmt.Stringer, returning the location's URI as the default string.

func (*Location) URI 

func (l *Location) URI() string

URI returns the Location's URI as a string.

func (*Location) Volume 

func (l *Location) Volume() string

Volume returns the bucket the location is contained in.

type Options 

type Options struct {
	AccessKeyID           string `json:"accessKeyId,omitempty"`
	SecretAccessKey       string `json:"secretAccessKey,omitempty"`
	SessionToken          string `json:"sessionToken,omitempty"`
	Region                string `json:"region,omitempty"`
	Endpoint              string `json:"endpoint,omitempty"`
	ACL                   string `json:"acl,omitempty"`
	Retry                 request.Retryer
	MaxRetries            int
	FileBufferSize        int   // Buffer size in bytes used with utils.TouchCopyBuffered
	DownloadPartitionSize int64 // Partition size in bytes used to multipart download large files using S3 Downloader
}

Options holds s3-specific options. Currently only client options are used.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.