Documentation
¶
Overview ¶
Package s3 implements a lightweight client of the AWS S3 API.
The Reader type can be used to view S3 objects as an io.Reader or io.ReaderAt.
Index ¶
- Constants
- Variables
- func BucketRegion(k *aws.SigningKey, bucket string) (string, error)
- func DeriveForBucket(bucket string) aws.DeriveFn
- func URL(k *aws.SigningKey, bucket, object string) (string, error)
- func ValidBucket(bucket string) bool
- type BucketFS
- func (b *BucketFS) Open(name string) (fs.File, error)
- func (b *BucketFS) OpenRange(name, etag string, start, width int64) (io.ReadCloser, error)
- func (b *BucketFS) Put(where string, contents []byte) (string, error)
- func (b *BucketFS) ReadDir(name string) ([]fs.DirEntry, error)
- func (b *BucketFS) Remove(fullpath string) error
- func (b *BucketFS) Sub(dir string) (fs.FS, error)
- func (b *BucketFS) VisitDir(name, seek, pattern string, walk fsutil.VisitDirFn) error
- func (b *BucketFS) WithContext(ctx context.Context) fs.FS
- type File
- func (f *File) Close() error
- func (f *File) Info() (fs.FileInfo, error)
- func (f *File) IsDir() bool
- func (f *File) ModTime() time.Time
- func (f *File) Mode() fs.FileMode
- func (f *File) Name() string
- func (f *File) Open() (fs.File, error)
- func (f *File) Path() string
- func (f *File) Read(p []byte) (int, error)
- func (f *File) Seek(offset int64, whence int) (int64, error)
- func (f *File) SelectJSON(query, infmt string) (io.ReadCloser, error)
- func (f *File) Size() int64
- func (f *File) Stat() (fs.FileInfo, error)
- func (f *File) Sys() interface{}
- func (f *File) Type() fs.FileMode
- type Prefix
- func (p *Prefix) Close() error
- func (p *Prefix) Info() (fs.FileInfo, error)
- func (p *Prefix) IsDir() bool
- func (p *Prefix) ModTime() time.Time
- func (p *Prefix) Mode() fs.FileMode
- func (p *Prefix) Name() string
- func (p *Prefix) Open(file string) (fs.File, error)
- func (p *Prefix) Read(_ []byte) (int, error)
- func (p *Prefix) ReadDir(n int) ([]fs.DirEntry, error)
- func (p *Prefix) Size() int64
- func (p *Prefix) Stat() (fs.FileInfo, error)
- func (p *Prefix) Sys() interface{}
- func (p *Prefix) Type() fs.FileMode
- func (p *Prefix) VisitDir(name, seek, pattern string, walk fsutil.VisitDirFn) error
- func (p *Prefix) WithContext(ctx context.Context) fs.FS
- type Reader
- type Uploader
- func (u *Uploader) Abort() error
- func (u *Uploader) Close(final []byte) error
- func (u *Uploader) Closed() bool
- func (u *Uploader) CompletedParts() int
- func (u *Uploader) CopyFrom(num int64, source *Reader, start int64, end int64) error
- func (u *Uploader) ETag() string
- func (u *Uploader) ID() string
- func (u *Uploader) MinPartSize() int
- func (u *Uploader) NextPart() int64
- func (u *Uploader) Size() int64
- func (u *Uploader) Start() error
- func (u *Uploader) Upload(num int64, contents []byte) error
- func (u *Uploader) UploadReaderAt(r io.ReaderAt, size int64) error
Constants ¶
const MinPartSize = 5 * 1024 * 1024
MinPartSize is the minimum size for all of the parts of a multi-part upload except for the final part.
Variables ¶
var ( // ErrInvalidBucket is returned from calls that attempt // to use a bucket name that isn't valid according to // the S3 specification. ErrInvalidBucket = errors.New("invalid bucket name") // ErrETagChanged is returned from read operations where // the ETag of the underlying file has changed since // the file handle was constructed. (This package guarantees // that file read operations are always consistent with respect // to the ETag originally associated with the file handle.) ErrETagChanged = errors.New("file ETag changed") )
var DefaultClient = http.Client{ Transport: &http.Transport{ ResponseHeaderTimeout: 60 * time.Second, MaxIdleConnsPerHost: 5, DisableCompression: true, DialContext: (&net.Dialer{ Timeout: 2 * time.Second, }).DialContext, }, }
DefaultClient is the default HTTP client used for requests made from this package.
Functions ¶
func BucketRegion ¶
func BucketRegion(k *aws.SigningKey, bucket string) (string, error)
BucketRegion returns the region associated with the given bucket.
func DeriveForBucket ¶
DeriveForBucket can be passed to aws.AmbientCreds as a DeriveFn that automatically re-derives keys so that they apply to the region in which the given bucket lives.
func URL ¶
func URL(k *aws.SigningKey, bucket, object string) (string, error)
URL returns a signed URL for a bucket and object that can be used directly with http.Get.
func ValidBucket ¶
ValidBucket returns whether or not bucket is a valid bucket name.
See https://docs.aws.amazon.com/AmazonS3/latest/userguide/bucketnamingrules.html
Note: ValidBucket does not allow '.' characters, since bucket names containing dots are not accessible over HTTPS. (AWS docs say "not recommended for uses other than static website hosting.")
Types ¶
type BucketFS ¶
type BucketFS struct {
Key *aws.SigningKey
Bucket string
Client *http.Client
Ctx context.Context
// DelayGet, if true, causes the
// Open call to use a HEAD operation
// rather than a GET operation.
// The first call to fs.File.Read will
// cause the full GET to be performed.
DelayGet bool
}
BucketFS implements fs.FS, fs.ReadDirFS, and fs.SubFS.
func (*BucketFS) Open ¶
Open implements fs.FS.Open
The returned fs.File will be either a *File or a *Prefix depending on whether name refers to an object or a common path prefix that leads to multiple objects. If name does not refer to an object or a path prefix, then Open returns an error matching fs.ErrNotExist.
func (*BucketFS) OpenRange ¶
OpenRange produces an io.ReadCloser that reads data from the file given by [name] with the etag given by [etag] starting at byte [start] and continuing for [width] bytes. If [etag] does not match the ETag of the object, then ErrETagChanged will be returned.
func (*BucketFS) Put ¶
Put performs a PutObject operation at the object key 'where' and returns the ETag of the newly-created object.
type File ¶
type File struct {
// Reader is a reader that points to
// the associated s3 object.
Reader
// contains filtered or unexported fields
}
File implements fs.File
func NewFile ¶
func NewFile(k *aws.SigningKey, bucket, object, etag string, size int64) *File
NewFile constructs a File that points to the given bucket, object, etag, and file size. The caller is assumed to have correctly determined these attributes in advance; this call does not perform any I/O to verify that the provided object exists or has a matching ETag and size.
func (*File) ModTime ¶
ModTime implements fs.DirEntry.ModTime. This returns the same value as f.Reader.LastModified.
func (*File) Path ¶
Path returns the full path to the S3 object within its bucket. See also blockfmt.NamedFile
func (*File) Read ¶
Read implements fs.File.Read
Note: Read is not safe to call from multiple goroutines simultaneously. Use ReadAt for parallel reads.
Also note: the first call to Read performs an HTTP request to S3 to read the entire contents of the object starting at the current read offset (zero by default, or another offset set via Seek). If you need to read a sub-range of the object, consider using f.Reader.RangeReader
func (*File) Seek ¶
Seek implements io.Seeker
Seek rejects offsets that are beyond the size of the underlying object.
func (*File) SelectJSON ¶
func (f *File) SelectJSON(query, infmt string) (io.ReadCloser, error)
SelectJSON runs [query] on [f] and returns the response as a JSON stream.
type Prefix ¶
type Prefix struct {
// Key is the signing key used to sign requests.
Key *aws.SigningKey `xml:"-"`
// Bucket is the bucket at the root of the "filesystem"
Bucket string `xml:"-"`
// Path is the path of this prefix.
// The value of Path should always be
// a valid path (see fs.ValidPath) plus
// a trailing forward slash to indicate
// that this is a pseudo-directory prefix.
Path string `xml:"Prefix"`
Client *http.Client `xml:"-"`
Ctx context.Context `xml:"-"`
// contains filtered or unexported fields
}
Prefix implements fs.File, fs.ReadDirFile, and fs.DirEntry, and fs.FS.
func (*Prefix) ModTime ¶
ModTime implements fs.FileInfo.ModTime
Note: currently ModTime returns the zero time.Time, as S3 prefixes don't have a meaningful modification time.
func (*Prefix) Open ¶
Open opens the object or pseudo-directory at the provided path. The returned fs.File will be a *File if the combined Prefix and path lead to an object; if the combind prefix and path produce another complete object prefix, then a *Prefix will be returned. If the combined prefix and path do not produce a prefix that is present within the target bucket, then an error matching fs.ErrNotExist is returned.
func (*Prefix) ReadDir ¶
ReadDir implements fs.ReadDirFile
Every returned fs.DirEntry will be either a Prefix or a File struct.
type Reader ¶
type Reader struct {
// Key is the sigining key that
// Reader uses to make HTTP requests.
// The key may have to be refreshed
// every so often (see aws.SigningKey.Expired)
Key *aws.SigningKey `xml:"-"`
// Client is the HTTP client used to
// make HTTP requests. By default it is
// populated with DefaultClient, but
// it may be set to any reasonable http client
// implementation.
Client *http.Client `xml:"-"`
// ETag is the ETag of the object in S3
// as returned by listing or a HEAD operation.
ETag string `xml:"ETag"`
// LastModified is the object's LastModified time
// as returned by listing or a HEAD operation.
LastModified time.Time `xml:"LastModified"`
// Size is the object size in bytes.
// It is populated on Open.
Size int64 `xml:"Size"`
// Bucket is the S3 bucket holding the object.
Bucket string `xml:"-"`
// Path is the S3 object key.
Path string `xml:"Key"`
}
Reader presents a read-only view of an S3 object
func Stat ¶
func Stat(k *aws.SigningKey, bucket, object string) (*Reader, error)
Stat performs a HEAD on an S3 object and returns an associated Reader.
func (*Reader) RangeReader ¶
func (r *Reader) RangeReader(off, width int64) (io.ReadCloser, error)
RangeReader produces an io.ReadCloser that reads bytes in the range from [off, off+width)
It is the caller's responsibility to call Close() on the returned io.ReadCloser.
type Uploader ¶
type Uploader struct {
// Key is the key used to sign requests.
// It cannot be nil.
Key *aws.SigningKey
// Client is the http client used to
// make requests. If it is nil, then
// DefaultClient will be used.
Client *http.Client
// ContentType, if not an empty string,
// will be the Content-Type of the new object.
ContentType string
Bucket, Object string
Scheme string
// Host, if not the empty string,
// is the host of the bucket.
// If Host is unset, then "s3.amazonaws.com" is used.
//
// Requests are always made to <bucket>.host
Host string
// Mbbs, if non-zero, is the expected
// link speed in Mbps. This number is
// used to determine the optimal parallelism
// for uploads.
// (For example, use Mbps = 25000 on a 25Gbps link, etc.)
Mbps int
// contains filtered or unexported fields
}
Uploader wraps the state of a multi-part upload.
To use an Uploader to create a multi-part object, populate all of the public fields of the Uploader and then call Uploader.Start, followed by zero or more calls to Uploader.UploadPart, followed by one call to Uploader.Close
func (*Uploader) Abort ¶
Abort aborts a multi-part upload.
Abort is *not* safe to call concurrently with Start, Close, or UploadPart.
If Start has not been called on the Uploader, or if the uploader has successfully finished uploading, Abort does nothing.
If Abort is called on a partially-finished Upload and returns without an error, then the state of the Uploader is reset so that Start may be called again to re-try the upload.
func (*Uploader) Close ¶
Close uploads the final part of the multi-part upload and asks S3 to finalize the object from its constituent parts. (If size is zero, then r may be nil, in which case no final part is uploaded before the multi-part object is finalized.)
Close will panic if Start has never been called or if Close has already been called and returned successfully.
func (*Uploader) CompletedParts ¶
CompletedParts returns the number of parts that have been successfully uploaded.
It is safe to call CompletedParts from multiple goroutines that may also be calling UploadPart, but be wary of logical races involving the number of uploaded parts.
func (*Uploader) CopyFrom ¶
CopyFrom performs a server side copy for the part number `num`.
Set `start` and `end` to `0` to copy the entire source object.
It is safe to call CopyFrom from multiple goroutines simultaneously. However, calls to CopyFrom must be synchronized to occur strictly after a call to Start and strictly before a call to Close.
As an optimization, most of the work for CopyFrom is performed asynchronously. Callers must call Close and check its return value in order to correctly handle errors from CopyFrom.
func (*Uploader) ETag ¶
ETag returns the ETag of the final upload. The return value of ETag is only valid after Close has been called.
func (*Uploader) ID ¶
ID returns the "Upload ID" of this upload. The return value of ID is only valid after Start has been called.
func (*Uploader) MinPartSize ¶
MinPartSize returns the minimum part size for the Uploader.
(The return value of MinPartSize is always s3.MinPartSize.)
func (*Uploader) NextPart ¶
NextPart atomically increments the internal part counter inside the uploader and returns the next available part number. Note that AWS multipart uploads have 1-based part numbers (i.e. the first part is part 1).
If the data to be uploaded is intrinsically un-ordered, then NextPart() can be used to greedily assign part numbers.
Note that currently the maximum part number allowed by AWS is 10000.
func (*Uploader) Start ¶
Start begins a multipart upload. Start must be called exactly once, before any calls to WritePart are made.
func (*Uploader) Upload ¶
Upload uploads the part number num from the ReadCloser r, which must return exactly size bytes of data. S3 prohibits multi-part upload parts smaller than 5MB (except for the final bytes), so size must be at least 5MB.
It is safe to call Upload from multiple goroutines simultaneously. However, calls to Upload must be synchronized to occur strictly after a call to Start and strictly before a call to Close.
func (*Uploader) UploadReaderAt ¶
UploadReaderAt is a utility method that performs a parallel upload of an io.ReaderAt of a given size.
UploadReaderAt closes the Uploader after uploading the entirety of the contents of r.
UploadReaderAt is not safe to call concurrently with UploadPart or Close.
Source Files