s3readerat

README

s3readerat

An implementation of io.ReaderAt that works using S3 GetObject and Range. Inspired by seekinghttp and httpreaderat.

Example

You can try out the included command-line program, seek-s3. It will let you fetch ranges of an S3 object, using the io.ReaderAt interface provided by seekings3.

$ go build ./cmd/seek-s3
$ ./seek-s3 -help
Usage of ./seek-s3:
  -debug
    	enable verbose output
  -limit int
    	limit the bytes to print (-1 is unlimited) (default -1)
  -offset int
    	offset parameter to seek (default -8)
  -whence int
    	whence parameter to seek (0 is start, 1 is current and 2 is end) (default 2)

For example, assuming your S3 object is a Parquet file, you can read the last 4 bytes.

$ ./seek-s3 -offset -4 -whence 2 -limit 4 s3://$BUCKET/$KEY | xxd
00000000: 5041 5231                                PAR1

Documentation

Overview

Package s3readerat implements io.ReaderAt using S3 GetObject and Range.

Index

• type S3ReaderAt
  • func New(client *s3.Client, bucket string, key string) (ra *S3ReaderAt, err error)
  • func NewWithSize(client *s3.Client, bucket string, key string, size int64) (ra *S3ReaderAt, err error)
  • func (ra *S3ReaderAt) ReadAt(p []byte, off int64) (n int, err error)
  • func (ra *S3ReaderAt) Size() (int64, error)
  • func (ra *S3ReaderAt) WithContext(ctx context.Context) *S3ReaderAt

Types

type S3ReaderAt

type S3ReaderAt struct {
	Debug bool
	// contains filtered or unexported fields
}

S3ReaderAt is io.ReaderAt implementation that makes HTTP Range Requests. New instances must be created with the New() function. It is safe for concurrent use.

func New

func New(client *s3.Client, bucket string, key string) (ra *S3ReaderAt, err error)

New creates a new S3ReaderAt.

func NewWithSize

func NewWithSize(client *s3.Client, bucket string, key string, size int64) (ra *S3ReaderAt, err error)

NewWithSize creates a new S3ReaderAt that skips checking the S3 object's size.

func (*S3ReaderAt) ReadAt

func (ra *S3ReaderAt) ReadAt(p []byte, off int64) (n int, err error)

ReadAt reads len(b) bytes from the remote file starting at byte offset off. It returns the number of bytes read and the error, if any. ReadAt always returns a non-nil error when n < len(b). At end of file, that error is io.EOF. It is safe for concurrent use.

func (*S3ReaderAt) Size

func (ra *S3ReaderAt) Size() (int64, error)

func (*S3ReaderAt) WithContext

func (ra *S3ReaderAt) WithContext(ctx context.Context) *S3ReaderAt