awss3reader

README

AWS S3 Reader

Efficient reader for large S3 files.

• Seek() via Byte-Range HTTP offsets^1
• zero-memory copy
• early HTTP Body termination

s3client := s3.New(session.Must(session.NewSession(
    aws.NewConfig().WithRegion("ap-southeast-1"),
)))

r := awss3reader.NewS3ReadSeeker(
    s3client,
    "nikolaydubina-blog-public",
    "videos/2024-02-22.mov",
    awss3reader.FixedChunkSizePolicy{Size: 1 << 20 * 40},
)
defer r.Close()

r.Seek(100, io.SeekCurrent)

res, err := io.ReadAll(r)

Related Work

• https://github.com/yacchi/s3-fast-reader — provides io.Reader interface, focuses on connection pool and parallelism, uses mocks for tests

Documentation

Index

• type ChunkSizePolicy
• type FixedChunkSizePolicy
  • func (s FixedChunkSizePolicy) ChunkSize() int
• type S3ReadSeeker
  • func NewS3ReadSeeker(s3client *s3.Client, bucket string, key string, ...) *S3ReadSeeker
  • func (s *S3ReadSeeker) Close() error
  • func (s *S3ReadSeeker) Read(b []byte) (int, error)
  • func (s *S3ReadSeeker) Seek(offset int64, whence int) (int64, error)

Examples

• S3ReadSeeker

Types

type ChunkSizePolicy

type ChunkSizePolicy interface {
	ChunkSize() int
}

ChunkSizePolicy is something that can tell how much data to fetch in single request for given S3 Object. With more advanced policies, Visit methods will be integrated.

type FixedChunkSizePolicy

type FixedChunkSizePolicy struct {
	Size int
}

FixedChunkSizePolicy always returns same chunk size.

func (FixedChunkSizePolicy) ChunkSize

func (s FixedChunkSizePolicy) ChunkSize() int

type S3ReadSeeker

type S3ReadSeeker struct {
	// contains filtered or unexported fields
}

S3ReadSeeker is a reader of given S3 Object. It utilizes HTTP Byte Ranges to read chunks of data from S3 Object. It uses zero-memory copy from underlying HTTP Body response. It uses early HTTP Body termination, if seeks are beyond current HTTP Body. It uses adaptive policy for chunk size fetching. This is useful for iterating over very large S3 Objects.

Example

package main

import (
	"io"

	"github.com/aws/aws-sdk-go/aws"
	"github.com/aws/aws-sdk-go/aws/session"
	"github.com/aws/aws-sdk-go/service/s3"

	awss3reader "github.com/rkrdeano/aws-s3-reader"
)

func main() {
	s3client := s3.New(session.Must(session.NewSession(
		aws.NewConfig().WithRegion("ap-southeast-1"),
	)))

	r := awss3reader.NewS3ReadSeeker(
		s3client,
		"nikolaydubina-blog-public",
		"videos/2024-02-22.mov",
		awss3reader.FixedChunkSizePolicy{Size: 1 << 20 * 40},
	)
	defer r.Close()

	r.Seek(100, io.SeekCurrent)
	res, err := io.ReadAll(r)

	if err != nil || len(res) == 0 {
		panic(err)
	}
}

func NewS3ReadSeeker

func NewS3ReadSeeker(
	s3client *s3.Client,
	bucket string,
	key string,
	chunkSizePolicy ChunkSizePolicy,
) *S3ReadSeeker

func (*S3ReadSeeker) Close

func (s *S3ReadSeeker) Close() error

func (*S3ReadSeeker) Read

func (s *S3ReadSeeker) Read(b []byte) (int, error)

func (*S3ReadSeeker) Seek

func (s *S3ReadSeeker) Seek(offset int64, whence int) (int64, error)

Seek assumes always can seek to position in S3 object. Seeking beyond S3 file size will result failures in Read calls.