writer

Documentation

Index

• Constants
• type Batch
  • func (b *Batch) Bytes() ([]byte, error)
  • func (b *Batch) LastPosition() sdk.Position
• type Local
  • func (w *Local) LastPosition() sdk.Position
  • func (w *Local) Write(_ context.Context, batch *Batch) error
• type S3
  • func NewS3(ctx context.Context, cfg *S3Config) (*S3, error)
  • func (w *S3) LastPosition() sdk.Position
  • func (w *S3) Write(ctx context.Context, batch *Batch) error
• type S3Config
• type Writer

Constants

const S3FilesWrittenLength = 100

S3FilesWrittenLength defines the number of last filenames an S3 Writer keep track of when storing files in S3. This is only used in tests.

Types

type Batch

type Batch struct {
	Format  format.Format
	Records []sdk.Record
}

Batch describes the data that needs to be saved by the Writer

func (*Batch) Bytes

func (b *Batch) Bytes() ([]byte, error)

Bytes returns a byte representation for the Writer to write into a file.

func (*Batch) LastPosition

func (b *Batch) LastPosition() sdk.Position

LastPosition returns the position of the last record in the batch.

type Local

type Local struct {
	Path     string
	Position sdk.Position
	Count    uint
}

Local writer dumps bytes into a local file. The file will be placed in a directory defined by path property.

func (*Local) LastPosition

func (w *Local) LastPosition() sdk.Position

LastPosition returns the last persisted position

func (*Local) Write

func (w *Local) Write(_ context.Context, batch *Batch) error

Write writes a batch into a file on a local file system so it could later be compared to a reference file.

type S3

type S3 struct {
	KeyPrefix    string
	Bucket       string
	Position     sdk.Position
	Error        error
	FilesWritten []string
	Client       *s3.Client
}

S3 writer stores batch bytes into an S3 bucket as a file.

func NewS3

func NewS3(ctx context.Context, cfg *S3Config) (*S3, error)

NewS3 takes an S3Config reference and produces an S3 Writer

func (*S3) LastPosition

func (w *S3) LastPosition() sdk.Position

LastPosition returns the last persisted position

func (*S3) Write

func (w *S3) Write(ctx context.Context, batch *Batch) error

Write stores the batch on AWS S3 as a file

type S3Config

type S3Config struct {
	AccessKeyID     string
	SecretAccessKey string
	SessionToken    string
	Region          string
	Bucket          string
	KeyPrefix       string
}

S3Config is a type used to initialize an S3 Writer

type Writer

type Writer interface {
	Write(context.Context, *Batch) error
	LastPosition() sdk.Position
}

Writer is an interface that is responsible for persisting record that Destination has accumulated in its buffers. The default writer the Destination would use is S3Writer, others exists to test local behavior.