common

Documentation

Overview

Package common provides functionality to read and write to a archive file and keep track of underlying *os.File to close. It also provides a handy layer to keep archive files under temporary name until they are closed and complete.

Index

• func BufferSize(bufferSize int) func(*BasicArchive) error
• func Compress(c bool) func(*BasicArchive) error
• func Logger(logger *zap.Logger) func(*BasicArchive) error
• type ArchiveFileDetails
• type BasicArchive
  • func NewBasicArchive(stageDir, prefix, extension string, options ...func(*BasicArchive) error) (ba *BasicArchive, err error)
  • func OpenArchive(fileName string, bufferSize int, deleteOnClose bool) (rf *BasicArchive, err error)
  • func (rf *BasicArchive) Close() (err error)
  • func (rf *BasicArchive) FinalizedFiles() map[string]ArchiveFileDetails
  • func (rf *BasicArchive) Flush() (err error)
  • func (rf *BasicArchive) Name() string
  • func (rf *BasicArchive) ProgressPrinter(statChan chan int64, fileName string, fileSize int64, wg *sync.WaitGroup)
  • func (rf *BasicArchive) Read(p []byte) (n int, err error)
  • func (rf *BasicArchive) Reset()
  • func (rf *BasicArchive) Rotate() (err error)
  • func (rf *BasicArchive) TrueContentLength() int64
  • func (rf *BasicArchive) Write(buf []byte) (int, error)
• type FinalizerFunc

Functions

func BufferSize

func BufferSize(bufferSize int) func(*BasicArchive) error

func Compress

func Compress(c bool) func(*BasicArchive) error

func Logger

func Logger(logger *zap.Logger) func(*BasicArchive) error

Types

type ArchiveFileDetails

type ArchiveFileDetails struct {
	FileName      string // Not filled when it is a checksum-only writer.
	BytesWritten  int64  // Always filled
	ChunksWritten int64  // Always filled, but does not accurately represent rows/requests.
	RowsWritten   int64  // Not filled by archiver
	Checksum      string // Sometimes filled: only by checksum-only writer.
}

type BasicArchive

type BasicArchive struct {
	Logger *zap.Logger

	ChunksWritten int64
	Finalizer     FinalizerFunc
	// contains filtered or unexported fields
}

BasicArchive encapsulates some common functionality between S3Archive, FileArchive, and AZArchive

func NewBasicArchive

func NewBasicArchive(stageDir, prefix, extension string,
	options ...func(*BasicArchive) error) (ba *BasicArchive, err error)

func OpenArchive

func OpenArchive(fileName string, bufferSize int, deleteOnClose bool) (rf *BasicArchive, err error)

OpenArchive opens an archive file for reading. `*BasicArchive` returned is an io.Reader

func (*BasicArchive) Close

func (rf *BasicArchive) Close() (err error)

Close works for both Read and Write. Additional logic for write to finalize the file from temp-name to final-name

func (*BasicArchive) FinalizedFiles

func (rf *BasicArchive) FinalizedFiles() map[string]ArchiveFileDetails

func (*BasicArchive) Flush

func (rf *BasicArchive) Flush() (err error)

Flush complements io.Writer

func (*BasicArchive) Name

func (rf *BasicArchive) Name() string

func (*BasicArchive) ProgressPrinter

func (rf *BasicArchive) ProgressPrinter(statChan chan int64, fileName string, fileSize int64, wg *sync.WaitGroup)

ProgressPrinter is a helper function. Currently only usable with Azure blob upload

func (*BasicArchive) Read

func (rf *BasicArchive) Read(p []byte) (n int, err error)

Read satisfies io.Reader interface - main logic is the transparent read from LZ4, Bufio, or Raw FP depending on how the file was opened

func (*BasicArchive) Reset

func (rf *BasicArchive) Reset()

func (*BasicArchive) Rotate

func (rf *BasicArchive) Rotate() (err error)

Rotate creates a new archive file or if one already exists, then it closes the current one and create another empty file To keep file sizes small, Rotate() must be called at regular intervals (either by size or time depending on your preference)

func (*BasicArchive) TrueContentLength

func (rf *BasicArchive) TrueContentLength() int64

func (*BasicArchive) Write

func (rf *BasicArchive) Write(buf []byte) (int, error)

Write satisfies io.Writer interface - main logic is the transparent write to LZ4, Bufio, or Raw FP depending on how the file was opened

type FinalizerFunc

type FinalizerFunc func() (finalFile ArchiveFileDetails, err error)

FinalizerFunc is a callback that will be called on Close. Typically one would rename the file to the final desired name OR upload the file to S3 or Azure Blobstore. Users of this library must provide this call back implementation.