multibuf

README

Multibuf

Bytes buffer that implements seeking and partially persisting to disk.

Please read http://godoc.org/github.com/mailgun/multibuf

Documentation

Overview

package multibuf implements buffer optimized for streaming large chunks of data, multiple reads and optional partial buffering to disk.

Index

• Constants
• func MaxBytes(m int64) optionSetter
• func MemBytes(m int64) optionSetter
• type MaxSizeReachedError
  • func (e *MaxSizeReachedError) Error() string
• type MultiReader
  • func New(input io.Reader, setters ...optionSetter) (MultiReader, error)
• type WriterOnce
  • func NewWriterOnce(setters ...optionSetter) (WriterOnce, error)

Constants

const (
	DefaultMemBytes = 1048576
	DefaultMaxBytes = -1
	// Equivalent of bytes.MinRead used in ioutil.ReadAll
	DefaultBufferBytes = 512
)

Functions

func MaxBytes

func MaxBytes(m int64) optionSetter

MaxBytes, ignored if set to value >=, if request exceeds the specified limit, the reader will return error, by default buffer is not limited, negative values mean no limit

func MemBytes

func MemBytes(m int64) optionSetter

MemBytes specifies the largest buffer to hold in RAM before writing to disk, default is 1MB

Types

type MaxSizeReachedError

type MaxSizeReachedError struct {
	MaxSize int64
}

MaxSizeReachedError is returned when the maximum allowed buffer size is reached when reading

func (*MaxSizeReachedError) Error

func (e *MaxSizeReachedError) Error() string

type MultiReader

type MultiReader interface {
	io.Reader
	io.Seeker
	io.Closer
	io.WriterTo

	// Size calculates and returns the total size of the reader and not the length remaining.
	Size() (int64, error)
}

MultiReader provides Read, Close, Seek and Size methods. In addition to that it supports WriterTo interface to provide efficient writing schemes, as functions like io.Copy use WriterTo when it's available.

func New

func New(input io.Reader, setters ...optionSetter) (MultiReader, error)

New returns MultiReader that can limit the size of the buffer and persist large buffers to disk. By default New returns unbound buffer that will read up to 1MB in RAM and will start buffering to disk It supports multiple functional optional arguments:

// Buffer up to 1MB in RAM and limit max buffer size to 20MB
multibuf.New(r, multibuf.MemBytes(1024 * 1024), multibuf.MaxBytes(1024 * 1024 * 20))

type WriterOnce

type WriterOnce interface {
	// Write implements io.Writer
	Write(p []byte) (int, error)
	// Reader transfers all data written to this writer to MultiReader. If there was no data written it retuns an error
	Reader() (MultiReader, error)
	// WriterOnce owns the data before Reader has been called, so Close will close all the underlying files if Reader has not been called.
	Close() error
}

WriterOnce implements write once, read many times writer. Create a WriterOnce and write to it, once Reader() function has been called, the internal data is transferred to MultiReader and this instance of WriterOnce should be no longer used.

func NewWriterOnce

func NewWriterOnce(setters ...optionSetter) (WriterOnce, error)

NewWriterOnce returns io.ReadWrite compatilble objkect that can limit the size of the buffer and persist large buffers to disk. WriterOnce implements write once, read many times writer. Create a WriterOnce and write to it, once Reader() function has been called, the internal data is transferred to MultiReader and this instance of WriterOnce should be no longer used. By default NewWriterOnce returns unbound buffer that will allow to write up to 1MB in RAM and will start buffering to disk It supports multiple functional optional arguments:

// Buffer up to 1MB in RAM and limit max buffer size to 20MB
multibuf.NewWriterOnce(r, multibuf.MemBytes(1024 * 1024), multibuf.MaxBytes(1024 * 1024 * 20))