wal

Documentation

Index

• func NewSegmentsRangeReader(dir string, first, last int) (io.ReadCloser, error)
• func NewSegmentsReader(dir string) (io.ReadCloser, error)
• func SegmentName(dir string, i int) string
• type CorruptionErr
  • func (e *CorruptionErr) Error() string
• type Reader
  • func NewReader(r io.Reader) *Reader
  • func (r *Reader) Err() error
  • func (r *Reader) Next() bool
  • func (r *Reader) Record() []byte
• type Segment
  • func CreateSegment(dir string, k int) (*Segment, error)
  • func OpenReadSegment(fn string) (*Segment, error)
  • func OpenWriteSegment(logger log.Logger, dir string, k int) (*Segment, error)
  • func (s *Segment) Dir() string
  • func (s *Segment) Index() int
• type WAL
  • func New(logger log.Logger, reg prometheus.Registerer, dir string) (*WAL, error)
  • func NewSize(logger log.Logger, reg prometheus.Registerer, dir string, segmentSize int) (*WAL, error)
  • func (w *WAL) Close() (err error)
  • func (w *WAL) Dir() string
  • func (w *WAL) Log(recs ...[]byte) error
  • func (w *WAL) Repair(origErr error) error
  • func (w *WAL) Segments() (first, last int, err error)
  • func (w *WAL) Truncate(i int) (err error)

Functions

func NewSegmentsRangeReader

func NewSegmentsRangeReader(dir string, first, last int) (io.ReadCloser, error)

NewSegmentsRangeReader returns a new reader over the given WAL segment range. If first or last are -1, the range is open on the respective end.

func NewSegmentsReader

func NewSegmentsReader(dir string) (io.ReadCloser, error)

NewSegmentsReader returns a new reader over all segments in the directory.

func SegmentName

func SegmentName(dir string, i int) string

SegmentName builds a segment name for the directory.

Types

type CorruptionErr

type CorruptionErr struct {
	Segment int
	Offset  int64
	Err     error
}

CorruptionErr is an error that's returned when corruption is encountered.

func (*CorruptionErr) Error

func (e *CorruptionErr) Error() string

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader reads WAL records from an io.Reader.

func NewReader

func NewReader(r io.Reader) *Reader

NewReader returns a new reader.

func (*Reader) Err

func (r *Reader) Err() error

Err returns the last encountered error wrapped in a corruption error. If the reader does not allow to infer a segment index and offset, a total offset in the reader stream will be provided.

func (*Reader) Next

func (r *Reader) Next() bool

Next advances the reader to the next records and returns true if it exists. It must not be called again after it returned false.

func (*Reader) Record

func (r *Reader) Record() []byte

Record returns the current record. The returned byte slice is only valid until the next call to Next.

type Segment

type Segment struct {
	*os.File
	// contains filtered or unexported fields
}

Segment represents a segment file.

func CreateSegment

func CreateSegment(dir string, k int) (*Segment, error)

CreateSegment creates a new segment k in dir.

func OpenReadSegment

func OpenReadSegment(fn string) (*Segment, error)

OpenReadSegment opens the segment with the given filename.

func OpenWriteSegment

func OpenWriteSegment(logger log.Logger, dir string, k int) (*Segment, error)

OpenWriteSegment opens segment k in dir. The returned segment is ready for new appends.

func (*Segment) Dir

func (s *Segment) Dir() string

Dir returns the directory of the segment.

func (*Segment) Index

func (s *Segment) Index() int

Index returns the index of the segment.

type WAL

type WAL struct {
	// contains filtered or unexported fields
}

WAL is a write ahead log that stores records in segment files. It must be read from start to end once before logging new data. If an error occurs during read, the repair procedure must be called before it's safe to do further writes.

Segments are written to in pages of 32KB, with records possibly split across page boundaries. Records are never split across segments to allow full segments to be safely truncated. It also ensures that torn writes never corrupt records beyond the most recent segment.

func New

func New(logger log.Logger, reg prometheus.Registerer, dir string) (*WAL, error)

New returns a new WAL over the given directory.

func NewSize

func NewSize(logger log.Logger, reg prometheus.Registerer, dir string, segmentSize int) (*WAL, error)

NewSize returns a new WAL over the given directory. New segments are created with the specified size.

func (*WAL) Close

func (w *WAL) Close() (err error)

Close flushes all writes and closes active segment.

func (*WAL) Dir

func (w *WAL) Dir() string

Dir returns the directory of the WAL.

func (*WAL) Log

func (w *WAL) Log(recs ...[]byte) error

Log writes the records into the log. Multiple records can be passed at once to reduce writes and increase throughput.

func (*WAL) Repair

func (w *WAL) Repair(origErr error) error

Repair attempts to repair the WAL based on the error. It discards all data after the corruption.

func (*WAL) Segments

func (w *WAL) Segments() (first, last int, err error)

Segments returns the range [first, n] of currently existing segments. If no segments are found, first and n are -1.

func (*WAL) Truncate

func (w *WAL) Truncate(i int) (err error)

Truncate drops all segments before i.