wal

Documentation

Overview

Package wal implements an append-only segmented write-ahead log with CRC-checked records and sequence numbers. Records are opaque byte payloads; the encoding of payload contents (record type, fields) is the caller's responsibility.

Index

• Constants
• Variables
• type Reader
  • func NewReader(dir string) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) Next() (Record, error)
  • func (r *Reader) TornTail() bool
• type Record
• type Writer
  • func Open(dir string) (*Writer, error)
  • func (w *Writer) Append(payload []byte) (uint64, error)
  • func (w *Writer) Close() error
  • func (w *Writer) FsyncCount() uint64
  • func (w *Writer) NextSeq() uint64
  • func (w *Writer) SetMaxSegmentBytes(n int64)
  • func (w *Writer) Sync() error
  • func (w *Writer) SyncUpTo(target uint64) error
  • func (w *Writer) TruncateThrough(snapshotSeq uint64) error

Constants

const DefaultMaxSegmentBytes int64 = 64 * 1024 * 1024

DefaultMaxSegmentBytes is the rotation threshold for new segments.

Variables

var ErrPayloadTooLarge = errors.New("wal: payload exceeds maximum frame size")

ErrPayloadTooLarge is returned by Append when a record's payload exceeds what the frame's uint32 length field can represent.

Types

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader iterates records across all segments in a directory. Not safe for concurrent use.

func NewReader

func NewReader(dir string) (*Reader, error)

NewReader opens a Reader over dir. If dir contains no segments the first Next returns io.EOF.

func (*Reader) Close

func (r *Reader) Close() error

Close releases the current file handle if any.

func (*Reader) Next

func (r *Reader) Next() (Record, error)

Next returns the next record or io.EOF at the end. If a torn record is detected at the tail of the newest segment, iteration ends cleanly with io.EOF and TornTail returns true. Reopening a Writer on the same directory truncates the torn tail. A CRC failure mid-stream (i.e. in any segment other than the newest, or before its end) is reported as an error.

func (*Reader) TornTail

func (r *Reader) TornTail() bool

TornTail reports whether iteration ended because the last segment had a torn or CRC-failed record at its tail.

type Record

type Record struct {
	Seq     uint64
	Payload []byte
}

Record is a single sequenced log entry.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is an append-only segmented WAL. Append buffers a record and Sync (or SyncUpTo) fsyncs the current segment. Append and Sync are safe to call from multiple goroutines but are serialized internally.

Group commit: callers that append under their own lock (so append order matches sequence order) can release that lock and call SyncUpTo without it. Concurrent SyncUpTo callers collapse into a single fsync — the first to claim the sync slot flushes for everyone, and the rest return as soon as that flush covers their sequence. This trades a small visibility-before- durability window for far fewer fsyncs under write load.

func Open

func Open(dir string) (*Writer, error)

Open opens (or creates) a WAL in dir. If the newest segment ends with a torn record it is truncated to the last valid record before the writer is opened for append. The next sequence number is one past the last durable record (or the newest segment's starting seq if that segment is empty).

func (*Writer) Append

func (w *Writer) Append(payload []byte) (uint64, error)

Append writes payload with the next sequence number. The record is in the OS write buffer after this returns; call Sync to make it durable.

func (*Writer) Close

func (w *Writer) Close() error

Close fsyncs and closes the current segment.

func (*Writer) FsyncCount

func (w *Writer) FsyncCount() uint64

FsyncCount returns the number of fsync syscalls the writer has issued. Used to quantify group-commit batching: under load it grows far slower than the number of appended records.

func (*Writer) NextSeq

func (w *Writer) NextSeq() uint64

NextSeq returns the sequence number the next Append will assign.

func (*Writer) SetMaxSegmentBytes

func (w *Writer) SetMaxSegmentBytes(n int64)

SetMaxSegmentBytes overrides the rotation threshold. Intended for tests.

func (*Writer) Sync

func (w *Writer) Sync() error

Sync fsyncs the current segment, making every record appended so far durable. It routes through the group-commit path so a synchronous Sync and concurrent SyncUpTo callers share one serialized fsync slot.

func (*Writer) SyncUpTo

func (w *Writer) SyncUpTo(target uint64) error

SyncUpTo makes every record through sequence target durable, batching concurrent callers into a single fsync (group commit). Callers append under their own lock (so append order matches sequence order) and call SyncUpTo without that lock held; the first to claim the sync slot fsyncs for all waiters, and the rest return as soon as that flush covers their sequence.

func (*Writer) TruncateThrough

func (w *Writer) TruncateThrough(snapshotSeq uint64) error

TruncateThrough removes segments whose records are entirely covered by a snapshot at snapshotSeq. The currently-open segment is never deleted.