wal

Documentation

Overview

Package wal implements a versioned, length-prefixed, CRC32C-checksummed Write-Ahead Log for the gograph durability stack.

The on-disk format is documented in FORMAT.md alongside this package. Each frame is self-describing; readers stop cleanly at the first torn or corrupted frame and report the byte offset where the cut occurred, leaving the file otherwise untouched.

Example

Example shows the core write-ahead-log loop: open a writer, append a few opaque payload frames, Sync them durably, then reopen the file with a Reader and replay every frame back in append order.

package main

import (
	"fmt"
	"os"
	"path/filepath"

	"github.com/FlavioCFOliveira/GoGraph/store/wal"
)

func main() {
	dir, err := os.MkdirTemp("", "wal-example")
	if err != nil {
		panic(err)
	}
	defer func() { _ = os.RemoveAll(dir) }()

	path := filepath.Join(dir, "wal")

	// Append three records. A WAL payload is opaque bytes as far as the
	// log is concerned; the durability stack above it (store/txn) gives
	// them meaning. Group-commit: several Appends, then a single Sync.
	w, err := wal.Open(path)
	if err != nil {
		panic(err)
	}
	for _, rec := range [][]byte{[]byte("alpha"), []byte("bravo"), []byte("charlie")} {
		if err := w.Append(rec); err != nil {
			panic(err)
		}
	}
	if err := w.Sync(); err != nil {
		panic(err)
	}
	if err := w.Close(); err != nil {
		panic(err)
	}

	// Replay: a fresh Reader iterates the frames in the order they were
	// appended, stopping cleanly at the first torn frame (none here).
	r, err := wal.OpenReader(path)
	if err != nil {
		panic(err)
	}
	defer func() { _ = r.Close() }()

	count := 0
	err = r.Replay(func(f wal.Frame) error {
		count++
		fmt.Printf("frame %d: %s\n", count, f.Payload)
		return nil
	})
	if err != nil {
		panic(err)
	}
	fmt.Printf("replayed %d frames\n", count)

}

Output:
frame 1: alpha
frame 2: bravo
frame 3: charlie
replayed 3 frames

Index

• Constants
• Variables
• func Encode(w io.Writer, f Frame) (int, error)
• type Frame
  • func Decode(r io.Reader) (Frame, error)
• type Reader
  • func NewReader(r io.Reader, closer io.Closer) *Reader
  • func OpenReader(path string) (*Reader, error)
  • func (r *Reader) Close() error
  • func (r *Reader) Frames() iter.Seq[Frame]
  • func (r *Reader) Replay(apply func(Frame) error) error
  • func (r *Reader) TailError() error
  • func (r *Reader) TailOffset() int64
• type Stats
• type Writer
  • func Open(path string) (*Writer, error)
  • func OpenWith(f walFile) (*Writer, error)
  • func (w *Writer) Append(payload []byte) error
  • func (w *Writer) AppendCtx(ctx context.Context, payload []byte) error
  • func (w *Writer) Close() error
  • func (w *Writer) Stats() Stats
  • func (w *Writer) Sync() error
  • func (w *Writer) SyncCtx(ctx context.Context) error
  • func (w *Writer) Truncate() (int64, error)

Examples

• Package

Constants

const CurrentVersion uint16 = 1

CurrentVersion is the WAL format version this package writes. Readers must accept all versions <= CurrentVersion; older versions are intentionally permitted so a fresh build can replay archives produced by previous releases.

const HeaderSize = 4 + 2 + 4 + 4

HeaderSize is the fixed number of bytes occupying the frame header (magic + version + length + crc32c).

Variables

var (
	// ErrBadMagic indicates the next four bytes did not match Magic.
	ErrBadMagic = errors.New("wal: bad frame magic")
	// ErrUnsupportedVersion indicates the frame version is newer
	// than this build knows how to parse.
	ErrUnsupportedVersion = errors.New("wal: unsupported frame version")
	// ErrCRCMismatch indicates the frame's CRC32C did not match the
	// re-computed value.
	ErrCRCMismatch = errors.New("wal: crc32c mismatch")
	// ErrTornFrame indicates the underlying reader returned EOF
	// before the frame was fully read.
	ErrTornFrame = errors.New("wal: torn frame at end of input")
	// ErrFrameTooLarge indicates the frame's declared payload length
	// exceeds maxFrameSize. A length field this large is treated as
	// corruption: the frame is rejected before any allocation, so a
	// crafted or corrupted length cannot force a large one-shot make.
	ErrFrameTooLarge = errors.New("wal: frame payload length exceeds maximum")
)

Errors returned by the reader.

var ErrWriterClosed = errors.New("wal: writer is closed")

ErrWriterClosed is returned by methods on a Writer that has already been closed.

var Magic = [4]byte{'G', 'G', 'W', 'A'}

Magic is the 4-byte identifier prefix of every WAL frame: ASCII "GGWA".

Functions

func Encode

func Encode(w io.Writer, f Frame) (int, error)

Encode writes f to w as a single binary frame. It returns the number of bytes written and any underlying writer error.

Types

type Frame

type Frame struct {
	Version uint16
	Payload []byte
}

Frame is the in-memory representation of one WAL frame.

func Decode

func Decode(r io.Reader) (Frame, error)

Decode reads the next frame from r. It returns ErrTornFrame when the reader ends mid-frame (clean tail truncation), ErrBadMagic on a missing magic, ErrUnsupportedVersion on a newer-than-supported version, and ErrCRCMismatch on integrity failure. Any other error is propagated from the underlying reader.

type Reader

type Reader struct {
	// contains filtered or unexported fields
}

Reader iterates the frames of a WAL file. It is read-only and stops cleanly at the first torn or corrupted frame, reporting the byte offset where the cut occurred via Reader.TailOffset.

Reader is not safe for concurrent use; create one Reader per goroutine that wishes to iterate.

func NewReader

func NewReader(r io.Reader, closer io.Closer) *Reader

NewReader builds a Reader over an io.Reader. closer may be nil if the caller owns the resource.

func OpenReader

func OpenReader(path string) (*Reader, error)

OpenReader opens path for read-only frame iteration.

func (*Reader) Close

func (r *Reader) Close() error

Close releases any underlying resource passed to NewReader or OpenReader.

func (*Reader) Frames

func (r *Reader) Frames() iter.Seq[Frame]

Frames returns an iterator over every frame in the WAL. The iterator stops at the first error; call Reader.TailError / Reader.TailOffset after iteration to inspect why.

func (*Reader) Replay

func (r *Reader) Replay(apply func(Frame) error) error

Replay applies apply to every frame in the WAL in order. If apply returns an error, replay stops with that error returned to the caller. After Replay returns, TailOffset/TailError describe where and why iteration stopped (frame-level errors).

func (*Reader) TailError

func (r *Reader) TailError() error

TailError returns the error that ended iteration (typically ErrTornFrame, ErrCRCMismatch, or ErrBadMagic), or nil when iteration ended at clean EOF.

func (*Reader) TailOffset

func (r *Reader) TailOffset() int64

TailOffset returns the byte offset (from the start of the input) where iteration stopped. After a successful iteration to EOF this equals the file size; after a torn frame this equals the start of the torn frame.

type Stats

type Stats struct {
	Frames     uint64 // total frames appended
	Bytes      uint64 // total bytes appended (header + payload)
	Syncs      uint64 // total Sync calls
	SyncFailed uint64 // Sync calls that returned an error
}

Stats is a snapshot of a Writer's lifetime counters. Counters are monotonic; subtract two snapshots to compute deltas. Values are read with sync/atomic.LoadUint64, so they may race slightly behind in-flight operations but never observe a torn value.

type Writer

type Writer struct {
	// contains filtered or unexported fields
}

Writer is a single-writer append-only log file. Callers append frames with Writer.Append and durably commit them with Writer.Sync; group-commit is achieved by appending several frames before a single Sync.

Writer is safe for concurrent calls to Writer.Append / Sync / Stats; all mutations serialise on an internal mutex.

func Open

func Open(path string) (*Writer, error)

Open opens or creates the WAL file at path for append-only writing. The file is created with mode 0o600 (owner read/write only) if it does not already exist; existing data is preserved and new frames are appended. The restrictive mode keeps the full graph mutation stream from being world-readable.

func OpenWith

func OpenWith(f walFile) (*Writer, error)

OpenWith builds a Writer over an already-open file handle. The caller transfers ownership: Writer.Close will call f.Close().

This constructor exists primarily for tests that inject a *testfs.FaultFile; production code should use Open.

func (*Writer) Append

func (w *Writer) Append(payload []byte) error

Append writes one frame with the given opaque payload to the underlying file. The frame is buffered in process memory; call Writer.Sync to durably commit.

func (*Writer) AppendCtx

func (w *Writer) AppendCtx(ctx context.Context, payload []byte) error

AppendCtx is the context-aware variant of Writer.Append. ctx.Err() is checked before acquiring the internal mutex and again before writing; on cancellation returns the wrapped ctx.Err.

func (*Writer) Close

func (w *Writer) Close() error

Close flushes any buffered frames, calls Sync once, and releases the underlying file.

func (*Writer) Stats

func (w *Writer) Stats() Stats

Stats returns a snapshot of the writer's lifetime counters.

func (*Writer) Sync

func (w *Writer) Sync() error

Sync flushes the buffered frames to the OS and then calls os.File.Sync so the data reaches durable storage before returning. It must be invoked at every transaction commit boundary.

func (*Writer) SyncCtx

func (w *Writer) SyncCtx(ctx context.Context) error

SyncCtx is the context-aware variant of Writer.Sync. ctx.Err() is checked before acquiring the internal mutex; on cancellation returns the wrapped ctx.Err without flushing.

func (*Writer) Truncate

func (w *Writer) Truncate() (int64, error)

Truncate empties the WAL: flushes any buffered frames, truncates the underlying file to zero bytes, and fsyncs the result so the empty state is durable on disk before returning. Subsequent Writer.Append calls write to offset 0 of the freshly-empty file.

Truncate is intended to be called by the checkpointer after a snapshot covering all WAL frames has been durably persisted; on success every frame previously durable in the WAL is logically folded into the snapshot.

Lifetime counters in Writer.Stats are NOT reset; the returned int64 reports the number of bytes that were in the file at the moment of truncation (after the in-memory buffer was flushed), which is the canonical measure of WAL bytes freed by this call.

On error the WAL may be in a partially-truncated state; callers should not continue using the Writer.