wal

Documentation

Overview

Package wal is the write-ahead log: the durability spine the database commits through (spec 05). It logs LOGICAL operation-batch frames, the exact serialized mutation the db layer later applies to segments, metadata columns, and indexes, so "what is durable" and "what is applied" are byte-identical, and redo during recovery shares one code path with normal operation. A commit frame makes a batch atomic and durable; a chained, salted 64-bit checksum lets recovery find the exact durable tail without trusting any external pointer.

This milestone implements the log, group commit, the synchronous levels, and the durable-tail reader. Physical page-image frames for torn-write protection have their frame type reserved here and are wired in when the checkpoint folds pages in place; the logical redo path is correct on its own because every mutation carries a unique commit version and idempotent page LSNs, so re-applying a committed batch is safe (spec 05, spec 06).

Index

• func Open(fs vfs.FS, path string, opts Options) (*WAL, RecoverResult, error)
• type CommittedBatch
• type Frame
• type FrameType
• type Options
• type RecoverResult
  • func Recover(readAt func(p []byte, off int64) (int, error), size int64) (RecoverResult, error)
  • func (r RecoverResult) CommittedAfter(lsn uint64) []CommittedBatch
• type Sync
• type WAL
  • func Create(fs vfs.FS, path string, opts Options) (*WAL, error)
  • func (w *WAL) Checkpointed(foldedLSN uint64) error
  • func (w *WAL) Close() error
  • func (w *WAL) Commit(version uint64) (uint64, error)
  • func (w *WAL) Flush() error
  • func (w *WAL) LSN() uint64
  • func (w *WAL) LogBatch(version uint64, encoded []byte) error
  • func (w *WAL) Path() string
  • func (w *WAL) Salt() uint64
  • func (w *WAL) Syncs() uint64

Functions

func Open

func Open(fs vfs.FS, path string, opts Options) (*WAL, RecoverResult, error)

Open reopens an existing -wal file and positions the writer to append after the durable tail. It runs the durable-tail scan (Recover) to recover the generation salt, the next LSN, the append offset, and the running checksum, so a frame appended next chains correctly onto the last durable frame and any torn or stale tail is overwritten. The returned RecoverResult carries the committed batches the caller must redo before serving (spec 05). If the file does not exist or its header is unreadable, Open returns an error and the caller falls back to Create.

Types

type CommittedBatch

type CommittedBatch struct {
	Version uint64
	LSN     uint64 // the op-batch frame's LSN
	Encoded []byte // serialized operation batch, decoded by the db layer
}

CommittedBatch is an operation batch whose commit frame was found durable. The recovery driver in the db layer replays these in LSN order through its Apply path (spec 05, spec 06).

type Frame

type Frame struct {
	Type    FrameType
	LSN     uint64
	Version uint64
	Payload []byte
}

Frame is one decoded WAL frame yielded by the reader during recovery.

type FrameType

type FrameType byte

FrameType tags each WAL frame (spec 05).

const (
	// FrameOpBatch carries a serialized operation batch: the logical mutation the
	// db layer replays through Apply during recovery.
	FrameOpBatch FrameType = 1
	// FrameCommit makes everything since the previous commit durable and atomic.
	// Its payload is the frame count of the batch it closes.
	FrameCommit FrameType = 2
	// FrameCheckpoint records that frames up to an LSN have been folded into the
	// main file; writing it rotates the salt for the next WAL generation.
	FrameCheckpoint FrameType = 3
	// FramePageImage is a full physical page image for torn-write protection. Its
	// type is reserved in this milestone; the checkpoint path wires it in later.
	FramePageImage FrameType = 4
)

type Options

type Options struct {
	PageSize int
	Sync     Sync
	// Salt seeds the initial WAL generation. Recovery rotates it at each
	// checkpoint; a caller may pass a fixed value for deterministic tests.
	Salt uint64
}

Options configure a WAL at create/open.

type RecoverResult

type RecoverResult struct {
	// Batches are the committed kv-batches in LSN order, ready to replay.
	Batches []CommittedBatch
	// LastCheckpointLSN is the highest foldedLSN recorded by a durable checkpoint
	// frame, or 0 if none. Frames at or before it are already in the main file.
	LastCheckpointLSN uint64
	// DurableLSN is the LSN of the last frame that chained correctly; the tail is
	// torn or stale beyond it.
	DurableLSN uint64
	// DurableEndOff is the file offset just past the last frame that chained
	// correctly -- the point a resumed writer appends from, overwriting any torn
	// or stale tail (used by wal.Open).
	DurableEndOff int64
	// DurableSum is the running chained checksum at DurableEndOff, the seed the
	// resumed writer's next frame chains from.
	DurableSum uint64
	// Salt is the WAL generation's salt read from the header.
	Salt uint64
	// TornTail is true if the scan stopped at a frame that failed the chain,
	// meaning the file held bytes past the durable region.
	TornTail bool
}

RecoverResult summarizes a recovery scan.

func Recover

func Recover(readAt func(p []byte, off int64) (int, error), size int64) (RecoverResult, error)

Recover walks the -wal file from its header, verifies the chained, salted checksum frame by frame, and returns the committed batches in the durable region. The first frame that fails the chain or carries a stale salt ends the durable log; everything past it is discarded as torn or left over from a previous generation (spec 05). A batch counts as committed only if a checksum-valid commit frame for it is reached (spec 05); a trailing batch with no commit frame is dropped.

readAt reads exactly len(p) bytes at off, or fewer at EOF; it mirrors vfs.File.ReadAt semantics. Passing the WAL file's ReadAt keeps this decoupled from the vfs package.

func (RecoverResult) CommittedAfter

func (r RecoverResult) CommittedAfter(lsn uint64) []CommittedBatch

CommittedAfter returns the committed batches with an LSN strictly greater than lsn, i.e. those not yet folded into the main file at the given checkpoint boundary. The recovery driver replays exactly these.

type Sync

type Sync int

Sync selects how aggressively commits are flushed (spec 05), mirroring SQLite's PRAGMA synchronous.

const (
	// SyncOff never fsyncs the WAL; the OS flushes on its own schedule. No
	// corruption (the checksum chain still holds), but recent commits can be lost.
	SyncOff Sync = iota
	// SyncNormal fdatasyncs at checkpoint and periodically, not every commit. The
	// WAL-mode default: crash-consistent, may lose the most recent commits.
	SyncNormal
	// SyncFull fdatasyncs the WAL on every commit (group-batched). Every acked
	// commit survives power loss.
	SyncFull
	// SyncExtra is SyncFull plus a directory/inode sync on file growth.
	SyncExtra
)

type WAL

type WAL struct {
	// contains filtered or unexported fields
}

WAL is an append-only log over one -wal file. It is not safe for concurrent use by multiple goroutines without external synchronization; the host serializes appends through the commit path (group commit batches concurrent committers above this layer in a later slice).

func Create

func Create(fs vfs.FS, path string, opts Options) (*WAL, error)

Create initializes a fresh -wal file and returns an open WAL positioned to append after the header.

func (*WAL) Checkpointed

func (w *WAL) Checkpointed(foldedLSN uint64) error

Checkpointed appends a checkpoint frame recording that the main file now contains every committed frame through foldedLSN, then rotates the salt so the folded frames cannot be mistaken for current ones on a later recovery. The caller must have already folded and fsynced the main file (spec 05: fold, fsync main, then advance the marker).

func (*WAL) Close

func (w *WAL) Close() error

Close releases the file. It does not sync; the caller checkpoints first for a clean shutdown.

func (*WAL) Commit

func (w *WAL) Commit(version uint64) (uint64, error)

Commit appends a commit frame for version and flushes per the sync level. After it returns at SyncFull/SyncExtra the batch is durable: a crash will redo it. The returned LSN is the commit frame's LSN, which the caller records as the checkpoint boundary once the batch is folded into the main file.

func (*WAL) Flush

func (w *WAL) Flush() error

Flush forces a sync regardless of level, used by NORMAL at checkpoint to finalize the deferred durability backlog (spec 05).

func (*WAL) LSN

func (w *WAL) LSN() uint64

LSN reports the next LSN that will be assigned.

func (*WAL) LogBatch

func (w *WAL) LogBatch(version uint64, encoded []byte) error

LogBatch appends a kv-batch frame carrying the serialized batch. It does not commit; call Commit to make the batch durable and atomic.

func (*WAL) Path

func (w *WAL) Path() string

Path reports the WAL file path.

func (*WAL) Salt

func (w *WAL) Salt() uint64

Salt reports the current WAL generation's salt.

func (*WAL) Syncs

func (w *WAL) Syncs() uint64

Syncs reports how many fsyncs the WAL has performed (observability).