go-wallib

README

go-wallib

A production-ready Write-Ahead Log library for Go: append-only, durable, and crash-safe, with monotonic Log Sequence Numbers, CRC32C-checksummed records, size-rolled segment files, low-water-mark cleanup, and a single-writer Singular Update Queue.

Features

• Durable, ordered appends. Every record is assigned a unique, monotonic, gapless Log Sequence Number (LSN) starting at 1.
• Crash recovery. On Open a torn tail (interrupted final write) is truncated; mid-log corruption is reported as a hard error; an interrupted segment roll (empty trailing segment) is reclaimed. Writer state is fully restored, so the next append continues with no gap.
• Configurable durability via SyncPolicy: SyncImmediate, SyncBatched (group commit), or SyncInterval (periodic background fsync).
• Segmentation & cleanup. The log rolls into size-bounded segments; a record is never split across files. Truncate reclaims whole obsolete segments at a low-water mark.
• Single writer (Singular Update Queue). One goroutine serializes and batches all writes; the API is safe for concurrent use.
• Readers & replay. A forward Reader cursor and Replay callback iterate committed entries from any LSN.
• Standard library only (plus an optional structured logger); no unsafe; all file access is confined to the log directory via os.Root.

Install

go get github.com/barnowlsnest/go-wal/pkg/wal

Requires Go 1.26 or newer.

Usage

package main

import (
	"context"
	"fmt"

	"github.com/barnowlsnest/go-wal/pkg/wal"
)

func main() {
	w, report, err := wal.Open("data/wal", wal.WithSyncPolicy(wal.SyncBatched))
	if err != nil {
		panic(err)
	}
	defer func() { _ = w.Close() }()

	fmt.Printf("recovered %d entries up to LSN %d\n",
		report.EntriesRecovered, report.LastLSN)

	lsn, err := w.Append(context.Background(), []byte(`{"op":"set","key":"k","value":1}`))
	if err != nil {
		panic(err)
	}
	fmt.Println("appended at LSN", lsn)

	// Replay everything from the beginning.
	err = w.Replay(0, func(entry wal.Entry) error {
		fmt.Printf("LSN %d: %s\n", entry.LSN, entry.Payload)
		return nil
	})
	if err != nil {
		panic(err)
	}

	// After persisting a snapshot at lsn, reclaim older segments.
	if err := w.Truncate(lsn); err != nil {
		panic(err)
	}
}

Following the log

Follower is a forward cursor that exposes committed records as a range-over-func iterator (iter.Seq2[uint64, []byte]). It works in two modes:

• Snapshot mode (default) — ends at the tail captured at creation time.
• Follow mode (wal.WithFollow()) — blocks at the tail and resumes as new records commit, like tail -f for the WAL.

follower, err := w.Follower(0, wal.WithFollow()) // 0 = from the beginning
if err != nil {
    panic(err)
}
defer func() { _ = follower.Close() }()

ctx, cancel := context.WithCancel(context.Background())
defer cancel()

for lsn, payload := range follower.Records(ctx) {
    fmt.Printf("LSN %d: %s\n", lsn, payload)
}
if err := follower.Err(); err != nil {
    // handle read error
}

Each payload is a fresh copy — callers may retain it after the loop body returns without risk of aliasing.

Multiple independent followers can tail the log concurrently; each maintains its own cursor and position independently of the writer and of other followers.

select integration. Use RecordsChan to receive entries over a channel, which composes naturally with select:

ch := follower.RecordsChan(ctx)
for entry := range ch {
    fmt.Printf("LSN %d: %s\n", entry.LSN, entry.Payload)
}

Truncation. When Truncate(upTo) reclaims segments that a lagging follower has not yet consumed, the follower's iterator ends and Err() returns wal.ErrTruncated. Check Err() after every loop.

Retention

There is no automatic retention policy. The log grows until the application explicitly reclaims space — there is no TTL, no size-based expiry, and no background purge goroutine.

Retention is caller-driven via Truncate(upToLSN) after you no longer need records at or below that LSN. The usual pattern is to persist a snapshot or checkpoint, then truncate at that LSN:

if err := w.Truncate(snapshotLSN); err != nil {
    panic(err)
}

Truncate deletes whole closed segment files and advances FirstLSN (the low-water mark). The active segment is never deleted, so entries below upToLSN that still live in a surviving segment remain readable — truncation is segment-granular best-effort reclamation, not a precise per-entry delete.

WithMaxSegmentSize only controls when new segment files are created during rolling; it does not delete old data. On Open, recovery may truncate a torn tail or remove empty trailing segments from an interrupted roll; those steps repair crash damage and are not retention policy.

Sync policies

Policy	When the fsync happens	Trade-off
SyncImmediate	before acknowledging each append	strongest durability, slowest
SyncBatched	once per group commit, before acknowledging the batch	high throughput under concurrency
SyncInterval	periodically, by a background goroutine	fastest, bounded data-loss window on crash

Call Sync() at any time to force a flush regardless of policy.

Durability & idempotency

Append is at-least-once. If it returns (lsn, nil), the record is durable per the configured SyncPolicy. If it returns an error, or the process dies before it returns, the record may or may not be durable, and a retry may create a duplicate with a new LSN. Deduplicate using a key embedded in the payload — not the LSN.

A canceled context is honored: an append whose context is done is never committed and never consumes an LSN.

Options

wal.WithSyncPolicy(wal.SyncBatched)      // durability policy (default SyncBatched)
wal.WithMaxSegmentSize(64 << 20)         // soft roll threshold, bytes (default 64 MiB)
wal.WithMaxRecordSize(64 << 20)          // hard per-record limit, bytes (default 64 MiB)
wal.WithBatchSize(256)                    // max appends coalesced per commit
wal.WithBatchTimeout(2 * time.Millisecond)   // group-commit linger (SyncBatched)
wal.WithFlushInterval(100 * time.Millisecond) // background fsync period (SyncInterval)
wal.WithLogger(logger)                    // structured logger (default: no-op)

Logging

wal.Logger is a structured, leveled interface (Debug/Info/Warn/Error(msg string, fields ...wal.Field)) and wal.Field is an alias for go-logslib's logger.Field, so a *logger.Logger satisfies it directly:

w, _, err := wal.Open("data/wal", wal.WithLogger(myLogslibLogger))

If no logger is supplied, a no-op logger is used.

On-disk format

Each segment file begins with a 28-byte header (Magic | Version | Flags | BaseLSN | CreatedAt | HeaderCRC) followed by framed records. Each record is CRC32C(4) | Length(4) | LSN(8) | Payload, with the CRC32C (Castagnoli) computed over Length || LSN || Payload. All integers are little-endian. Segment filenames are the zero-padded base LSN (e.g. 00000000000000000001.wal).

License

MIT