wal

README

WAL: High-Performance Write-Ahead Log in Go

A high-performance, concurrent-safe, and crash-resilient Write-Ahead Log (WAL) library for Go. Designed for building databases, message queues, or any system requiring data durability.

Features

• 🚀 High Performance: Buffered I/O, optimized locking strategies and non-blocking background sync.
• 🛡️ Data Integrity: CRC32 checksums (Castagnoli), append-only logic and automatic corruption repair on startup.
• 🧵 Concurrency Safe: Thread-safe writers and readers using sync.RWMutex and atomic operations.
• 🔄 Log Rotation: Automatic segment rotation based on configurable size.
• 💾 Flexible Sync Strategies: Choose between Performance (Background), Safety (Always), or Balance (OSCache).
• 🔍 Iterator API: Memory-efficient sequential reading of logs.
• 🧵 Crash Safety: Automatic recovery from power failures. Detects and truncates corrupted log tails (partial writes) on startup.

Architecture

On-Disk Format

Each segment file consists of a sequence of binary encoded entries.

+-------------------+-------------------+-------------------+----------------------+
|   CRC32 (4 bytes) |   Size (8 bytes)  |  Offset (8 bytes) |   Payload (N bytes)  |
+-------------------+-------------------+-------------------+----------------------+
| Checksum of Data  | Length of Payload | Logical Position  | The actual data      |
+-------------------+-------------------+-------------------+----------------------+

• CRC (Cyclic Redundancy Check): Ensures data integrity.
• Size & Offset: Enable fast reading without parsing the entire file.
• Payload: The actual data.

Installation

go get github.com/hungpdn/wal

Usage

Writing Data

package main

import (
 "log"
 "github.com/hungpdn/wal"
)

func main() {
 cfg := wal.Config{
    WALDir:       "./wal_data",
    SegmentSize:  10 * 1024 * 1024, // 10MB
    SyncStrategy: wal.SyncStrategyOSCache,
 }

 w, _ := wal.New(cfg)
 defer w.Close()

 // Write data
 w.Write([]byte("Hello WAL"))
 w.Write([]byte("Another log entry"))
}

Reading Data (Replay)

w, _ := wal.New(cfg) // Auto-recovers on open

iter, _ := w.NewReader()
defer iter.Close()

for iter.Next() {
   data := iter.Value()
   log.Printf("Log Data: %s", string(data))
}

if err := iter.Err(); err != nil {
   log.Printf("Error reading wal: %v", err)
}

Configuration

Field	Type	Default	Description
WALDir	string	./wal	Directory to store segment files.
SegmentSize	int64	10MB	Max size of a single segment file before rotation.
BufferSize	int	4KB	Size of the in-memory buffer.
SyncStrategy	int	Background	0: Background, 1: Always (Fsync), 2: OSCache (Recm).
SyncInterval	uint	1000ms	Interval for background sync execution.

Sync Strategies

• SyncStrategyBackground (0): Fastest. Writes to buffer. OS handles disk sync. Risk of data loss on OS crash.
• SyncStrategyAlways (1): Safest. fsync on every write. Slowest performance.
• SyncStrategyOSCache (2): Recommended. Flushes to OS cache immediately, background fsync every interval. Safe against app crashes, slight risk on power loss.

Contributing

Contributions are welcome! Please fork the repository and open a pull request.

1. Fork the Project.
2. Create your Feature Branch (git checkout -b feature/AmazingFeature)
3. Commit your Changes (git commit -m 'Add some AmazingFeature')
4. Push to the Branch (git push origin feature/AmazingFeature)
5. Open a Pull Request

License

MIT License. See LICENSE file.

TODO

• Log level, metrics
• Tests
• Retention policy (remove, upload to s3, gcs, etc)
• Index
• CI
• Benchmarks
• Documentation
• Open source template (makefile, license, code of conduct, contributing, etc)
• Distributed Replication

Reference

• tidwall/wal
• Log: What Every Software Engineer Should Know About Real-Time Data's Unifying Abstraction

Documentation

Index

• Constants
• Variables
• type Config
  • func (cfg *Config) SetDefault()
• type Iterator
  • func (it *Iterator) Close() error
  • func (it *Iterator) Err() error
  • func (it *Iterator) Next() bool
  • func (it *Iterator) Value() []byte
• type Reader
• type Segment
• type SyncStrategy
• type WAL
  • func New(cfg Config) (*WAL, error)
  • func (w *WAL) Close() error
  • func (w *WAL) NewReader() (*Iterator, error)
  • func (w *WAL) StopSync()
  • func (w *WAL) Sync() error
  • func (w *WAL) Write(payload []byte) error

Constants

const (
	KB = 1024    // 1 Kilobyte
	MB = KB * KB // 1 Megabyte
)

Variables

var DefaultConfig = Config{
	WALDir:        "./wal",
	BufferSize:    4 * KB,
	SegmentSize:   10 * MB,
	SegmentPrefix: "segment",
	SyncStrategy:  SyncStrategyBackground,
	SyncInterval:  1000,
}

DefaultConfig provides default configuration values for WAL.

var (
	ErrInvalidCRC = errors.New("wal: invalid crc, data corruption detected")
)

Types

type Config

type Config struct {
	WALDir        string       // Directory to store WAL files
	BufferSize    int          // Buffered writes size in bytes (e.g., 4KB)
	SegmentSize   int64        // Maximum size of each file (e.g., 10MB)
	SegmentPrefix string       // Prefix for segment file names (e.g., "segment")
	SyncStrategy  SyncStrategy // Sync strategy
	SyncInterval  uint         // Sync interval in milliseconds for background sync
}

Config holds the configuration for WAL.

func (*Config) SetDefault

func (cfg *Config) SetDefault()

SetDefault sets default values for any zero-value fields in the Config.

type Iterator

type Iterator struct {
	// contains filtered or unexported fields
}

Iterator: Implementation of Reader

func (*Iterator) Close

func (it *Iterator) Close() error

Close: Close the current file and release its resources

func (*Iterator) Err

func (it *Iterator) Err() error

Err: Returns an error if the browsing process is interrupted

func (*Iterator) Next

func (it *Iterator) Next() bool

Next: Read the next entry. Returns true if successful, false if no more data or error.

func (*Iterator) Value

func (it *Iterator) Value() []byte

Value: Get the data of the current log entry

type Reader

type Reader interface {
	Next() bool    // Move to the next log
	Value() []byte // Get the data of the current log
	Err() error    // Get error if any
	Close() error  // Close the reader
}

Reader: Interface to read logs sequentially

type Segment

type Segment struct {
	// contains filtered or unexported fields
}

Segment: Represents a physical file

type SyncStrategy

type SyncStrategy int

SyncStrategy defines the synchronization strategy for WAL.

const (
	// Fastest but highest risk, only write to buffer, Flush and Sync every second.
	// Data loss if app crash or OS crash.
	// Default strategy.
	SyncStrategyBackground SyncStrategy = 0

	// The safest, Flush and Sync immediately on each Write.
	// Never lose data. Slowest.
	SyncStrategyAlways SyncStrategy = 1

	// Balanced, Flush to OS Cache immediately, but Sync every second. (Recommended)
	// Safe on app crash. Only lose data on OS crash/power loss.
	SyncStrategyOSCache SyncStrategy = 2
)

type WAL

type WAL struct {
	// contains filtered or unexported fields
}

WAL: Write-Ahead Log structure

func New

func New(cfg Config) (*WAL, error)

NewWAL: Initialize and recover data

func (*WAL) Close

func (w *WAL) Close() error

Close: Safely close the WAL

func (*WAL) NewReader

func (w *WAL) NewReader() (*Iterator, error)

NewReader: Create an Iterator starting from the oldest segment

func (*WAL) StopSync

func (w *WAL) StopSync()

StopSync stops sync goroutine and WAITS for it to finish

func (*WAL) Sync

func (w *WAL) Sync() error

Sync: Public method to sync data to disk Caller does not need to hold the lock

func (*WAL) Write

func (w *WAL) Write(payload []byte) error

Write: Append a new entry to the WAL