wal

README

wal

Simple, idiomatic write-ahead log.

Usage

Each write-ahead log requires a dedicated directory for writing log files, writing to a directory containing other files is undefined behaviour. Two methods exist to open a new WAL, Open which uses DefaultSize or OpenSize which allows the user to specify a maximum log file size. The log file writer checks the log file size before writing and advances to the next log file if required, but only if the log is not empty, allowing for writes beyond the specified size.

The maximum log file record size is 4294967295 bytes, or 4GB. If larger chunk size is required, the caller has to write the contents in chunks of 4GB and recombine them later.

Documentation

Overview

Package wal implements a Write-Ahead Log.

Writing logs

Each write-ahead log requires a dedicated directory for writing log files, writing to a directory containing other files is undefined behaviour. Two methods exist to open a new log, `Open` which uses `DefaultSize` or `OpenSize` which allows the user to specify a maximum log file size. The log file writer checks the log file size before writing and advances to the next log file if required, but only if the log is not empty, allowing for writes beyond the specified size.

Reading logs

It's the reader's duty to remember where they left off reading. This allows the wal package to return a read only scanner for each Write-Ahead Log.

Record size

The maximum log file record size is 4GiB. If larger chunk size is required, the caller has to write the contents in chunks of 4GiB and recombine them later. Writes that exceed this limit will result in an error.

Index

• Constants
• type Log
  • func Open(dir string, syncInterval time.Duration) (Log, error)
  • func OpenSize(dir string, syncInterval time.Duration, size int64) (Log, error)
• type Scanner
  • func Read(dir string, start uint64) (Scanner, error)

Examples

• Open
• OpenSize
• Read

Constants

const (
	// DefaultSize is the default maximum size for a log file.
	DefaultSize = 1 << 24 // 16 MB

	// SyncNever skips file syncing.
	SyncNever time.Duration = -1

	// SyncImmediately syncs after each Write method call.
	SyncImmediately time.Duration = 0
)

Types

type Log

type Log interface {
	// First entry in the log.
	First() (sequence uint64, exists bool)

	// Last entry in the log.
	Last() (sequence uint64, exists bool)

	// Scanner for the log.
	Scanner(start uint64) Scanner

	// Write new data to the log.
	Write(p []byte) (n int, err error)

	// Close the log.
	Close() error
}

Log is a Write-Ahead Log implementation. Logs are concurrency safe.

func Open

func Open(dir string, syncInterval time.Duration) (Log, error)

Open a WAL log at the given directory. At most one log may exist in the directory.

Example

package main

import (
	"time"

	"maze.io/x/wal"
)

func main() {
	// Open a write-ahead log in directory "log" with the default log size,
	// syncing every 100ms if there has been a write.
	w, err := wal.Open("log", 100*time.Millisecond)
	if err != nil {
		panic(err)
	}

	// Make sure our WAL is closed.
	defer w.Close()

	// Start writing to the WAL...
}

func OpenSize

func OpenSize(dir string, syncInterval time.Duration, size int64) (Log, error)

OpenSize opens a WAL log at the given directory. At most one log may exist in the directory. The supplied size indicates the maximum log file size. If the size <= 0, the DefaultSize shall be used. The sync interval specifies how often the log is synced to disk. If the sync interval == 0, the log is synced immediately after each write. If the sync interval < 0, the log is never synced.

Example

package main

import (
	"maze.io/x/wal"
)

func main() {
	// Open a write-ahead log in directory "log" with maximum log size of 1GB,
	// syncing after each write.
	w, err := wal.OpenSize("log", wal.SyncImmediately, 1<<30) // 1 GB log size
	if err != nil {
		panic(err)
	}

	// Make sure our WAL is closed.
	defer w.Close()

	// Start writing to the WAL...
}

type Scanner

type Scanner interface {
	// Scan advances the Scanner to the next log record, which will be
	// available through the Bytes method. After Scan returns false, the
	// Err method will return any error that occurred during scanning.
	Scan() bool

	// Bytes returns the most recent token generated by a call to Scan. The
	// underlying array may point to data that will be overwritten by a
	// subsequent call to Scan.
	Bytes() []byte

	// Err returns the last error occurred during scanning. If EOF is
	// returned, no new log entries are available currently.
	Err() error

	// Close the scanner and any logs it may be referencing.
	Close() error
}

Scanner can scan over log records. Scanners are not concurrency safe.

func Read

func Read(dir string, start uint64) (Scanner, error)

Read a write-ahead log at the specified directory.

Example

package main

import (
	"time"

	"maze.io/x/wal"
)

func main() {
	// Open a write-ahead log for reading in directory "log" and continue
	// scanning at entry 1234.
	r, err := wal.Read("log", 1234)
	if err != nil {
		panic(err)
	}

	// Consume all entries that can be read.
	for r.Scan() {
		// Consume r.Bytes() ...
	}

	// Check reader for fatal errors.
	if err = r.Err(); err != nil {
		panic(err)
	}

	// Wait for new data to appear.
	for {
		if r.Scan() {
			// Consume r.Bytes() ...
		} else if err = r.Err(); err != nil {
			// Handle fatal errors.
			panic(err)
		} else {
			// Wait a second and scan again.
			time.Sleep(time.Second)
		}
	}
}