wal

README

wal

Write ahead log for Go.

Features

• High durability
• Fast writes
• Low memory footprint
• Monotonic indexes
• Log truncation from front or back.

Getting Started

Installing

To start using wal, install Go and run go get:

$ go get -u github.com/tidwall/wal

This will retrieve the library.

Example

// open a new log file
l, _ := Open("mylog", nil)

// write some entries
l.Write(1, []byte("first entry"))
l.Write(2, []byte("second entry"))
l.Write(3, []byte("third entry"))

// read an entry
data, _ := l.Read(1)
println(string(data))  // output: first entry

// close the log
l.Close()

Contact

Josh Baker @tidwall

License

wal source code is available under the MIT License.

Documentation

Index

• Variables
• type Batch
  • func (b *Batch) Clear()
  • func (b *Batch) Write(index uint64, data []byte)
• type Durability
• type Log
  • func Open(path string, opts *Options) (*Log, error)
  • func (l *Log) Close() error
  • func (l *Log) FirstIndex() (index uint64, err error)
  • func (l *Log) LastIndex() (index uint64, err error)
  • func (l *Log) Read(index uint64) (data []byte, err error)
  • func (l *Log) Sync()
  • func (l *Log) TruncateBack(lastIndex uint64) error
  • func (l *Log) TruncateFront(firstIndex uint64) error
  • func (l *Log) Write(index uint64, data []byte) error
  • func (l *Log) WriteBatch(b *Batch) error
• type LogFormat
• type Options

Variables

var (
	// ErrCorrupt is returns when the log is corrupt.
	ErrCorrupt = errors.New("log corrupt")

	// ErrClosed is returned when an operation cannot be completed because
	// the log is closed.
	ErrClosed = errors.New("log closed")

	// ErrNotFound is returned when an entry is not found.
	ErrNotFound = errors.New("not found")

	// ErrOutOfOrder is returned from Write() when the index is not equal to
	// LastIndex()+1. It's required that log monotonically grows by one and has
	// no gaps. Thus, the series 10,11,12,13,14 is valid, but 10,11,13,14 is
	// not because there's a gap between 11 and 13. Also, 10,12,11,13 is not
	// valid because 12 and 11 are out of order.
	ErrOutOfOrder = errors.New("out of order")

	// ErrOutOfRange is returned from TruncateFront() and TruncateBack() when
	// the index not in the range of the log's first and last index. Or, this
	// may be returned when the caller is attempting to remove *all* entries;
	// The log requires that at least one entry exists following a truncate.
	ErrOutOfRange = errors.New("out of range")
)

var DefaultOptions = &Options{
	Durability:  High,
	SegmentSize: 52428800,
	LogFormat:   Binary,
}

DefaultOptions for Open().

Types

type Batch

type Batch struct {
	// contains filtered or unexported fields
}

Batch of entries. Used to write multiple entries at once using WriteBatch().

func (*Batch) Clear

func (b *Batch) Clear()

Clear the batch for reuse.

func (*Batch) Write

func (b *Batch) Write(index uint64, data []byte)

Write an entry to the batch

type Durability

type Durability int8

Durability policy.

const (
	// Low durability flushes data to operating system only when the buffer is
	// at capacity. A process crash could cause data loss.
	Low Durability = -1
	// Medium durability flushes data to the operating system after every
	// new entry is addded to the log. A server crash could cause data loss.
	Medium Durability = 0
	// High durability syncs data to disk after every new entry is added to
	// the log. All entries are persisted to disk and crashes will not cause
	// data loss.
	High Durability = 1
)

type Log

type Log struct {
	// contains filtered or unexported fields
}

Log represents a write ahead log

func Open

func Open(path string, opts *Options) (*Log, error)

Open a new write ahead log

func (*Log) Close

func (l *Log) Close() error

Close the log

func (*Log) FirstIndex

func (l *Log) FirstIndex() (index uint64, err error)

FirstIndex returns the index of the first entry in the log. Returns zero when log has no entries.

func (*Log) LastIndex

func (l *Log) LastIndex() (index uint64, err error)

LastIndex returns the index of the last entry in the log. Returns zero when log has no entries.

func (*Log) Read

func (l *Log) Read(index uint64) (data []byte, err error)

Read an entry from the log. This function reads an entry from disk and is optimized for sequential reads. Randomly accessing entries is slow.

func (*Log) Sync

func (l *Log) Sync()

Sync performs an fsync on the log. This is not necessary when the durability is set to High.

func (*Log) TruncateBack

func (l *Log) TruncateBack(lastIndex uint64) error

TruncateBack truncates the back of the log by removing all entries that are after the provided `lastIndex`. In other words the entry at `lastIndex` becomes the last entry in the log.

func (*Log) TruncateFront

func (l *Log) TruncateFront(firstIndex uint64) error

TruncateFront truncates the front of the log by removing all entries that are before the provided `firstIndex`. In other words the entry at `firstIndex` becomes the first entry in the log.

func (*Log) Write

func (l *Log) Write(index uint64, data []byte) error

Write an entry to the log.

func (*Log) WriteBatch

func (l *Log) WriteBatch(b *Batch) error

WriteBatch writes the entries in the batch to the log in the order that they were added to the batch. The batch is cleared upon a successful return.

type LogFormat

type LogFormat byte

LogFormat is the format of the log files.

const (
	// Binary format writes entries in binary. This is the default and, unless
	// a good reason otherwise, should be used in production.
	Binary LogFormat = 0
	// JSON format writes entries as JSON lines. This causes larger, human
	// readable files.
	JSON LogFormat = 1
)

type Options

type Options struct {
	// Durability policy. Default is High.
	Durability Durability
	// SegmentSize of each segment. This is just a target value, actual size
	// may differ. Default is 50 MB.
	SegmentSize int
	// LogFormat is the format of the log files. Default is Binary.
	LogFormat LogFormat
}

Options for Log