wal

README

wal

A simple and fast write ahead log for Go.

Features

• High durability
• Fast operations
• Monotonic indexes
• Batch writes
• Log truncation from front or back.

Getting Started

Installing

To start using wal, install Go and run go get:

$ go get -u github.com/tidwall/wal

This will retrieve the library.

Example

// open a new log file
log, err := Open("mylog", nil)

// write some entries
err = log.Write(1, []byte("first entry"))
err = log.Write(2, []byte("second entry"))
err = log.Write(3, []byte("third entry"))

// read an entry
data, err := log.Read(1)
println(string(data))  // output: first entry

// close the log
err = log.Close()

Batch writes:

// write three entries as a batch
batch := new(Batch)
batch.Write(1, []byte("first entry"))
batch.Write(2, []byte("second entry"))
batch.Write(3, []byte("third entry"))

err = log.WriteBatch(batch)

Truncating:

// write some entries
err = log.Write(1, []byte("first entry"))
...
err = log.Write(1000, []byte("thousandth entry"))

// truncate the log from index starting 350 and ending with 950.
err = l.TruncateFront(350)
err = l.TruncateBack(950)

Contact

Josh Baker @tidwall

License

wal source code is available under the MIT License.

Documentation

Index

• Variables
• type Batch
  • func (b *Batch) Clear()
  • func (b *Batch) Write(index uint64, data []byte)
• type File
  • func NewReader(name string, flag int, perm os.FileMode) (File, error)
  • func NewWriter(name string, flag int, perm os.FileMode) (File, error)
• type Log
  • func Open(path string, opts *Options) (*Log, error)
  • func (l *Log) ClearCache() error
  • func (l *Log) Close() error
  • func (l *Log) FirstIndex() (index uint64, err error)
  • func (l *Log) LastIndex() (index uint64, err error)
  • func (l *Log) Read(index uint64) (data []byte, err error)
  • func (l *Log) Sync() error
  • func (l *Log) TruncateBack(index uint64) error
  • func (l *Log) TruncateFront(index uint64) error
  • func (l *Log) Write(index uint64, data []byte) error
  • func (l *Log) WriteBatch(b *Batch) error
• type LogFormat
• type Options

Variables

var (
	// ErrCorrupt is returns when the log is corrupt.
	ErrCorrupt = errors.New("log corrupt")

	// ErrClosed is returned when an operation cannot be completed because
	// the log is closed.
	ErrClosed = errors.New("log closed")

	// ErrNotFound is returned when an entry is not found.
	ErrNotFound = errors.New("not found")

	// ErrOutOfOrder is returned from Write() when the index is not equal to
	// LastIndex()+1. It's required that log monotonically grows by one and has
	// no gaps. Thus, the series 10,11,12,13,14 is valid, but 10,11,13,14 is
	// not because there's a gap between 11 and 13. Also, 10,12,11,13 is not
	// valid because 12 and 11 are out of order.
	ErrOutOfOrder = errors.New("out of order")

	// ErrOutOfRange is returned from TruncateFront() and TruncateBack() when
	// the index not in the range of the log's first and last index. Or, this
	// may be returned when the caller is attempting to remove *all* entries;
	// The log requires that at least one entry exists following a truncate.
	ErrOutOfRange = errors.New("out of range")
)

var DefaultOptions = &Options{
	NoSync:           false,
	SegmentSize:      20971520,
	LogFormat:        Binary,
	SegmentCacheSize: 2,
	NoCopy:           false,
	DirPerms:         0750,
	FilePerms:        0640,
}

DefaultOptions for Open().

Types

type Batch

type Batch struct {
	// contains filtered or unexported fields
}

Batch of entries. Used to write multiple entries at once using WriteBatch().

func (*Batch) Clear

func (b *Batch) Clear()

Clear the batch for reuse.

func (*Batch) Write

func (b *Batch) Write(index uint64, data []byte)

Write an entry to the batch

type File

type File interface {
	io.Seeker
	io.Reader
	io.Writer
	io.Closer
	Sync() error
}

File is a file interface

func NewReader

func NewReader(name string, flag int, perm os.FileMode) (File, error)

func NewWriter

func NewWriter(name string, flag int, perm os.FileMode) (File, error)

type Log

type Log struct {
	// contains filtered or unexported fields
}

Log represents a write ahead log

func Open

func Open(path string, opts *Options) (*Log, error)

Open a new write ahead log

func (*Log) ClearCache

func (l *Log) ClearCache() error

ClearCache clears the segment cache

func (*Log) Close

func (l *Log) Close() error

Close the log.

func (*Log) FirstIndex

func (l *Log) FirstIndex() (index uint64, err error)

FirstIndex returns the index of the first entry in the log. Returns zero when log has no entries.

func (*Log) LastIndex

func (l *Log) LastIndex() (index uint64, err error)

LastIndex returns the index of the last entry in the log. Returns zero when log has no entries.

func (*Log) Read

func (l *Log) Read(index uint64) (data []byte, err error)

Read an entry from the log. Returns a byte slice containing the data entry.

func (*Log) Sync

func (l *Log) Sync() error

Sync performs an fsync on the log. This is not necessary when the NoSync option is set to false.

func (*Log) TruncateBack

func (l *Log) TruncateBack(index uint64) error

TruncateBack truncates the back of the log by removing all entries that are after the provided `index`. In other words the entry at `index` becomes the last entry in the log.

func (*Log) TruncateFront

func (l *Log) TruncateFront(index uint64) error

TruncateFront truncates the front of the log by removing all entries that are before the provided `index`. In other words the entry at `index` becomes the first entry in the log.

func (*Log) Write

func (l *Log) Write(index uint64, data []byte) error

Write an entry to the log.

func (*Log) WriteBatch

func (l *Log) WriteBatch(b *Batch) error

WriteBatch writes the entries in the batch to the log in the order that they were added to the batch. The batch is cleared upon a successful return.

type LogFormat

type LogFormat byte

LogFormat is the format of the log files.

const (
	// Binary format writes entries in binary. This is the default and, unless
	// a good reason otherwise, should be used in production.
	Binary LogFormat = 0
	// JSON format writes entries as JSON lines. This causes larger, human
	// readable files.
	JSON LogFormat = 1
)

type Options

type Options struct {
	// NoSync disables fsync after writes. This is less durable and puts the
	// log at risk of data loss when there's a server crash.
	NoSync bool
	// SegmentSize of each segment. This is just a target value, actual size
	// may differ. Default is 20 MB.
	SegmentSize int
	// LogFormat is the format of the log files. Default is Binary.
	LogFormat LogFormat
	// SegmentCacheSize is the maximum number of segments that will be held in
	// memory for caching. Increasing this value may enhance performance for
	// concurrent read operations. Default is 1
	SegmentCacheSize int
	// NoCopy allows for the Read() operation to return the raw underlying data
	// slice. This is an optimization to help minimize allocations. When this
	// option is set, do not modify the returned data because it may affect
	// other Read calls. Default false
	NoCopy bool
	// Perms represents the datafiles modes and permission bits
	DirPerms  os.FileMode
	FilePerms os.FileMode
}

Options for Log