wal

Documentation

Overview

Package wal provides the implementation of a general purpose write-ahead log.

• The write-ahead log is made up of individual entries, which contain arbitrary data as a payload and metadata in the form of payload length and a checksum.
• Entries are stored in segment files. Each segment file consists of a file header describing some details of the entries stored in the segment. After the file header, the entries follow one after the other.All segment files are assumed to be located in the same directory. Every segment file has the sequence number of its first entry as its file name, padded with leading zeros to be 20 characters in length with a `.wal` file extension.
• The write-ahead log abstracts away the fact that entries are stored in segment files and provides a uniform interface for reading and writing entries without knowing those details.
• Sequence numbers uniquely identify every entry. They are unsigned 64-bit integers which are monotonically increasing.

Index

• Constants
• Variables
• func RegisterMetrics(registerer prometheus.Registerer) error
• type EntryChecksumType
• type EntryLengthEncoding
• type Reader
• type Writer
• type WriterOption

Constants

const (
	EntryChecksumTypeCrc32 = intencoding.EntryChecksumTypeCrc32
	EntryChecksumTypeCrc64 = intencoding.EntryChecksumTypeCrc64
)

const (
	EntryLengthEncodingUint16  = intencoding.EntryLengthEncodingUint16
	EntryLengthEncodingUint32  = intencoding.EntryLengthEncodingUint32
	EntryLengthEncodingUint64  = intencoding.EntryLengthEncodingUint64
	EntryLengthEncodingUvarint = intencoding.EntryLengthEncodingUvarint
)

Variables

var GetSegments = intsegment.GetSegments

GetSegments returns a list of sequence numbers representing the start of the corresponding segment. The sequence numbers are sorted in ascending order.

var Init = intwal.Init

Init initializes a new write-ahead log in the given directory.

var InitIfRequired = intwal.InitIfRequired

InitIfRequired initializes the write-ahead log if it is not yet initialized.

var IsInitialized = intwal.IsInitialized

IsInitialized reports if there is already a write-ahead log available in the given directory.

var NewReader = intwal.NewReader

NewReader creates a new Reader starting at the given sequence number. It will find the segment the sequence number belongs to and read all entries up until the requested sequence number.

var WithEntryChecksumType = intwal.WithEntryChecksumType

WithEntryChecksumType overwrites the default entry checksum type. Can be used with Init and Reader.ToWriter.

var WithEntryLengthEncoding = intwal.WithEntryLengthEncoding

WithEntryLengthEncoding overwrites the default entry length encoding. Can be used with Init and Reader.ToWriter.

var WithMaxSegmentSize = intwal.WithMaxSegmentSize

WithMaxSegmentSize overwrites the default maximum segment size which causes rollover into a new segment when reached. Can be used with Reader.ToWriter.

var WithPreAllocationSize = intwal.WithPreAllocationSize

WithPreAllocationSize overwrites the default pre-allocation size of new segment files. Can be used with Init and Reader.ToWriter.

var WithRolloverCallback = intwal.WithRolloverCallback

WithRolloverCallback sets the given callback for being triggered when the current segment is rolled. Can be used with Reader.ToWriter.

var WithSyncPolicyGrouped = intwal.WithSyncPolicyGrouped

WithSyncPolicyGrouped overwrites the default sync policy with sync policy grouped. Can be used with Reader.ToWriter.

var WithSyncPolicyImmediate = intwal.WithSyncPolicyImmediate

WithSyncPolicyImmediate overwrites the default sync policy with sync policy immediate. Can be used with Reader.ToWriter.

var WithSyncPolicyNone = intwal.WithSyncPolicyNone

WithSyncPolicyNone overwrites the default sync policy with sync policy none. Can be used with Reader.ToWriter.

var WithSyncPolicyPeriodic = intwal.WithSyncPolicyPeriodic

WithSyncPolicyPeriodic overwrites the default sync policy with sync policy periodic. Can be used with Reader.ToWriter.

Functions

func RegisterMetrics

func RegisterMetrics(registerer prometheus.Registerer) error

RegisterMetrics registers all metrics collectors with the given prometheus registerer.

Types

type EntryChecksumType

type EntryChecksumType = intencoding.EntryChecksumType

EntryChecksumType describes the type of checksum applied to an entry.

type EntryLengthEncoding

type EntryLengthEncoding = intencoding.EntryLengthEncoding

EntryLengthEncoding describes the way the length of an entry is encoded.

type Reader

type Reader = intwal.Reader

Reader provides functionality to read the write-ahead log. It abstracts away the fact that the write-ahead log is split into multiple segments.

Instances of this struct are NOT safe for concurrent use. Either use it on a single Go routine or provide your own external synchronization.

type Writer

type Writer = intwal.Writer

Writer provides the main functionality for writing to the write-ahead log. It abstracts away the fact that the WAL is distributed over several segment files and does rollover into new segments as necessary.

Writer is safe to use from multiple Go routines concurrently.

You can only create a writer with the Reader.ToWriter function. This makes sure that you have read all entries before writing to the write-ahead log.

type WriterOption

type WriterOption = intwal.WriterOption

WriterOption describes the function signature which all writer options need to implement.