encoding

Documentation

Overview

Package encoding provides functionality for encoding and decoding data to disk.

Index

• Constants
• Variables
• func ReadEntryChecksumCrc32(reader io.Reader, buffer []byte, data []byte) (int, error)
• func ReadEntryChecksumCrc64(reader io.Reader, buffer []byte, data []byte) (int, error)
• func ReadEntryLengthUint16(reader io.Reader, buffer []byte) (uint64, int, error)
• func ReadEntryLengthUint32(reader io.Reader, buffer []byte) (uint64, int, error)
• func ReadEntryLengthUint64(reader io.Reader, buffer []byte) (uint64, int, error)
• func ReadEntryLengthUvarint(reader io.Reader, buffer []byte) (uint64, int, error)
• func ReadUvarint(r io.Reader, buffer []byte) (uint64, int, error)
• func WriteEntryChecksumCrc32(writer io.Writer, buffer []byte, data []byte) error
• func WriteEntryChecksumCrc64(writer io.Writer, buffer []byte, data []byte) error
• func WriteEntryLengthUint16(writer io.Writer, buffer []byte, length uint64) error
• func WriteEntryLengthUint32(writer io.Writer, buffer []byte, length uint64) error
• func WriteEntryLengthUint64(writer io.Writer, buffer []byte, length uint64) error
• func WriteEntryLengthUvarint(writer io.Writer, buffer []byte, length uint64) error
• func WriteHeader(writer io.Writer, buffer []byte, header Header) error
• type EntryChecksumReader
  • func GetEntryChecksumReader(entryChecksumType EntryChecksumType) (EntryChecksumReader, error)
• type EntryChecksumType
  • func (e EntryChecksumType) String() string
• type EntryChecksumWriter
  • func GetEntryChecksumWriter(entryChecksumType EntryChecksumType) (EntryChecksumWriter, error)
• type EntryLengthEncoding
  • func (e EntryLengthEncoding) String() string
• type EntryLengthReader
  • func GetEntryLengthReader(entryLengthEncoding EntryLengthEncoding) (EntryLengthReader, error)
• type EntryLengthWriter
  • func GetEntryLengthWriter(entryLengthEncoding EntryLengthEncoding) (EntryLengthWriter, error)
• type Header
  • func ReadHeader(reader io.Reader, buffer []byte) (Header, error)

Constants

const DefaultEntryChecksumType = EntryChecksumTypeCrc32

DefaultEntryChecksumType is the checksum type which should work fine for most use cases.

const DefaultEntryLengthEncoding = EntryLengthEncodingUint32

DefaultEntryLengthEncoding is the length encoding which should work fine for most use cases.

const HeaderSize = 4 + 2 + 1 + 1 + 8

HeaderSize provides the size in bytes of the header. Helpful for reading the full header before decoding individual elements.

const HeaderVersion = 1

HeaderVersion provides the currently supported header version.

const MaxChecksumBufferLen = crc64.Size

MaxChecksumBufferLen is the size of the buffer which is big enough for all supported checksum types.

const MaxLengthBufferLen = binary.MaxVarintLen64

MaxLengthBufferLen is the size of the buffer which is big enough for all supported length encodings.

Variables

var (
	ErrEntryChecksumTypeUnsupported = errors.New("unsupported WAL entry checksum type")
	ErrEntryChecksumMismatch        = errors.New("WAL entry checksum mismatch")
)

var (
	ErrEntryLengthEncodingUnsupported = errors.New("unsupported WAL entry length encoding")
	ErrEntryLengthOverflow            = errors.New("WAL entry length overflow")
)

var (
	ErrHeaderInvalidMagicBytes  = errors.New("invalid WAL header magic bytes")
	ErrHeaderUnsupportedVersion = errors.New("unsupported WAL header version")
)

var DefaultHeader = Header{
	Magic:               Magic,
	Version:             HeaderVersion,
	EntryLengthEncoding: DefaultEntryLengthEncoding,
	EntryChecksumType:   DefaultEntryChecksumType,
	FirstSequenceNumber: 0,
}

DefaultHeader provides a header configuration which is a sane default in most situations.

var Endian = binary.LittleEndian

Endian is the endianness the write-ahead log uses for serializing/deserializing integers to file.

var EntryChecksumTypes = []EntryChecksumType{
	EntryChecksumTypeCrc32,
	EntryChecksumTypeCrc64,
}

EntryChecksumTypes provides a list of supported checksum types. Helpful for writing tests and benchmarks which iterate over all possibilities.

var EntryLengthEncodings = []EntryLengthEncoding{
	EntryLengthEncodingUint16,
	EntryLengthEncodingUint32,
	EntryLengthEncodingUint64,
	EntryLengthEncodingUvarint,
}

EntryLengthEncodings provides a list of supported length encodings. Helpful for writing tests and benchmarks which iterate over all possibilities.

var Magic = [4]byte{'W', 'A', 'L', 0}

Magic holds the magic bytes expected at the start of the file.

Functions

func ReadEntryChecksumCrc32

func ReadEntryChecksumCrc32(reader io.Reader, buffer []byte, data []byte) (int, error)

ReadEntryChecksumCrc32 reads the checksum from the reader as uint32. The buffer is required to avoid allocations and should be big enough to hold the checksum temporarily. The data is the data to calculate the checksum over and compare to the checksum which was read. The return value is the number of bytes read from reader.

func ReadEntryChecksumCrc64

func ReadEntryChecksumCrc64(reader io.Reader, buffer []byte, data []byte) (int, error)

ReadEntryChecksumCrc64 reads the checksum from the reader as uint64. The buffer is required to avoid allocations and should be big enough to hold the checksum temporarily. The data is the data to calculate the checksum over and compare to the checksum which was read. The return value is the number of bytes read from reader.

func ReadEntryLengthUint16

func ReadEntryLengthUint16(reader io.Reader, buffer []byte) (uint64, int, error)

ReadEntryLengthUint16 reads the length from the reader encoded as uint16. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. The return value is the length decoded from reader and the number of bytes read.

func ReadEntryLengthUint32

func ReadEntryLengthUint32(reader io.Reader, buffer []byte) (uint64, int, error)

ReadEntryLengthUint32 reads the length from the reader encoded as uint32. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. The return value is the length decoded from reader and the number of bytes read.

func ReadEntryLengthUint64

func ReadEntryLengthUint64(reader io.Reader, buffer []byte) (uint64, int, error)

ReadEntryLengthUint64 reads the length from the reader encoded as uint64. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. The return value is the length decoded from reader and the number of bytes read.

func ReadEntryLengthUvarint

func ReadEntryLengthUvarint(reader io.Reader, buffer []byte) (uint64, int, error)

ReadEntryLengthUvarint reads the length from the reader encoded as uvarint. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. The return value is the length decoded from reader and the number of bytes read.

func ReadUvarint

func ReadUvarint(r io.Reader, buffer []byte) (uint64, int, error)

ReadUvarint reads an encoded unsigned integer from r and returns it as a uint64. The error is io.EOF only if no bytes were read. If an io.EOF happens after reading some but not all the bytes, ReadUvarint returns io.ErrUnexpectedEOF.

This is a modification of binary.ReadUvarint to work with an io.Reader instead of an io.ByteReader, to use the buffer for scratch space and to return the number of bytes read. These modifications significantly improve performance by a factor of over 10 and do not require any memory allocations.

func WriteEntryChecksumCrc32

func WriteEntryChecksumCrc32(writer io.Writer, buffer []byte, data []byte) error

WriteEntryChecksumCrc32 writes the checksum to the writer as uint32. The buffer is required to avoid allocations and should be big enough to hold the checksum temporarily. The data is the data to calculate the checksum over.

func WriteEntryChecksumCrc64

func WriteEntryChecksumCrc64(writer io.Writer, buffer []byte, data []byte) error

WriteEntryChecksumCrc64 writes the checksum to the writer as uint64. The buffer is required to avoid allocations and should be big enough to hold the checksum temporarily. The data is the data to calculate the checksum over.

func WriteEntryLengthUint16

func WriteEntryLengthUint16(writer io.Writer, buffer []byte, length uint64) error

WriteEntryLengthUint16 writes the length to the writer encoded as uint16. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. An error is returned when the given length exceeds the maximum possible length.

func WriteEntryLengthUint32

func WriteEntryLengthUint32(writer io.Writer, buffer []byte, length uint64) error

WriteEntryLengthUint32 writes the length to the writer encoded as uint32. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily. An error is returned when the given length exceeds the maximum possible length.

func WriteEntryLengthUint64

func WriteEntryLengthUint64(writer io.Writer, buffer []byte, length uint64) error

WriteEntryLengthUint64 writes the length to the writer encoded as uint64. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily.

func WriteEntryLengthUvarint

func WriteEntryLengthUvarint(writer io.Writer, buffer []byte, length uint64) error

WriteEntryLengthUvarint writes the length to the writer encoded as uvarint. The buffer is required to avoid allocations and should be big enough to hold the encoded length temporarily.

func WriteHeader

func WriteHeader(writer io.Writer, buffer []byte, header Header) error

WriteHeader writes the segment header to the writer. The buffer is required to avoid allocations and should be big enough to hold the full header temporarily.

Types

type EntryChecksumReader

type EntryChecksumReader func(reader io.Reader, buffer []byte, data []byte) (int, error)

EntryChecksumReader is the function signature which all entry checksum reader functions need to implement. reader is the source to read the checksum from. buffer is a temporary scratch space for converting slices of bytes to integers without having to allocate memory. data is the data to calculate the checksum over and compare with the checksum read from reader. The return values are the number of bytes read and any error which occurred during reading.

func GetEntryChecksumReader

func GetEntryChecksumReader(entryChecksumType EntryChecksumType) (EntryChecksumReader, error)

GetEntryChecksumReader returns the entry checksum reader function matching the entry checksum type.

type EntryChecksumType

type EntryChecksumType int

EntryChecksumType describes the type of checksum applied to an entry.

const (
	EntryChecksumTypeCrc32 EntryChecksumType = iota + 1 // We do not start at 0 to detect missing values.
	EntryChecksumTypeCrc64
)

func (EntryChecksumType) String

func (e EntryChecksumType) String() string

String returns a string representation of the checksum.

type EntryChecksumWriter

type EntryChecksumWriter func(writer io.Writer, buffer []byte, data []byte) error

EntryChecksumWriter is the function signature which all entry checksum writer functions need to implement. writer is the destination to write the checksum to. buffer is a temporary scratch space for converting integers to slices of bytes without having to allocate memory. data is the data to actually compute the checksum over.

func GetEntryChecksumWriter

func GetEntryChecksumWriter(entryChecksumType EntryChecksumType) (EntryChecksumWriter, error)

GetEntryChecksumWriter returns the entry checksum writer function matching the entry checksum type.

type EntryLengthEncoding

type EntryLengthEncoding int

EntryLengthEncoding describes the way the length of an entry is encoded.

const (
	EntryLengthEncodingUint16 EntryLengthEncoding = iota + 1 // We do not start at 0 to detect missing values.
	EntryLengthEncodingUint32
	EntryLengthEncodingUint64
	EntryLengthEncodingUvarint
)

func (EntryLengthEncoding) String

func (e EntryLengthEncoding) String() string

String returns a string representation of the length encoding.

type EntryLengthReader

type EntryLengthReader func(reader io.Reader, buffer []byte) (uint64, int, error)

EntryLengthReader is the function signature which all entry length reader functions need to implement. reader is the source to read the length from. buffer is a temporary scratch space for converting slices of bytes to integers without having to allocate memory. The return values are the number of bytes read and any error which occurred during reading.

func GetEntryLengthReader

func GetEntryLengthReader(entryLengthEncoding EntryLengthEncoding) (EntryLengthReader, error)

GetEntryLengthReader returns the entry length reader function matching the entry length encoding.

type EntryLengthWriter

type EntryLengthWriter func(writer io.Writer, buffer []byte, length uint64) error

EntryLengthWriter is the function signature which all entry length writer functions need to implement. writer is the destination to write the length to. buffer is a temporary scratch space for converting integers to slices of bytes without having to allocate memory. length is the length to encode.

func GetEntryLengthWriter

func GetEntryLengthWriter(entryLengthEncoding EntryLengthEncoding) (EntryLengthWriter, error)

GetEntryLengthWriter returns the entry length writer function matching the entry length encoding.

type Header

type Header struct {
	// These are the magic bytes to identify a segment file. This must always be "WAL" followed by a zero value byte.
	// Encoded as four bytes.
	Magic [4]byte

	// The version of the segment file format to use. This allows us to evolve the header and the file format over
	// time if necessary. Encoded as two bytes.
	Version uint16

	// Describes the way the entry length is encoded in the segment file. Encoded as a single byte.
	EntryLengthEncoding EntryLengthEncoding

	// Describes the way the entry checksum is encoded in the segment file. Encoded as a single byte.
	EntryChecksumType EntryChecksumType

	// The sequence number the first entry in the segment file has. Note that the file name and this header value
	// should always match. To have the first sequence number stored in the header makes it possible to detect
	// accidental file renames.
	// Encoded as eight bytes.
	FirstSequenceNumber uint64
}

Header describes the segment file header which is located at the start of every segment file.

func ReadHeader

func ReadHeader(reader io.Reader, buffer []byte) (Header, error)

ReadHeader reads the segment header from the reader. The buffer is required to avoid allocations and should be big enough to hold the full header temporarily. An error is returned when the header does not match expectations (like magic bytes, version, etc.).