spool

Documentation

Overview

Package spool is the gateway's durable accounting spool: an append-only file per process where every batch of usage records is written, and fsync'd, before it is inserted into the store. A store outage or a crash then loses nothing that reached the spool: the records are inserted when the store is back, or by the next process that starts on the same directory. Inserts are idempotent (keyed by the record ID), so replaying a batch that was already written before a crash bills nothing twice.

A spool file is a sequence of frames:

magic 'L' 'F' | length uint32 BE | CRC-32C of the payload, uint32 BE | payload

whose payload is one JSON-encoded store.Record. After a crash the file is untrusted input: a torn tail (an incomplete last frame) is expected and ignored; anything else that does not check out (bad magic, bad CRC, a length beyond MaxFrame, a payload that is not a record) stops the reading there and marks the file corrupt, kept for an operator instead of deleted.

Index

• Constants
• Variables
• func Encode(dst []byte, recs []store.Record) ([]byte, error)
• type Orphan
  • func Orphans(dir, skip string, maxBytes int64) ([]*Orphan, error)
  • func (o *Orphan) Done() error
  • func (o *Orphan) Release()
• type Spool
  • func Open(dir string, max int64) (*Spool, error)
  • func (s *Spool) Append(recs []store.Record) error
  • func (s *Spool) Close() error
  • func (s *Spool) Path() string
  • func (s *Spool) Reset() error
  • func (s *Spool) Size() int64
• type Tail
  • func Read(r io.Reader) ([]store.Record, Tail, error)
  • func (t Tail) String() string

Constants

const MaxFrame = 64 << 10

MaxFrame bounds one frame's payload: a usage record is a few hundred bytes, tags included.

Variables

var (

	// ErrFull is returned by Append when the batch would take the spool
	// beyond its bound.
	ErrFull = errors.New("spool: full")
)

Functions

func Encode

func Encode(dst []byte, recs []store.Record) ([]byte, error)

Encode appends the frames of recs to dst.

Types

type Orphan

type Orphan struct {
	Path    string
	Records []store.Record
	Tail    Tail
	// contains filtered or unexported fields
}

Orphan is a spool file left by a process that is gone.

func Orphans

func Orphans(dir, skip string, maxBytes int64) ([]*Orphan, error)

Orphans returns the spool files of dir that no live process holds (their lock is free), except skip, read and locked for the caller. A file that is not a regular file of this user's making is left alone.

func (*Orphan) Done

func (o *Orphan) Done() error

Done removes a replayed orphan (renamed *.corrupt when its tail was corrupt, so an operator can look at what could not be read).

func (*Orphan) Release

func (o *Orphan) Release()

Release gives an orphan back, unreplayed, for a later attempt.

type Spool

type Spool struct {
	// contains filtered or unexported fields
}

Spool is one process's spool file.

func Open

func Open(dir string, max int64) (*Spool, error)

Open creates dir (0700) if needed and a new spool file in it (0600), locked for this process.

func (*Spool) Append

func (s *Spool) Append(recs []store.Record) error

Append writes recs and fsyncs; ErrFull when they do not fit.

func (*Spool) Close

func (s *Spool) Close() error

Close closes the file, removing it when empty; a file that still holds records stays for the next process to replay.

func (*Spool) Path

func (s *Spool) Path() string

Path is the spool file's path.

func (*Spool) Reset

func (s *Spool) Reset() error

Reset empties the spool: everything it held is stored.

func (*Spool) Size

func (s *Spool) Size() int64

Size is the bytes the spool holds.

type Tail

type Tail int

Tail says how a file ended.

const (
	// Clean: the file ends at a frame boundary.
	Clean Tail = iota
	// Torn: the last frame is incomplete (a write cut by a crash).
	Torn
	// Corrupt: a frame does not check out; what follows was not read.
	Corrupt
)

func Read

func Read(r io.Reader) ([]store.Record, Tail, error)

Read decodes frames from r until its end, a torn tail or a corrupt frame. It never allocates more than MaxFrame for one frame, whatever r holds.

func (Tail) String

func (t Tail) String() string