wal

Documentation

Overview

Package wal is the agent's write-ahead log: an atomic JSON snapshot of state.GlobalState, flushed every 60s and read back on boot to seed state before the scraper starts.

Two things a change here must preserve:

• u64 fields cross the wire as decimal strings. JSON numbers are doubles and lose precision above 2^53, which is real byte counts.
• The write is tmp → fsync → rename → fsync the parent directory. Skipping the directory fsync leaves the renames unjournaled, so a power loss can roll back to the previous snapshot.

docs/architecture/primer.md#write-ahead-log

Index

• Constants
• Variables
• func EnsureDir(path string) error
• func Quarantine(path string) (moved bool, err error)
• func Save(path, agentBuild string, records []state.Record, ...) error
• type LoadResult
  • func Load(path string) (LoadResult, error)
• type LoadSource
• type Metrics
  • func NewMetrics() *Metrics
  • func (m *Metrics) Collectors() []prometheus.Collector
  • func (m *Metrics) ObserveCopy(d time.Duration)
  • func (m *Metrics) RecordLoadFallback(from string)
  • func (m *Metrics) SetCountersReset(unixSeconds int64)

Constants

const (
	BackupSuffix     = ".bak"
	TempSuffix       = ".tmp"
	QuarantineSuffix = ".quarantine"
)

BackupSuffix is appended to the WAL path for the rotated-aside previous snapshot. TempSuffix is the in-progress write target. QuarantineSuffix is where Quarantine preserves an unreadable snapshot out of the Save rotation's reach.

const (
	StageMarshal       = "marshal"
	StageWrite         = "write"
	StageFsync         = "fsync"
	StageRenameBak     = "rename_bak"
	StageRenameCurrent = "rename_current"
	StageDirSync       = "dir_sync"
)

Stage labels for lachesis_wal_flush_failures_total{stage=...}. The label value set is part of the package's wire contract — operators query and dashboard against these strings — so they live here as exported consts rather than open-coded at each call site.

const (
	LoadFallbackBak   = "bak"
	LoadFallbackEmpty = "empty"
)

Fallback labels for lachesis_wal_load_fallback_total{from=...}. Passed to Metrics.RecordLoadFallback from the boot loader. LoadFromPrimary is not a fallback and has no label.

const SchemaVersion uint = 7

SchemaVersion is the on-disk envelope version. Every bump so far has been purely additive, so an older file loads without migration; a NEWER file is refused, because starting anyway lets flush rotation destroy the only forward snapshot. Bumping this means adding a row to the history table, not just changing the number.

docs/architecture/data-structures.md#wal-schema-history

Variables

var ErrSchemaNewer = errors.New("wal: snapshot schema newer than this build")

ErrSchemaNewer reports a snapshot whose schema_version this build does not understand — typically a downgrade after a crash mid-upgrade. Load wraps it with the file and version details. Callers must treat it as fatal rather than starting empty: each flush overwrites one rotation generation, so a started agent destroys the only forward snapshot within two flushes.

Functions

func EnsureDir

func EnsureDir(path string) error

EnsureDir creates path's parent directory if missing and verifies it is writable with a probe file (created then removed). Intended for boot: a missing or read-only WAL directory otherwise surfaces only as flush-failure counters after Load mistook ENOENT for a first boot — the agent would run with zero crash durability while looking healthy.

func Quarantine

func Quarantine(path string) (moved bool, err error)

Quarantine moves an unreadable snapshot out of the Save rotation's reach — without it the next two flushes destroy the evidence. One slot only: a later quarantine overwrites the earlier, bounding disk use across a crash loop while keeping the most recent failure.

func Save

func Save(path, agentBuild string, records []state.Record, settled []state.TenantSettledRecord, serverSettled []state.ServerSettledRecord, totalSettled []state.TotalSettledRecord, countersResetAt int64, m *Metrics) error

Save writes one snapshot via the atomic rotation in the package doc. The slices MUST come from a single SnapshotForWAL call — separately taken ones tear across a concurrent fold and persist the folded bytes twice or not at all. agentBuild is informational; m may be nil.

Types

type LoadResult

type LoadResult struct {
	Records         []state.Record
	TenantSettled   []state.TenantSettledRecord
	ServerSettled   []state.ServerSettledRecord
	TotalSettled    []state.TotalSettledRecord
	CountersResetAt int64
	Source          LoadSource
}

LoadResult bundles a successful Load. Every section is nil (or 0) on LoadEmpty and on snapshots older than the version that introduced it — the zero value is meaningful in each case.

docs/architecture/data-structures.md#wal-schema-history

func Load

func Load(path string) (LoadResult, error)

Load reads path, falling back to path+".bak" on parse failure. Both files missing returns a LoadResult with Source=LoadEmpty (no error — first boot is normal). A snapshot from a newer build is never a fallback case: Load returns an error wrapping ErrSchemaNewer without consulting the other file.

type LoadSource

type LoadSource int

LoadSource indicates which file Load succeeded against, or that no WAL was present at all.

const (
	LoadFromPrimary LoadSource = iota
	LoadFromBackup
	LoadEmpty
)

LoadFromPrimary / LoadFromBackup / LoadEmpty are the possible outcomes of Load. Health metrics in the metrics package use LoadFromBackup and LoadEmpty as labels.

type Metrics

type Metrics struct {
	// contains filtered or unexported fields
}

Metrics holds the WAL instruments: the three flush phases (copy-under-lock, marshal, write+fsync+rename), the per-stage failure counter, the boot-load fallback counter, and the state-restart epoch the boot restore decides. nil is acceptable everywhere observations land.

docs/architecture/metrics.md

func NewMetrics

func NewMetrics() *Metrics

NewMetrics constructs the WAL instrument bundle. Buckets follow the design SLOs: 1 ms..1 s for the flush phase (where the p99 target is < 50 ms), Prometheus defaults for the other two. Every known label child of the two counters is seeded at zero — a labelled counter emits no series until its first increment, so without the seed a healthy agent shows "No data" instead of 0 on the dashboard.

func (*Metrics) Collectors

func (m *Metrics) Collectors() []prometheus.Collector

Collectors returns every instrument in the bundle, suitable for passing to prometheus.Registerer.MustRegister.

func (*Metrics) ObserveCopy

func (m *Metrics) ObserveCopy(d time.Duration)

ObserveCopy records one snapshot-copy duration. Intended for the caller of [GlobalState.SnapshotForWAL] (it's the only place that can time the RLock-held section).

func (*Metrics) RecordLoadFallback

func (m *Metrics) RecordLoadFallback(from string)

RecordLoadFallback bumps the load-fallback counter. The caller (typically the boot loader) decides whether the bak / empty outcome is a fallback worth recording — primary loads should not call this.

func (*Metrics) SetCountersReset

func (m *Metrics) SetCountersReset(unixSeconds int64)

SetCountersReset publishes the state-restart epoch the boot restore decided — once, before workers start; the value then holds for the process lifetime (the process_start_time_seconds idiom).