alerts

Documentation

Overview

Package alerts reads the alerts of the installations the sessions work on and turns two readings into NEW and RESOLVED lines.

This file is the pure half: an Alertmanager answer becomes a set keyed by fingerprint, two sets become change lines, one set becomes the grouped snapshot. Reading an Alertmanager is read.go, the baseline is state.go.

Index

• Constants
• Variables
• func Below(severity, floor string) bool
• func Glob(pattern, s string) bool
• func Import(st *State, dir string) ([]string, error)
• func IsNew(line string) bool
• func Key(fingerprint string, a Alert) string
• func Line(kind, installation, severity, team, alertname, where, since string, ...) string
• func ReadAnswers(dir string) (map[string][]Raw, error)
• func ResolveContext(name, explicit string, template func(string) string, contexts []string) string
• func WriteAnswer(dir, installation string, raw []Raw) (string, error)
• type Alert
• type Answer
• type Damper
• type Firing
  • func Match(firing []Firing, q string) []Firing
  • func (f Firing) Name() string
• type Flap
• type Installation
• type Owner
• type Quiet
  • func (q Quiet) String() string
• type Raw
• type Reader
  • func (r Reader) Contexts(ctx context.Context) []string
  • func (r Reader) Read(ctx context.Context, targets []Target) []Answer
• type Resolved
• type Rules
  • func (r Rules) Count(installation string, ans Answer) (active, paging int)
  • func (r Rules) Normalize(raw []Raw, installation string) Set
  • func (r Rules) SnapshotLines(installation string, ans Answer, now time.Time) []string
  • func (r Rules) Step(installation string, prev *Installation, ans Answer, now time.Time) ([]string, *Installation)
  • func (r Rules) Triage(installation string, prev *Installation, ans Answer, now time.Time) (lines, quiet []string, next *Installation)
• type Set
• type State
  • func (st *State) Firing(severity string) []Firing
• type Status
• type Store
  • func NewStore(dir string) *Store
  • func (s *Store) Load() (*State, error)
  • func (s *Store) Own() (bool, *Owner, error)
  • func (s *Store) Path() string
  • func (s *Store) Release() error
  • func (s *Store) Save(st *State) error
• type Target
  • func Targets(configured []Target, leased map[string]string, template func(string) string, ...) []Target
• type Unowned
  • func UnownedPages(firing []Firing, grace time.Duration, ...) []Unowned
  • func (u Unowned) Line() string

Constants

const New = "NEW"

New is the kind of a line about an alert that started firing.

const Page = "page"

Page is the severity that pages the on-call person; it is printed in capitals.

const Repeat = "repeat of an alert that never ended (same start)"

Repeat is the reason of a quiet NEW line whose alert resolved within the damper's window with the same start: Alertmanager keeps an alert's start while it fires, so the RESOLVED was a reading that missed it (an Alertmanager restart, a reconnect), not an end, and its NEW was said.

const UnseenRepeat = 15 * time.Minute

UnseenRepeat is how often the watch repeats that an unreachable installation's alerts are unseen, with the time since its last reading.

const WorkingHours = "InhibitionOutsideWorkingHours"

WorkingHours is the alert that inhibits the alerts labelled cancel_if_outside_working_hours outside working hours. It only keeps them from paging: the team's alerts it inhibits are read all the same.

Variables

var Severities = []string{"none", "info", "warning", "notify", "critical", "page"}

Severities are the severities from lowest to highest, as Giant Swarm's rules (none, notify, page) and kube-prometheus-stack's (info, warning, critical) set them. A floor is one of them; an alert whose severity is none of them is never below a floor.

Functions

func Below

func Below(severity, floor string) bool

Below reports whether severity ranks under floor.

func Glob

func Glob(pattern, s string) bool

Glob reports whether s matches pattern, in which * matches any run of characters, slashes included; an empty pattern matches everything.

func Import

func Import(st *State, dir string) ([]string, error)

Import adds the per-installation baselines in dir (one <installation>.json each, as {"reachable": …, "alerts": {…}}) for every installation the baseline does not know yet, so a switch from another watcher prints no burst of NEW lines. It returns the imported installations.

func IsNew

func IsNew(line string) bool

IsNew reports whether a line of Step is about an alert that started firing.

func Key

func Key(fingerprint string, a Alert) string

Key is a firing alert's identity in the baseline: its fingerprint and its start, so an alert that resolves and fires again is a new one, owned by nobody.

func Line

func Line(kind, installation, severity, team, alertname, where, since string, now time.Time) string

Line is one alert's change: NEW, RESOLVED or FLAPPING.

func ReadAnswers

func ReadAnswers(dir string) (map[string][]Raw, error)

ReadAnswers are the answers recorded in dir, by installation.

func ResolveContext

func ResolveContext(name, explicit string, template func(string) string, contexts []string) string

ResolveContext is the explicit context, else the templated one (template maps a name to its context, "" for none), else the context named <name>, else the one ending in @<name>.

func WriteAnswer

func WriteAnswer(dir, installation string, raw []Raw) (string, error)

WriteAnswer records an installation's answer as dir/<installation>.json, for alerts replay and as a test fixture. It returns the file.

Types

type Alert

type Alert struct {
	Severity  string `json:"severity"`
	Team      string `json:"team"`
	Alertname string `json:"alertname"`
	Cluster   string `json:"cluster"`
	Where     string `json:"where"`
	Since     string `json:"since"`
	// contains filtered or unexported fields
}

Alert is what a line prints of one alert.

type Answer

type Answer struct {
	OK     bool
	Alerts []Raw
	Why    string
}

Answer is one installation's reading: its alerts, or why it did not answer.

type Damper

type Damper struct {
	Changes int
	Window  time.Duration
}

Damper holds back an alert that changes too often: its Changes-th NEW or RESOLVED within Window is one FLAPPING line, and its changes print nothing until it has been stable for Window. Changes under 2 is no damper.

type Firing

type Firing struct {
	Installation string
	Key          string
	Alert
}

Firing is one alert of the baseline with its key.

func Match

func Match(firing []Firing, q string) []Firing

Match are the firing alerts q names: <installation>/<alertname>, an alertname on any installation, or a fingerprint.

func (Firing) Name

func (f Firing) Name() string

Name is the alert as alerts own takes it: <installation>/<alertname>.

type Flap

type Flap struct {
	Changes  []time.Time `json:"changes"`
	Flapping bool        `json:"flapping,omitempty"`
}

Flap is the damper's memory of one alert: its changes within the window, and whether it was reported flapping.

type Installation

type Installation struct {
	Reachable bool `json:"reachable"`
	Alerts    Set  `json:"alerts"`
	// Seen is its last reading; zero when it has not answered since the
	// baseline has the field.
	Seen time.Time `json:"seen,omitzero"`
	// Said is when the watch last said it is unreachable.
	Said time.Time `json:"said,omitzero"`
	// Flaps are the damper's records of the alerts that changed within its
	// window, by fingerprint.
	Flaps map[string]*Flap `json:"flaps,omitempty"`
	// Resolved are the alerts resolved within the damper's window, by
	// fingerprint, for the quiet repeat.
	Resolved map[string]Resolved `json:"resolved,omitempty"`
}

Installation is an installation's baseline: its last set, kept while it does not answer. A nil Alerts means it has not been seen yet.

type Owner

type Owner struct {
	PID   int       `json:"pid"`
	Since time.Time `json:"since"`
}

Owner is the process that reads the alerts and keeps the baseline.

type Quiet

type Quiet struct {
	Installation string `yaml:"installation" json:"installation,omitempty"`
	Cluster      string `yaml:"cluster" json:"cluster,omitempty"`
	Alertname    string `yaml:"alertname" json:"alertname,omitempty"`
	Severity     string `yaml:"severity" json:"severity,omitempty"`
}

Quiet is a rule whose alerts' changes wake nobody: the watch keeps their lines out of its output and logs them instead. Every field is a glob (* matches any run of characters); an empty one matches everything. A rule never quiets an alert of the team, one on an installation in play, or a page, unless the rule names a cluster: a page on another team's test cluster is that team's.

func (Quiet) String

func (q Quiet) String() string

String names the rule in a quiet line.

type Raw

type Raw struct {
	Fingerprint string            `json:"fingerprint"`
	Labels      map[string]string `json:"labels"`
	StartsAt    string            `json:"startsAt"`
	Status      Status            `json:"status"`
}

Raw is one alert as the Alertmanager v2 API returns it.

type Reader

type Reader struct {
	// Kubectl is the kubectl binary.
	Kubectl string
	// Timeout bounds the reading of one installation, forwards included.
	Timeout time.Duration
	// Tenant is the Mimir tenant read first; empty, only the plain
	// Alertmanager is read.
	Tenant string
}

Reader reads Alertmanagers through kubectl port-forwards.

func (Reader) Contexts

func (r Reader) Contexts(ctx context.Context) []string

Contexts are the kubeconfig's context names.

func (Reader) Read

func (r Reader) Read(ctx context.Context, targets []Target) []Answer

Read reads every target in parallel, each within r.Timeout, and returns the answers in the targets' order. Every port-forward has ended when it returns.

type Resolved

type Resolved struct {
	Since string    `json:"since"`
	At    time.Time `json:"at"`
}

Resolved is a resolved alert the baseline remembers for Repeat: its start and when it resolved.

type Rules

type Rules struct {
	// Ignore are alert names that never appear.
	Ignore []string
	// Team is the team whose alerts are marked in capitals and counted.
	Team string
	// Collapse is the number of changes of one alertname in one run above
	// which they are one line with a count.
	Collapse int
	// Floors are the lowest severity printed, per installation. An alert
	// below its floor stays in the baseline and is never printed, so a
	// changed floor prints no burst of NEW or RESOLVED lines.
	Floors map[string]string
	// Flap is the flap damper.
	Flap Damper
	// Quiet are the rules whose alerts' changes wake nobody (Triage);
	// their alerts repeating after a reading that missed them are quiet
	// too.
	Quiet []Quiet
	// InPlay reports whether an installation is in play (leased, merged
	// into): none of its alerts is quiet. nil is none in play.
	InPlay func(installation string) bool
}

Rules are the settings the lines depend on.

func (Rules) Count

func (r Rules) Count(installation string, ans Answer) (active, paging int)

Count is how many alerts the snapshot shows of an answer and how many of them page.

func (Rules) Normalize

func (r Rules) Normalize(raw []Raw, installation string) Set

Normalize turns an answer's alerts into the set, ignored names dropped.

func (Rules) SnapshotLines

func (r Rules) SnapshotLines(installation string, ans Answer, now time.Time) []string

SnapshotLines is the current set grouped by severity, team, alertname and cluster, pages and the team's alerts first.

func (Rules) Step

func (r Rules) Step(installation string, prev *Installation, ans Answer, now time.Time) ([]string, *Installation)

Step returns the lines one watch run prints for an installation and the baseline it leaves. prev is nil on the installation's first run.

func (Rules) Triage

func (r Rules) Triage(installation string, prev *Installation, ans Answer, now time.Time) (lines, quiet []string, next *Installation)

Triage is Step with the changes the quiet rules hold back: lines wake the reader, quiet are the lines that wake nobody, each ending in its reason.

type Set

type Set map[string]Alert

Set is an installation's alerts by fingerprint.

type State

type State struct {
	Owner         *Owner                   `json:"owner,omitempty"`
	Installations map[string]*Installation `json:"installations"`
}

State is the alert baseline: every installation's last set, and the watch that owns it, so two watches during a hand-over never split the NEW and RESOLVED lines between them.

func (*State) Firing

func (st *State) Firing(severity string) []Firing

Firing are the alerts of the reachable installations' baselines, at severity when it is not empty, by installation, alertname and object. An unreachable installation's last set is not firing: it may have resolved.

type Status

type Status struct {
	InhibitedBy []string `json:"inhibitedBy,omitempty"`
}

Status is an alert's state; InhibitedBy are the fingerprints of the alerts that inhibit it.

type Store

type Store struct {
	// contains filtered or unexported fields
}

Store keeps the baseline in alerts.json in beekeeper's state directory. Only the process holding alerts.lock reads alerts into it.

func NewStore

func NewStore(dir string) *Store

NewStore returns the baseline kept in dir.

func (*Store) Load

func (s *Store) Load() (*State, error)

Load reads the baseline; a missing file is an empty one.

func (*Store) Own

func (s *Store) Own() (bool, *Owner, error)

Own makes this process the baseline's owner unless another one is; then it returns false and that owner. Owning lasts until Release or the process ends.

func (*Store) Path

func (s *Store) Path() string

Path is the baseline file.

func (*Store) Release

func (s *Store) Release() error

Release gives the baseline up.

func (*Store) Save

func (s *Store) Save(st *State) error

Save writes the baseline with one rename.

type Target

type Target struct {
	Name    string `json:"name"`
	Context string `json:"context"`
	Why     string `json:"why"`
}

Target is an installation to read, the kube context that reaches it and why it is read.

func Targets

func Targets(configured []Target, leased map[string]string, template func(string) string, contexts []string) []Target

Targets are the configured installations, then every leased one whose name resolves to a kube context (a kind lab's or the browser's does not), each with its context resolved (template: ResolveContext's).

type Unowned

type Unowned struct {
	Firing
	// More is how many more alerts of the name are unowned.
	More int
	// Since is when it was last owned, or its start when it never was.
	Since time.Time
	// For is how long it has had no owner.
	For time.Duration
	// Mark is the grace period it is in, unique per page and period: the
	// line is said once per mark.
	Mark string
}

Unowned is a page that has had no owner for at least the grace period: the longest unowned of its installation's alerts of the same name.

func UnownedPages

func UnownedPages(firing []Firing, grace time.Duration, ownedUntil func(key string) (time.Time, bool), now time.Time) []Unowned

UnownedPages are the pages among firing that have had no owner for grace or longer, one per installation and alertname. ownedUntil tells of an alert's key when its last owner stopped owning it: zero while one owns it, false when it never had one. A page is due once per grace period it stays unowned: its Mark changes with each.

func (Unowned) Line

func (u Unowned) Line() string

Line is the watch's line for the page.