Help us shape the future of Go! Take the 2026 Go Developer Survey

alerts

package
v0.130.0 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Oct 12, 2026 License: Apache-2.0 Imports: 20 Imported by: 0

Documentation

Overview

Package alerts reads the alerts of the installations the sessions work on and turns two readings into NEW and RESOLVED lines.

This file is the pure half: an Alertmanager answer becomes a set keyed by fingerprint, two sets become change lines, one set becomes the grouped snapshot. Reading an Alertmanager is read.go, the baseline is state.go.

Index

Constants

View Source 
const Acknowledged = "acknowledged"

Acknowledged is the status of an incident someone acknowledged.

View Source 
const New = "NEW"

New is the kind of a line about an alert that started firing.

View Source 
const Page = "page"

Page is the severity that pages the on-call person; it is printed in capitals.

View Source 
const PagerDutyGrace = 3 * time.Minute

PagerDutyGrace is how long a PagerDuty that read does not answer before it is said unseen: muster's auth_required while it reconnects the server lasts a reading or two.

View Source 
const Repeat = "repeat of an alert that never ended (same start)"

Repeat is the reason of a quiet NEW line whose alert resolved within the damper's window with the same start: Alertmanager keeps an alert's start while it fires, so the RESOLVED was a reading that missed it (an Alertmanager restart, a reconnect), not an end, and its NEW was said.

View Source 
const UnseenRepeat = 15 * time.Minute

UnseenRepeat is how often the watch repeats that an unreachable installation's alerts are unseen, with the time since its last reading.

View Source 
const WorkingHours = "InhibitionOutsideWorkingHours"

WorkingHours is the alert that inhibits the alerts labelled cancel_if_outside_working_hours outside working hours. It only keeps them from paging: the team's alerts it inhibits are read all the same.

Variables

View Source 
var Severities = []string{"none", "info", "warning", "notify", "critical", "page"}

Severities are the severities from lowest to highest, as Giant Swarm's rules (none, notify, page) and kube-prometheus-stack's (info, warning, critical) set them. A floor is one of them; an alert whose severity is none of them is never below a floor.

Functions

func Below added in v0.12.0 

func Below(severity, floor string) bool

Below reports whether severity ranks under floor.

func Glob added in v0.36.0 

func Glob(pattern, s string) bool

Glob reports whether s matches pattern, in which * matches any run of characters, slashes included; an empty pattern matches everything.

func Import 

func Import(st *State, dir string) ([]string, error)

Import adds the per-installation baselines in dir (one <installation>.json each, as {"reachable": …, "alerts": {…}}) for every installation the baseline does not know yet, so a switch from another watcher prints no burst of NEW lines. It returns the imported installations.

func IsNew added in v0.12.0 

func IsNew(line string) bool

IsNew reports whether a line of Step is about an alert that started firing.

func Key added in v0.68.0 

func Key(fingerprint string, a Alert) string

Key is a firing alert's identity in the baseline: its fingerprint and its start, so an alert that resolves and fires again is a new one, owned by nobody.

func Line added in v0.18.0 

func Line(kind, installation, severity, team, alertname, where, since string, now time.Time) string

Line is one alert's change: NEW, RESOLVED or FLAPPING.

func ReadAnswers added in v0.12.0 

func ReadAnswers(dir string) (map[string][]Raw, error)

ReadAnswers are the answers recorded in dir, by installation.

func ResolveContext 

func ResolveContext(name, explicit string, template func(string) string, contexts []string) string

ResolveContext is the explicit context, else the templated one (template maps a name to its context, "" for none), else the context named <name>, else the one ending in @<name>.

func WriteAnswer added in v0.12.0 

func WriteAnswer(dir, installation string, raw []Raw) (string, error)

WriteAnswer records an installation's answer as dir/<installation>.json, for alerts replay and as a test fixture. It returns the file.

Types

type Alert 

type Alert struct {
	Severity  string `json:"severity"`
	Team      string `json:"team"`
	Alertname string `json:"alertname"`
	Cluster   string `json:"cluster"`
	Where     string `json:"where"`
	Since     string `json:"since"`
	// contains filtered or unexported fields
}

Alert is what a line prints of one alert.

type Answer 

type Answer struct {
	OK     bool
	Alerts []Raw
	Why    string
}

Answer is one installation's reading: its alerts, or why it did not answer.

type Damper added in v0.12.0 

type Damper struct {
	Changes int
	Window  time.Duration
}

Damper holds back an alert that changes too often: its Changes-th NEW or RESOLVED within Window is one FLAPPING line, and its changes print nothing until it has been stable for Window. Changes under 2 is no damper.

type Firing added in v0.68.0 

type Firing struct {
	Installation string
	Key          string
	Alert
}

Firing is one alert of the baseline with its key.

func Match added in v0.68.0 

func Match(firing []Firing, q string) []Firing

Match are the firing alerts q names: <installation>/<alertname>, an alertname on any installation, or a fingerprint.

func (Firing) Name added in v0.68.0 

func (f Firing) Name() string

Name is the alert as alerts own takes it: <installation>/<alertname>.

type Flap added in v0.12.0 

type Flap struct {
	Changes  []time.Time `json:"changes"`
	Flapping bool        `json:"flapping,omitempty"`
}

Flap is the damper's memory of one alert: its changes within the window, and whether it was reported flapping.

type Incident added in v0.111.0 

type Incident struct {
	Number int    `json:"number"`
	Title  string `json:"title"`
	// Status is triggered or acknowledged.
	Status string `json:"status"`
	// Installation is the installation label of the incident's first
	// alert; empty while it has not been read.
	Installation string `json:"installation,omitempty"`
	Since        string `json:"since"`
}

Incident is an open PagerDuty incident as the watch keeps it.

type Installation 

type Installation struct {
	Reachable bool `json:"reachable"`
	Alerts    Set  `json:"alerts"`
	// Seen is its last reading; zero when it has not answered since the
	// baseline has the field.
	Seen time.Time `json:"seen,omitzero"`
	// Said is when the watch last said it is unreachable.
	Said time.Time `json:"said,omitzero"`
	// Flaps are the damper's records of the alerts that changed within its
	// window, by fingerprint.
	Flaps map[string]*Flap `json:"flaps,omitempty"`
	// Resolved are the alerts resolved within the damper's window, by
	// fingerprint, for the quiet repeat.
	Resolved map[string]Resolved `json:"resolved,omitempty"`
}

Installation is an installation's baseline: its last set, kept while it does not answer. A nil Alerts means it has not been seen yet.

type Owner 

type Owner struct {
	PID   int       `json:"pid"`
	Since time.Time `json:"since"`
}

Owner is the process that reads the alerts and keeps the baseline.

type PagerDuty added in v0.111.0 

type PagerDuty struct {
	Reachable bool                `json:"reachable"`
	Incidents map[string]Incident `json:"incidents"`
	Seen      time.Time           `json:"seen,omitzero"`
	Said      time.Time           `json:"said,omitzero"`
}

PagerDuty is the incidents' baseline: the last set, kept while PagerDuty does not answer. A nil Incidents means it has not been read yet.

type PagerDutyAnswer added in v0.111.0 

type PagerDutyAnswer struct {
	OK        bool
	Incidents map[string]Incident
	Why       string
}

PagerDutyAnswer is one reading of the open incidents, by incident id, or why it did not answer.

type PagerDutyReader added in v0.111.0 

type PagerDutyReader struct {
	Call     ToolCall
	Services []string
}

PagerDutyReader reads the open incidents of Services.

func (PagerDutyReader) Read added in v0.111.0 

func (p PagerDutyReader) Read(ctx context.Context, known map[string]Incident) PagerDutyAnswer

Read lists the triggered and acknowledged incidents of the services, of every age, and reads the installation of each one known lacks.

type Quiet added in v0.36.0 

type Quiet struct {
	Installation string `yaml:"installation" json:"installation,omitempty"`
	Cluster      string `yaml:"cluster" json:"cluster,omitempty"`
	Alertname    string `yaml:"alertname" json:"alertname,omitempty"`
	Severity     string `yaml:"severity" json:"severity,omitempty"`
}

Quiet is a rule whose alerts' changes wake nobody: the watch keeps their lines out of its output and logs them instead. Every field is a glob (* matches any run of characters); an empty one matches everything. A rule never quiets an alert of the team, one on an installation in play, or a page, unless the rule names a cluster: a page on another team's test cluster is that team's.

func (Quiet) String added in v0.36.0 

func (q Quiet) String() string

String names the rule in a quiet line.

type Raw 

type Raw struct {
	Fingerprint string            `json:"fingerprint"`
	Labels      map[string]string `json:"labels"`
	StartsAt    string            `json:"startsAt"`
	Status      Status            `json:"status"`
}

Raw is one alert as the Alertmanager v2 API returns it.

type Reader 

type Reader struct {
	// Kubectl is the kubectl binary.
	Kubectl string
	// Timeout bounds the reading of one installation, forwards included.
	Timeout time.Duration
	// Tenant is the Mimir tenant read first; empty, only the plain
	// Alertmanager is read.
	Tenant string
	// Keep holds each installation's port-forward open for the next read,
	// and the kubeconfig's contexts until it changes; Close ends them.
	Keep bool
	// contains filtered or unexported fields
}

Reader reads Alertmanagers through kubectl port-forwards. A Reader that keeps its forwards holds each installation's open across reads, until Close: a read is then an HTTP request, no kubectl or Teleport credential plugin run (kubectl caches the credential until it expires).

func (*Reader) Close added in v0.112.1 

func (r *Reader) Close()

Close ends every port-forward the Reader keeps.

func (*Reader) Contexts 

func (r *Reader) Contexts(ctx context.Context) []string

Contexts are the kubeconfig's context names; a Reader that keeps its forwards asks kubectl again only once the kubeconfig changed.

func (*Reader) Read 

func (r *Reader) Read(ctx context.Context, targets []Target) []Answer

Read reads every target in parallel, each within r.Timeout, and returns the answers in the targets' order. Unless the Reader keeps its forwards, every port-forward has ended when it returns.

type Resolved added in v0.36.0 

type Resolved struct {
	Since string    `json:"since"`
	At    time.Time `json:"at"`
}

Resolved is a resolved alert the baseline remembers for Repeat: its start and when it resolved.

type Rules 

type Rules struct {
	// Ignore are alert names that never appear.
	Ignore []string
	// Team is the team whose alerts are marked in capitals and counted.
	Team string
	// Collapse is the number of changes of one alertname in one run above
	// which they are one line with a count.
	Collapse int
	// Floors are the lowest severity printed, per installation. An alert
	// below its floor stays in the baseline and is never printed, so a
	// changed floor prints no burst of NEW or RESOLVED lines.
	Floors map[string]string
	// Flap is the flap damper.
	Flap Damper
	// Quiet are the rules whose alerts' changes wake nobody (Triage);
	// their alerts repeating after a reading that missed them are quiet
	// too.
	Quiet []Quiet
	// InPlay reports whether an installation is in play (leased, merged
	// into): none of its alerts is quiet. nil is none in play.
	InPlay func(installation string) bool
}

Rules are the settings the lines depend on.

func (Rules) Count added in v0.54.0 

func (r Rules) Count(installation string, ans Answer) (active, paging int)

Count is how many alerts the snapshot shows of an answer and how many of them page.

func (Rules) Normalize 

func (r Rules) Normalize(raw []Raw, installation string) Set

Normalize turns an answer's alerts into the set, ignored names dropped.

func (Rules) PagerDutySnapshot added in v0.111.0 

func (r Rules) PagerDutySnapshot(ans PagerDutyAnswer, now time.Time) []string

PagerDutySnapshot is the open incidents, oldest first.

func (Rules) PagerDutyStep added in v0.111.0 

func (r Rules) PagerDutyStep(prev *PagerDuty, ans PagerDutyAnswer, now time.Time) ([]string, *PagerDuty)

PagerDutyStep returns the lines one reading prints and the baseline it leaves; prev is nil on the first reading. It mirrors Triage: a first look lists the open incidents, a PagerDuty that does not answer keeps the set and is said, once it is PagerDutyGrace unseen, again every UnseenRepeat.

func (Rules) SnapshotLines 

func (r Rules) SnapshotLines(installation string, ans Answer, now time.Time) []string

SnapshotLines is the current set grouped by severity, team, alertname and cluster, pages and the team's alerts first.

func (Rules) Step 

func (r Rules) Step(installation string, prev *Installation, ans Answer, now time.Time) ([]string, *Installation)

Step returns the lines one watch run prints for an installation and the baseline it leaves. prev is nil on the installation's first run.

func (Rules) Triage added in v0.36.0 

func (r Rules) Triage(installation string, prev *Installation, ans Answer, now time.Time) (lines, quiet []string, next *Installation)

Triage is Step with the changes the quiet rules hold back: lines wake the reader, quiet are the lines that wake nobody, each ending in its reason.

type Set 

type Set map[string]Alert

Set is an installation's alerts by fingerprint.

type State 

type State struct {
	Owner         *Owner                   `json:"owner,omitempty"`
	Installations map[string]*Installation `json:"installations"`
	// PagerDuty is the open incidents' baseline, in the PagerDuty store
	// only.
	PagerDuty *PagerDuty `json:"pagerduty,omitempty"`
}

State is the alert baseline: every installation's last set, and the watch that owns it, so two watches during a hand-over never split the NEW and RESOLVED lines between them.

func (*State) Firing added in v0.68.0 

func (st *State) Firing(severity string) []Firing

Firing are the alerts of the reachable installations' baselines, at severity when it is not empty, by installation, alertname and object. An unreachable installation's last set is not firing: it may have resolved.

type Status added in v0.27.2 

type Status struct {
	InhibitedBy []string `json:"inhibitedBy,omitempty"`
}

Status is an alert's state; InhibitedBy are the fingerprints of the alerts that inhibit it.

type Store 

type Store struct {
	// contains filtered or unexported fields
}

Store keeps the baseline in <name>.json in beekeeper's state directory. Only the process holding <name>.lock reads alerts into it.

func NewPagerDutyStore added in v0.111.0 

func NewPagerDutyStore(dir string) *Store

NewPagerDutyStore returns the incidents' baseline kept in dir: read on its own cadence, so it never waits for, or overwrites, the installations'.

func NewStore 

func NewStore(dir string) *Store

NewStore returns the installations' baseline kept in dir.

func (*Store) Load 

func (s *Store) Load() (*State, error)

Load reads the baseline; a missing file is an empty one.

func (*Store) Own 

func (s *Store) Own() (bool, *Owner, error)

Own makes this process the baseline's owner unless another one is; then it returns false and that owner. Owning lasts until Release or the process ends.

func (*Store) Path added in v0.9.0 

func (s *Store) Path() string

Path is the baseline file.

func (*Store) Release 

func (s *Store) Release() error

Release gives the baseline up.

func (*Store) Save 

func (s *Store) Save(st *State) error

Save writes the baseline with one rename.

type Target 

type Target struct {
	Name    string `json:"name"`
	Context string `json:"context"`
	Why     string `json:"why"`
}

Target is an installation to read, the kube context that reaches it and why it is read.

func Targets 

func Targets(configured []Target, leased map[string]string, template func(string) string, contexts []string) []Target

Targets are the configured installations, then every leased one whose name resolves to a kube context (a kind lab's or the browser's does not), each with its context resolved (template: ResolveContext's).

type ToolCall added in v0.111.0 

type ToolCall func(ctx context.Context, tool string, args map[string]any) (string, error)

ToolCall calls a muster tool of the PagerDuty server by its short name (list_incidents) and returns its text.

type Unowned added in v0.68.0 

type Unowned struct {
	Firing
	// More is how many more alerts of the name are unowned.
	More int
	// Since is when it was last owned, or its start when it never was.
	Since time.Time
	// For is how long it has had no owner.
	For time.Duration
	// Mark is the grace period it is in, unique per page and period: the
	// line is said once per mark.
	Mark string
}

Unowned is a page that has had no owner for at least the grace period: the longest unowned of its installation's alerts of the same name.

func UnownedPages added in v0.68.0 

func UnownedPages(firing []Firing, grace time.Duration, ownedUntil func(key string) (time.Time, bool), now time.Time) []Unowned

UnownedPages are the pages among firing that have had no owner for grace or longer, one per installation and alertname. ownedUntil tells of an alert's key when its last owner stopped owning it: zero while one owns it, false when it never had one. A page is due once per grace period it stays unowned: its Mark changes with each.

func (Unowned) Line added in v0.68.0 

func (u Unowned) Line() string

Line is the watch's line for the page.

Source Files

View all Source files

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.