evaluator

Documentation

Overview

Package evaluator runs alert rules on a fixed interval, opens problems when their condition is breached, and resolves problems whose breach is no longer present. Built-in rules cover the typical APM signals (error rate, P99 latency, request-rate drops).

v0.8.354 — HA audit 🟡#2: the ForSec sustain clock (breachSince) and the CooldownSec gate (lastResolved) lived ONLY in per-pod memory, so every leader failover and every rolling deploy (maxUnavailable:0 ⇒ leader moves on EVERY release) reset them: a breach 9 minutes into a 10-minute sustain window restarted from zero on the new leader (alert delayed another full ForSec), and a just-resolved problem could re-open straight past its cooldown (flap + duplicate page).

Design: write-through, read-fallback. The in-memory maps stay the hot path (unchanged semantics for a stable leader); every stamp WRITE mirrors to Redis, and an in-memory MISS (fresh leader) hydrates from Redis once before falling back to "no stamp". Redis errors, misses, and the Noop cache (SwitchableCache pre-reconnect, v0.8.344) all degrade to the pre-v0.8.354 in-memory-only behavior — stamp IO never blocks or fails a tick.

Lock discipline: breachMu guards ONLY the map access; all Redis IO happens outside the mutex (values are copied first).

Index

• type Evaluator
  • func New(store *chstore.Store, interval time.Duration, lock cache.Lock, ...) *Evaluator
  • func (e *Evaluator) SetLogs(logs logstore.Store)
  • func (e *Evaluator) SetStampCache(c cache.Cache)
  • func (e *Evaluator) Start(ctx context.Context)

Types

type Evaluator

type Evaluator struct {
	// contains filtered or unexported fields
}

func New

func New(store *chstore.Store, interval time.Duration, lock cache.Lock, notifier *notify.Notifier) *Evaluator

New takes a cache.Lock so multiple Coremetry replicas only run the evaluation loop once per tick, and a notifier so PROBLEM OPENED transitions fan out to email/slack/etc.

func (*Evaluator) SetLogs

func (e *Evaluator) SetLogs(logs logstore.Store)

SetLogs wires the log backend so the saved-search alert path (rules with LogQuery != "") can count matches via logstore. Called from main() once buildLogStore has resolved the backend — keeps the New() constructor lean + avoids reordering boot-time wiring around the evaluator.

func (*Evaluator) SetStampCache

func (e *Evaluator) SetStampCache(c cache.Cache)

SetStampCache wires the shared cache so sustain/cooldown stamps survive leader failover. Called from main() once, like SetLogs — keeps New()'s signature stable. nil (never wired) or a Noop inner keeps the evaluator on pure in-memory stamps.

func (*Evaluator) Start

func (e *Evaluator) Start(ctx context.Context)

Start runs the evaluation loop until ctx is cancelled. Built-in rules are seeded by every replica — that's safe (UpsertAlertRule is idempotent on id). Only the actual evaluation pass is leader-gated.