Documentation
¶
Overview ¶
Package anomaly runs a Watchdog/Lookout-style baseline check on a few key signals (error_rate, p99 latency, request_rate). For each (service, metric) it builds a 24h baseline of 5-minute buckets, then compares the most-recent bucket against that distribution. Significant deviations (|z-score| > openZ) are surfaced as Problems with rule_id="anomaly:*", auto-resolved when the value returns inside resolveZ.
This is intentionally simple — no seasonality, no trend removal. It catches sudden spikes well; slow drifts are better handled by SLO burn.
Index ¶
Constants ¶
This section is empty.
Variables ¶
This section is empty.
Functions ¶
func HypothesisPromptBlockTR ¶ added in v0.8.394
func HypothesisPromptBlockTR(h *chstore.RootCauseHypothesis) string
HypothesisPromptBlockTR renders one persisted hypothesis as the Turkish evidence block the problem prompts carry. Returns "" for a nil hypothesis or one without a clear top suspect (a synthesized "no clear cause" row adds noise, not signal) — so a caller can unconditionally append the result and hypothesis-absent behaviour stays byte-identical to the pre-fusion prompt.
Types ¶
type Detector ¶
type Detector struct {
// contains filtered or unexported fields
}
type EvidenceBundle ¶ added in v0.8.45
type EvidenceBundle struct {
Problem chstore.Problem // the triggering problem
CoFiring []chstore.Problem // other OPEN problems on the SAME service
Signals []chstore.AnomalyEvent // active log_pattern / trace_op anomalies on the service
Deploy *chstore.RecentDeployEntry // a deploy of the service just before onset
Neighbors []NeighborProblem // open problems on direct topology neighbours
Confidence int // distinct corroborating evidence types present (incl. the trigger)
}
EvidenceBundle is the corroborating signal set for one triggering Problem.
type LogPatternAnomaly ¶
type LogPatternAnomaly struct {
Pattern string `json:"pattern"` // human-readable name
Regex string `json:"regex"` // the raw re2 used
Kind string `json:"kind"` // "new" | "spike"
CurrentCount uint64 `json:"currentCount"`
BaselineCount uint64 `json:"baselineCount"` // trailing window
Ratio float64 `json:"ratio"` // current / max(baseline,1)
Service string `json:"service"` // service emitting most matches in current window
Sample string `json:"sample"` // representative log body, truncated
LastSeenNs int64 `json:"lastSeenNs"`
// TopServices — v0.5.287. Per-service breakdown of current
// window hits, top 5, count desc. The /logs LogPatternStrip
// renders these as a rosette under the pattern chip so the
// operator can see "OOMKilled fires on foo-svc (12) and
// bar-svc (3)" without expanding or filtering.
TopServices []logstore.PatternServiceHit `json:"topServices,omitempty"`
// Tokens — v0.5.306. The lowercase body substrings any of
// which guarantees a regex match. Exposed to the frontend
// so the /anomalies "logs ↗" link can build a precise OR
// query that lands on the actual log lines, instead of
// the previous behaviour (link only narrowed to the
// service). E.g. "Disk full" carries
// ["no space left", "disk full", "enospc"]
// → link becomes /logs?service=X&q=("no space left" OR
// "disk full" OR "enospc"). Curated per pattern in the
// patterns[] slice below.
Tokens []string `json:"tokens,omitempty"`
}
LogPatternAnomaly is one production-grade signal pattern that either started firing for the first time within the window (Kind="new") or jumped 2x+ over its trailing baseline (Kind="spike"). What an SRE wants to see in their morning inbox: not the raw log volume, just what changed.
func DetectLogPatterns ¶
func DetectLogPatterns(ctx context.Context, store logstore.Store, window time.Duration) ([]LogPatternAnomaly, error)
DetectLogPatterns runs each pattern against the raw `logs` CH table over a current window + a much longer trailing baseline (default: 5-min current vs 1-hour trailing). Returns only the patterns that changed significantly: brand new, or 2x+ over the per-window-length baseline rate.
The asymmetric windows keep an anomaly visible for ~1 hour after it first fires — a 5m-vs-5m comparison has the spike fall into baseline within minutes, which makes the anomaly section flicker. With a 1h baseline the same spike stays visible until the baseline absorbs it.
Performance:
- One query per pattern (combines current + baseline via countIf), N=11 → 11 round-trips total instead of 22.
- All N queries fire in parallel; total cold-cache cost is bounded by the slowest single query, not the sum.
- Each query is partition-pruned to ~10 minutes of logs and the regex runs against the LowCardinality body column.
- Caller should cache the result for 60s; the detector is idempotent and the cache absorbs page reloads.
At 1B logs/day this completes in well under a second cold; warm requests serve directly from Redis.
v0.5.241 — refactored to take a logstore.Store instead of *chstore.Store so the detector works against BOTH the CH and the ES log backend. CH path remains the regex+tokenbf prefilter route; ES path uses query_string token-OR against the body field (regex is ignored; tokens must be zero-false-negative vs the regex). Cross-backend correctness depends on detector authors keeping the Tokens list synchronized with the Regex.
type LogTemplateAnomaly ¶ added in v0.6.27
type LogTemplateAnomaly struct {
TemplateID string
Template string
Service string // first service in the template's seen-services array
FirstSeenNs int64
LastSeenNs int64
TotalCount uint64
Sample string
}
LogTemplateAnomaly is one Drain template that crossed into existence within `window`. The same fingerprint (template ID) surfaces every tick until the row falls outside the window — UpsertAnomalyEvent dedupes via ReplacingMergeTree(version).
func DetectNewLogTemplates ¶ added in v0.6.27
func DetectNewLogTemplates(ctx context.Context, store *chstore.Store, window time.Duration) ([]LogTemplateAnomaly, error)
DetectNewLogTemplates returns Drain templates whose first_seen landed inside [now-window, now]. The templater puller writes these rows; we just query them.
Window guidance: the templater puller runs every 5min by default (see templater/puller.go), so window=10min covers the last two puller cycles — enough to absorb any clock drift between puller + recorder. The anomaly_event dedupe collapses repeated detections.
type NeighborProblem ¶ added in v0.8.45
type NeighborProblem struct {
Problem chstore.Problem
Direction string // "calls" (trigger → neighbour) | "called_by" (neighbour → trigger)
Score float64 // propagation score in [0,1] — downstream suspects only; 0 for upstream / no-error edges
Hops int // topology distance: 1 = direct, 2 = transitive downstream (decayed); 0 = unscored
}
NeighborProblem is an open problem on a service adjacent to the trigger, annotated with the call direction relative to the triggering service and (Faz 6) its root-cause propagation score.
type ProblemExplainer ¶ added in v0.5.254
type ProblemExplainer struct {
// contains filtered or unexported fields
}
func NewProblemExplainer ¶ added in v0.5.254
func (*ProblemExplainer) Start ¶ added in v0.5.254
func (e *ProblemExplainer) Start(ctx context.Context)
Start runs the explainer loop until ctx is cancelled. Initial tick fires immediately so a problem opened during pod startup gets explained without waiting a full interval.
type Recorder ¶
type Recorder struct {
// contains filtered or unexported fields
}
Recorder is the persistence side of the anomaly system. The existing detectors (log_patterns.go, trace_ops.go) are pure readers that compute "what looks anomalous in the last 5 minutes". This recorder runs them on a tick and upserts an AnomalyEvent row per detection so the operator can later answer "what fired in the last hour, even if it cleared".
Events go through ReplacingMergeTree keyed on the (kind, pattern, service) fingerprint — same pattern firing across two consecutive ticks updates the same row, advancing last_seen and tracking peak_ratio. "Cleared" status is derived in the query layer from last_seen freshness, so we don't need a separate sweep job.
func NewRecorder ¶
func NewRecorder(store *chstore.Store, logs logstore.Store, interval, window time.Duration, lock cache.Lock) *Recorder
NewRecorder builds a recorder that ticks every `interval` and each tick scans `window` of recent data. Default 60s tick is fine for the human-grade "anomalies in the last hour" UX — faster ticks just multiply CH load with no operator benefit.
type RootCauseSynthesizer ¶ added in v0.8.168
type RootCauseSynthesizer struct {
// contains filtered or unexported fields
}
func NewRootCauseSynthesizer ¶ added in v0.8.168
func NewRootCauseSynthesizer(store *chstore.Store, lock cache.Lock) *RootCauseSynthesizer
func (*RootCauseSynthesizer) Start ¶ added in v0.8.168
func (s *RootCauseSynthesizer) Start(ctx context.Context)
Start runs the synthesis loop until ctx is cancelled. Initial tick fires immediately so an anchor that opened during pod startup gets a hypothesis without waiting a full interval (same as the explainer).
type TraceOpAnomaly ¶
type TraceOpAnomaly struct {
Service string `json:"service"`
Operation string `json:"operation"`
Kind string `json:"kind"` // "new_error" | "error_spike"
CurrentErrors uint64 `json:"currentErrors"`
BaselineErrors uint64 `json:"baselineErrors"`
Ratio float64 `json:"ratio"` // current / max(baseline, 1)
SampleTraceID string `json:"sampleTraceId"` // representative trace for one-click drill-in
LastSeenNs int64 `json:"lastSeenNs"`
}
TraceOpAnomaly is a per-(service, operation) error or latency signal that's either brand new or up sharply over baseline. Different from the service-wide metric anomaly detector in that it pinpoints the SPECIFIC operation that's misbehaving — the SRE's first question after "is service X broken" is "which endpoint inside X".
func DetectTraceOpAnomalies ¶
func DetectTraceOpAnomalies(ctx context.Context, store *chstore.Store, window time.Duration) ([]TraceOpAnomaly, error)
DetectTraceOpAnomalies finds per-operation error spikes over the last `window` against a longer trailing baseline (1h or 12×window, whichever is larger). Two qualifying conditions using the per-window-equivalent baseline:
- baseline_per_window == 0 AND current_errors >= 3 ("new error pattern", i.e. an op that started failing now)
- baseline_per_window > 0 AND ratio >= 2 (existing op whose error rate just doubled)
The asymmetric baseline (5-min current vs 1-hour trailing) keeps fresh spikes visible for ~1 hour — a 5m-vs-5m comparison would have the spike fall into baseline within minutes, flickering the anomaly section as windows slide.
One CH query LEFT JOINs the current and trailing windows over the (service_name, name) primary-key prefix, so even at 1B spans/day the scan is bounded to the matching slice.
Source Files