anomaly

Documentation

Overview

Package anomaly runs a Watchdog/Lookout-style baseline check on a few key signals (error_rate, p99 latency, request_rate). For each (service, metric) it builds a 24h baseline of 5-minute buckets, then compares the most-recent bucket against that distribution. Significant deviations (|z-score| > openZ) are surfaced as Problems with rule_id="anomaly:*", auto-resolved when the value returns inside resolveZ.

This is intentionally simple — no seasonality, no trend removal. It catches sudden spikes well; slow drifts are better handled by SLO burn.

Index

• func HypothesisPromptBlockTR(h *chstore.RootCauseHypothesis) string
• func PickDeploysAroundStart(deps []chstore.Deploy, firstSeen int64) []string
• type Detector
  • func New(store *chstore.Store, interval time.Duration, lock cache.Lock, ...) *Detector
  • func (d *Detector) Start(ctx context.Context)
• type EvidenceBundle
• type ExceptionExplainInput
  • func BuildExceptionExplainInput(ctx context.Context, store *chstore.Store, logs logstore.Store, ...) ExceptionExplainInput
• type ExceptionExplainer
  • func NewExceptionExplainer(store *chstore.Store, logs logstore.Store, cop *copilot.Service, ...) *ExceptionExplainer
  • func (e *ExceptionExplainer) Start(ctx context.Context)
• type LogPatternAnomaly
  • func DetectLogPatterns(ctx context.Context, store logstore.Store, window time.Duration) ([]LogPatternAnomaly, error)
• type LogTemplateAnomaly
  • func DetectNewLogTemplates(ctx context.Context, store *chstore.Store, window time.Duration) ([]LogTemplateAnomaly, error)
• type NeighborProblem
• type ProblemExplainer
  • func NewProblemExplainer(store *chstore.Store, cop *copilot.Service, lock cache.Lock) *ProblemExplainer
  • func (e *ProblemExplainer) Start(ctx context.Context)
• type Recorder
  • func NewRecorder(store *chstore.Store, logs logstore.Store, interval, window time.Duration, ...) *Recorder
  • func (r *Recorder) Start(ctx context.Context)
• type RootCauseSynthesizer
  • func NewRootCauseSynthesizer(store *chstore.Store, lock cache.Lock) *RootCauseSynthesizer
  • func (s *RootCauseSynthesizer) Start(ctx context.Context)
• type TraceOpAnomaly
  • func DetectTraceOpAnomalies(ctx context.Context, store *chstore.Store, window time.Duration) ([]TraceOpAnomaly, error)

Functions

func HypothesisPromptBlockTR

func HypothesisPromptBlockTR(h *chstore.RootCauseHypothesis) string

HypothesisPromptBlockTR renders one persisted hypothesis as the Turkish evidence block the problem prompts carry. Returns "" for a nil hypothesis or one without a clear top suspect (a synthesized "no clear cause" row adds noise, not signal) — so a caller can unconditionally append the result and hypothesis-absent behaviour stays byte-identical to the pre-fusion prompt.

func PickDeploysAroundStart

func PickDeploysAroundStart(deps []chstore.Deploy, firstSeen int64) []string

PickDeploysAroundStart — GetServiceDeploys ASC döner; FirstSeen ÖNCESİNDEN son 3 + SONRASINDAN ilk 2 seçilir ve yön açıkça yazılır. Saf — exception_context_test.go: düz "son 5" kesimi uzun ömürlü gruplarda asıl adayı (başlangıçtan hemen önceki deploy) düşürüyordu, negatif "önce" ise LLM'e yanlış kanıt oluyordu (v0.9.414 bulguları).

Types

type Detector

type Detector struct {
	// contains filtered or unexported fields
}

func New

func New(store *chstore.Store, interval time.Duration, lock cache.Lock, notifier *notify.Notifier) *Detector

New takes a cache.Lock so multiple replicas don't all open the same anomaly, and a notifier so PROBLEM OPENED transitions email/slack out.

func (*Detector) Start

func (d *Detector) Start(ctx context.Context)

type EvidenceBundle

type EvidenceBundle struct {
	Problem    chstore.Problem            // the triggering problem
	CoFiring   []chstore.Problem          // other OPEN problems on the SAME service
	Signals    []chstore.AnomalyEvent     // active log_pattern / trace_op anomalies on the service
	Deploy     *chstore.RecentDeployEntry // a deploy of the service just before onset
	Neighbors  []NeighborProblem          // open problems on direct topology neighbours
	Confidence int                        // distinct corroborating evidence types present (incl. the trigger)
	// Deep (v0.9.510) — P1 soruşturmasının topladığı ek kanıt: pod
	// doygunluğu, exception grupları, log şablonları, yavaş operasyonlar
	// + "neye bakıldı" denetim izi. YALNIZ P1 anchor'larında dolu
	// (investigation.go, maliyet kapısı). Boş = derin soruşturma koşmadı;
	// prompt o zaman bugünküyle birebir aynı kalır.
	Deep chstore.DeepEvidence
}

EvidenceBundle is the corroborating signal set for one triggering Problem.

type ExceptionExplainInput

type ExceptionExplainInput struct {
	User     string   // narration user prompt'u
	EvTraces []string // kanıt trace id'leri (örnek tablosu kutulaması)
	EvSpans  []string // kanıt span id'leri
	// LogsBlock — User'ın İÇİNDEKİ log bölümü, ayrıca taşınır (v0.9.482).
	// AI çekmecesi sohbeti bu paketi narration bütçesine sığdırırken önce
	// span/trace listesini budar, LOGLARI KORUR — operatörün takip
	// soruları ("logda ne yazıyor") log içeriğine dairdir. Explain
	// yolunda kullanılmaz; User bayt-bayt eskisidir.
	LogsBlock string
}

ExceptionExplainInput — kurulan girdi + deterministik kanıt.

func BuildExceptionExplainInput

func BuildExceptionExplainInput(ctx context.Context, store *chstore.Store, logs logstore.Store, g *chstore.ExceptionGroup) ExceptionExplainInput

BuildExceptionExplainInput — grup meta + occurrence trendi + temsilî stacktrace + en yeni örneğin TAM trace'i + o trace'in logları + FirstSeen-merkezli deploy penceresi. logs nil olabilir (CH-only kurulum ya da işçi bağlamı) — log bloğu atlanır.

type ExceptionExplainer

type ExceptionExplainer struct {
	// contains filtered or unexported fields
}

func NewExceptionExplainer

func NewExceptionExplainer(store *chstore.Store, logs logstore.Store, cop *copilot.Service, lock cache.Lock) *ExceptionExplainer

func (*ExceptionExplainer) Start

func (e *ExceptionExplainer) Start(ctx context.Context)

type LogPatternAnomaly

type LogPatternAnomaly struct {
	Pattern       string  `json:"pattern"` // human-readable name
	Regex         string  `json:"regex"`   // the raw re2 used
	Kind          string  `json:"kind"`    // "new" | "spike"
	CurrentCount  uint64  `json:"currentCount"`
	BaselineCount uint64  `json:"baselineCount"` // trailing window
	Ratio         float64 `json:"ratio"`         // current / max(baseline,1)
	Service       string  `json:"service"`       // service emitting most matches in current window
	Sample        string  `json:"sample"`        // representative log body, truncated
	LastSeenNs    int64   `json:"lastSeenNs"`
	// TopServices — v0.5.287. Per-service breakdown of current
	// window hits, top 5, count desc. The /logs LogPatternStrip
	// renders these as a rosette under the pattern chip so the
	// operator can see "OOMKilled fires on foo-svc (12) and
	// bar-svc (3)" without expanding or filtering.
	TopServices []logstore.PatternServiceHit `json:"topServices,omitempty"`
	// Tokens — v0.5.306. The lowercase body substrings any of
	// which guarantees a regex match. Exposed to the frontend
	// so the /anomalies "logs ↗" link can build a precise OR
	// query that lands on the actual log lines, instead of
	// the previous behaviour (link only narrowed to the
	// service). E.g. "Disk full" carries
	//   ["no space left", "disk full", "enospc"]
	// → link becomes /logs?service=X&q=("no space left" OR
	// "disk full" OR "enospc"). Curated per pattern in the
	// patterns[] slice below.
	Tokens []string `json:"tokens,omitempty"`
}

LogPatternAnomaly is one production-grade signal pattern that either started firing for the first time within the window (Kind="new") or jumped 2x+ over its trailing baseline (Kind="spike"). What an SRE wants to see in their morning inbox: not the raw log volume, just what changed.

func DetectLogPatterns

func DetectLogPatterns(ctx context.Context, store logstore.Store, window time.Duration) ([]LogPatternAnomaly, error)

DetectLogPatterns runs each pattern against the raw `logs` CH table over a current window + a much longer trailing baseline (default: 5-min current vs 1-hour trailing). Returns only the patterns that changed significantly: brand new, or 2x+ over the per-window-length baseline rate.

The asymmetric windows keep an anomaly visible for ~1 hour after it first fires — a 5m-vs-5m comparison has the spike fall into baseline within minutes, which makes the anomaly section flicker. With a 1h baseline the same spike stays visible until the baseline absorbs it.

Performance:

• One query per pattern (combines current + baseline via countIf), N=11 → 11 round-trips total instead of 22.
• All N queries fire in parallel; total cold-cache cost is bounded by the slowest single query, not the sum.
• Each query is partition-pruned to ~10 minutes of logs and the regex runs against the LowCardinality body column.
• Caller should cache the result for 60s; the detector is idempotent and the cache absorbs page reloads.

At 1B logs/day this completes in well under a second cold; warm requests serve directly from Redis.

v0.5.241 — refactored to take a logstore.Store instead of *chstore.Store so the detector works against BOTH the CH and the ES log backend. CH path remains the regex+tokenbf prefilter route; ES path uses query_string token-OR against the body field (regex is ignored; tokens must be zero-false-negative vs the regex). Cross-backend correctness depends on detector authors keeping the Tokens list synchronized with the Regex.

type LogTemplateAnomaly

type LogTemplateAnomaly struct {
	TemplateID  string
	Template    string
	Service     string // first service in the template's seen-services array
	FirstSeenNs int64
	LastSeenNs  int64
	TotalCount  uint64
	Sample      string
}

LogTemplateAnomaly is one Drain template that crossed into existence within `window`. The same fingerprint (template ID) surfaces every tick until the row falls outside the window — UpsertAnomalyEvent dedupes via ReplacingMergeTree(version).

func DetectNewLogTemplates

func DetectNewLogTemplates(ctx context.Context, store *chstore.Store, window time.Duration) ([]LogTemplateAnomaly, error)

DetectNewLogTemplates returns Drain templates whose first_seen landed inside [now-window, now]. The templater puller writes these rows; we just query them.

Window guidance: the templater puller runs every 5min by default (see templater/puller.go), so window=10min covers the last two puller cycles — enough to absorb any clock drift between puller + recorder. The anomaly_event dedupe collapses repeated detections.

type NeighborProblem

type NeighborProblem struct {
	Problem   chstore.Problem
	Direction string  // "calls" (trigger → neighbour) | "called_by" (neighbour → trigger)
	Score     float64 // propagation score in [0,1] — downstream suspects only; 0 for upstream / no-error edges
	Hops      int     // topology distance: 1 = direct, 2 = transitive downstream (decayed); 0 = unscored
}

NeighborProblem is an open problem on a service adjacent to the trigger, annotated with the call direction relative to the triggering service and (Faz 6) its root-cause propagation score.

type ProblemExplainer

type ProblemExplainer struct {
	// contains filtered or unexported fields
}

func NewProblemExplainer

func NewProblemExplainer(store *chstore.Store, cop *copilot.Service, lock cache.Lock) *ProblemExplainer

func (*ProblemExplainer) Start

func (e *ProblemExplainer) Start(ctx context.Context)

Start runs the explainer loop until ctx is cancelled. Initial tick fires immediately so a problem opened during pod startup gets explained without waiting a full interval.

type Recorder

type Recorder struct {
	// contains filtered or unexported fields
}

Recorder is the persistence side of the anomaly system. The existing detectors (log_patterns.go, trace_ops.go) are pure readers that compute "what looks anomalous in the last 5 minutes". This recorder runs them on a tick and upserts an AnomalyEvent row per detection so the operator can later answer "what fired in the last hour, even if it cleared".

Events go through ReplacingMergeTree keyed on the (kind, pattern, service) fingerprint — same pattern firing across two consecutive ticks updates the same row, advancing last_seen and tracking peak_ratio. "Cleared" status is derived in the query layer from last_seen freshness, so we don't need a separate sweep job.

func NewRecorder

func NewRecorder(store *chstore.Store, logs logstore.Store, interval, window time.Duration, lock cache.Lock) *Recorder

NewRecorder builds a recorder that ticks every `interval` and each tick scans `window` of recent data. Default 60s tick is fine for the human-grade "anomalies in the last hour" UX — faster ticks just multiply CH load with no operator benefit.

func (*Recorder) Start

func (r *Recorder) Start(ctx context.Context)

Start kicks the recorder into a goroutine. Caller cancels via the supplied context. Multi-replica deployments use the lock to elect a single writer per tick — the lock TTL is generous (the interval × 2) so a slow CH doesn't kill liveness.

type RootCauseSynthesizer

type RootCauseSynthesizer struct {
	// contains filtered or unexported fields
}

func NewRootCauseSynthesizer

func NewRootCauseSynthesizer(store *chstore.Store, lock cache.Lock) *RootCauseSynthesizer

func (*RootCauseSynthesizer) Start

func (s *RootCauseSynthesizer) Start(ctx context.Context)

Start runs the synthesis loop until ctx is cancelled. Initial tick fires immediately so an anchor that opened during pod startup gets a hypothesis without waiting a full interval (same as the explainer).

type TraceOpAnomaly

type TraceOpAnomaly struct {
	Service        string  `json:"service"`
	Operation      string  `json:"operation"`
	Kind           string  `json:"kind"` // "new_error" | "error_spike"
	CurrentErrors  uint64  `json:"currentErrors"`
	BaselineErrors uint64  `json:"baselineErrors"`
	Ratio          float64 `json:"ratio"` // current / max(baseline, 1)
	// CurrentCalls — the denominator the qualification now insists on
	// (v0.9.327). Shipping it means the row can say "42 errors of 3,100
	// calls" instead of a bare count the operator has to go look up.
	CurrentCalls  uint64  `json:"currentCalls"`
	ErrorShare    float64 `json:"errorShare"`    // CurrentErrors / CurrentCalls, 0..1
	SampleTraceID string  `json:"sampleTraceId"` // representative trace for one-click drill-in
	LastSeenNs    int64   `json:"lastSeenNs"`
}

TraceOpAnomaly is a per-(service, operation) error or latency signal that's either brand new or up sharply over baseline. Different from the service-wide metric anomaly detector in that it pinpoints the SPECIFIC operation that's misbehaving — the SRE's first question after "is service X broken" is "which endpoint inside X".

func DetectTraceOpAnomalies

func DetectTraceOpAnomalies(ctx context.Context, store *chstore.Store, window time.Duration) ([]TraceOpAnomaly, error)

DetectTraceOpAnomalies finds per-operation error spikes over the last `window` against a longer trailing baseline (1h or 12×window, whichever is larger, capped at 24h).

v0.8.504 (perf raporu #1): pre-MV sürüm her koşuda raw spans'i İKİ kez tarıyordu (window cur + 1-24h base GROUP BY) — 60s tick'te lokalde bile 10-19s/koşu, ~700K satır; 1B span/gün'de dakikada milyonlarca satır. Sayımlar artık operation_summary_5m'den okunur (MV-first invariant: "raw spans for an aggregate = bug"); raw spans'e yalnız KALİFİYE ≤50 çiftin örnek trace'i için dar, service_name-prefix'li ikinci sorgu gider. Bedel: pencereler 5m bucket'a hizalanır — tespit en fazla ~5dk gecikir (v0.8.315/316'da kabul edilmiş desen).

The asymmetric baseline (window vs ≥1h trailing) keeps fresh spikes visible for ~1 hour — a window-vs-window comparison would have the spike fall into baseline within minutes, flickering the anomaly section as windows slide.