notify

Documentation

Overview

Package notify dispatches Problem alerts to user-configured notification channels: email (SMTP), Slack/Mattermost (incoming-webhook compatible), generic webhook (raw JSON POST), and WhatsApp (via Twilio's Messages API).

Two design decisions worth calling out:

1. SMTP credentials live in the system_settings ClickHouse table — not in config.yaml — so the admin UI can rotate them without a restart. Reads happen via the in-memory cache below; writes invalidate it.
2. Sending is fire-and-forget from the evaluator/anomaly tick. Failures are logged but do not block or retry — alert spam from a flaky SMTP is worse than a missed alert (which the operator notices anyway via the Problems page).

v0.9.587 — problem bildirimlerinde tekrar tabanı (dedup floor).

NEDEN: bu oturumda İKİ KEZ bildirim fırtınası oldu ve ikisi de aynı şekildeydi — bir dedektör, ZATEN AÇIK bir problemi "açık değil" sanıp her tikte yeniden açtı:

v0.9.575  OpenProblemsSnapshot ByID araması hep ıskalıyordu
v0.9.585  kronik tipler her tikte yeniden eskale oluyordu

İkisinde de TETİKLEYİCİYİ kapattım. Ama koruma katmanı YOKTU: aynı şekilli ÜÇÜNCÜ bir hata, operatörün Slack'ine ve çağrı cihazına dakikada bir sayfa göndermeye devam ederdi. Bu dosya o tabanı koyar.

NE DEĞİL: bu bir bildirim POLİTİKASI değil. Anahtar (problem, kanal, durum, ciddiyet) dörtlüsü olduğu için yalnızca BİREBİR AYNI gönderimi bastırabilir:

• eskalasyon warning→critical → ciddiyet farklı → GEÇER
• çözülme bildirimi → durum farklı → GEÇER
• başka bir kanal → kanal farklı → GEÇER

Yani bastırdığı tek şey, 60 saniye önce gönderilmiş olanın aynısı — bunu kimse istemez.

NEREDE: SendProblemAlert'in kanal döngüsünde, sendOne'da DEĞİL. sendOne "test gönder" ve runbook-tamamlandı yollarını da taşıyor ve onlar TEKRARLANAN DURUM BİLDİRİMİ değil, OPERATÖR EYLEMİ: admin "Test gönder"e iki kez basarsa iki test gitmeli. Kapıyı problem yayılımına koymak bu ayrımı yapı gereği korur.

Index

• func ValidateWebhookTemplate(tmpl string) error
• type EmailChannelConfig
• type EventPublisher
• type ExceptionNotifier
  • func NewExceptionNotifier(store *chstore.Store, n *Notifier, lock cache.Lock) *ExceptionNotifier
  • func (e *ExceptionNotifier) Start(ctx context.Context)
• type Notifier
  • func New(store *chstore.Store) *Notifier
  • func (n *Notifier) ListZoomChannels(ctx context.Context, ...) ([]ZoomChannel, error)
  • func (n *Notifier) PublicURL() string
  • func (n *Notifier) Publish(kind string, payload any)
  • func (n *Notifier) SMTP(ctx context.Context) SMTPSettings
  • func (n *Notifier) SaveSMTP(ctx context.Context, s SMTPSettings) error
  • func (n *Notifier) SendMail(ctx context.Context, to []string, subject, body string) error
  • func (n *Notifier) SendProblemAlert(ctx context.Context, p chstore.Problem)
  • func (n *Notifier) SendRunbookComplete(ctx context.Context, e chstore.RunbookExecution, channelTypes []string)
  • func (n *Notifier) SendTest(ctx context.Context, c chstore.NotificationChannel) error
  • func (n *Notifier) SetEventBus(bus EventPublisher)
  • func (n *Notifier) SetPublicURL(u string)
  • func (n *Notifier) SetSMTPCacheTTL(d time.Duration)
• type SMTPSettings
  • func (s SMTPSettings) Configured() bool
• type SlackChannelConfig
• type TeamsChannelConfig
• type WebhookChannelConfig
• type WhatsAppChannelConfig
• type ZoomChannel
• type ZoomChatChannelConfig

Functions

func ValidateWebhookTemplate

func ValidateWebhookTemplate(tmpl string) error

ValidateWebhookTemplate — kanal KAYDINDA çağrılır: parse + örnek problem'le deneme render'ı; hatalı şablon hiç kaydedilmez.

Types

type EmailChannelConfig

type EmailChannelConfig struct {
	Recipients []string `json:"recipients"` // one or more; comma-split also supported in UI
}

EmailChannelConfig is the per-channel JSON for type=email.

type EventPublisher

type EventPublisher interface {
	Publish(kind string, payload any)
}

EventPublisher is the minimum interface we need from the SSE broker. Defined here as an interface (rather than depending on internal/sse directly) so the notify package stays import-cycle-free — the broker can use chstore types if it ever wants to without circling back.

type ExceptionNotifier

type ExceptionNotifier struct {
	// contains filtered or unexported fields
}

func NewExceptionNotifier

func NewExceptionNotifier(store *chstore.Store, n *Notifier, lock cache.Lock) *ExceptionNotifier

func (*ExceptionNotifier) Start

func (e *ExceptionNotifier) Start(ctx context.Context)

type Notifier

type Notifier struct {
	// contains filtered or unexported fields
}

Notifier is the small surface the evaluator + anomaly worker call into. Construction is cheap; share one across the process.

func New

func New(store *chstore.Store) *Notifier

func (*Notifier) ListZoomChannels

func (n *Notifier) ListZoomChannels(
	ctx context.Context, accountID, clientID, clientSecret, oauthBaseURL, apiBaseURL string,
	skipVerify bool,
) ([]ZoomChannel, error)

ListZoomChannels fetches every channel the configured S2S OAuth app can see — walks the channels endpoint's pagination (page_size=50, next_page_token) until exhausted or a sane cap is hit. Used by the Settings UI's "List my channels" helper so the operator picks from a searchable list instead of pasting JIDs by hand.

Caps:

• 20 pages of 50 = 1000 channels max. Large Zoom workspaces can have thousands, but past 1000 the picker becomes unusable anyway and we'd rather fail loud than return a truncated list silently. The error message instructs the operator to narrow scope (use a less-privileged service account or filter on the receiving end).
• 25s overall wall-clock — context cancel breaks the loop so an unresponsive Zoom doesn't hang the request.

func (*Notifier) PublicURL

func (n *Notifier) PublicURL() string

PublicURL reads the configured base URL — safe to call from any goroutine; helpers below use it to build per-problem deep links.

func (*Notifier) Publish

func (n *Notifier) Publish(kind string, payload any)

Publish surfaces the event bus to other workers (evaluator, anomaly detector) that already have a Notifier reference but shouldn't import the broker directly. Safe pass-through; nil bus = no-op.

func (*Notifier) SMTP

func (n *Notifier) SMTP(ctx context.Context) SMTPSettings

SMTP returns the cached settings (read-through). Cache TTL is config-driven (cfg.Background.SMTPCacheTTL) so operators can dial it up on big CH clusters; defaults to 30s when unset. Safe for concurrent callers.

func (*Notifier) SaveSMTP

func (n *Notifier) SaveSMTP(ctx context.Context, s SMTPSettings) error

SaveSMTP persists new settings and busts the in-memory cache.

func (*Notifier) SendMail

func (n *Notifier) SendMail(ctx context.Context, to []string, subject, body string) error

SendMail is a generic plain-text mailer over the operator's configured SMTP — used by surfaces that aren't problem alerts (status-page subscriber double-opt-in, future password-reset flows, etc.). Returns an error when SMTP isn't configured so the caller can decide whether the operation should fail or continue silently (status-page subscribe falls back to "we recorded your email; the operator will deliver the link manually" when SMTP isn't wired up).

func (*Notifier) SendProblemAlert

func (n *Notifier) SendProblemAlert(ctx context.Context, p chstore.Problem)

SendProblemAlert fans out a problem to every channel that wants this severity. Errors are logged per-channel; partial failures don't abort the rest.

Also fires an SSE event so the browser-side React Query caches invalidate immediately rather than waiting for the next poll. Kind is "problem.open" / "problem.resolve" so the client can decide what to invalidate (open events bump the sidebar badge; resolve events also do, plus drop a row from the open list).

func (*Notifier) SendRunbookComplete

func (n *Notifier) SendRunbookComplete(ctx context.Context, e chstore.RunbookExecution, channelTypes []string)

SendRunbookComplete fans a finished runbook execution out to the configured notification channels and publishes a runbook.complete SSE event. It reuses the Problem-shaped channel path via a synthetic Problem (the channels format Severity/Service/RuleName/Description). Called only when the runbook opted in (NotifyOnComplete). completed → info; failed → critical. (v0.7.7)

func (*Notifier) SendTest

func (n *Notifier) SendTest(ctx context.Context, c chstore.NotificationChannel) error

SendTest dispatches a synthetic Problem to a single channel — used by the "Send test" button on the settings UI so admins can verify config without waiting for a real incident.

func (*Notifier) SetEventBus

func (n *Notifier) SetEventBus(bus EventPublisher)

SetEventBus wires the SSE broker. Called once at startup; the notifier stores the reference and publishes Problem.* events from SendProblemAlert.

func (*Notifier) SetPublicURL

func (n *Notifier) SetPublicURL(u string)

SetPublicURL configures the deep-link base for notification bodies. Trims trailing slashes so URL composition is just base + "/route" with no double slash.

func (*Notifier) SetSMTPCacheTTL

func (n *Notifier) SetSMTPCacheTTL(d time.Duration)

SetSMTPCacheTTL lets main.go wire the configurable refresh interval after construction. Zero is treated as "use the 30s default" so unit tests that build a Notifier without touching config still behave correctly.

type SMTPSettings

type SMTPSettings struct {
	Host       string `json:"host"`
	Port       int    `json:"port"`
	Username   string `json:"username"`
	Password   string `json:"password"`
	From       string `json:"from"`
	FromName   string `json:"fromName"`
	StartTLS   bool   `json:"startTLS"`
	SkipVerify bool   `json:"skipVerify"`
}

SMTPSettings is the JSON shape we persist under system_settings["smtp"].

func (SMTPSettings) Configured

func (s SMTPSettings) Configured() bool

type SlackChannelConfig

type SlackChannelConfig struct {
	WebhookURL string `json:"webhookUrl"`
}

SlackChannelConfig powers both type=slack and type=mattermost — they accept the same incoming-webhook JSON shape.

type TeamsChannelConfig

type TeamsChannelConfig struct {
	WebhookURL string `json:"webhookUrl"`
}

TeamsChannelConfig — Microsoft Teams incoming webhook. Same shape as Slack at the URL level (single endpoint), different payload format (Office-365 Connector / Adaptive Card JSON).

type WebhookChannelConfig

type WebhookChannelConfig struct {
	URL string `json:"url"`
	// Headers (v0.8.445) — özel istek başlıkları; harici agent
	// platformları (GenAI Studio) auth key'lerini buradan taşır.
	// Değerler kanal config'inde durur — kanal yönetimi zaten
	// admin-only, Slack webhook URL'leriyle aynı gizlilik sınıfı.
	Headers map[string]string `json:"headers,omitempty"`
	// BodyTemplate (v0.8.445) — opsiyonel Go text/template gövdesi;
	// boşken eski {problem, coremetryUrl} JSON'ı aynen gider (geriye
	// uyumlu). Alanlar: {{.Problem.*}} (chstore.Problem) ve
	// {{.CoremetryURL}}. Şablon hatası kanal kaydında yakalanır;
	// runtime render hatasında default payload gönderilir + log.
	BodyTemplate string `json:"bodyTemplate,omitempty"`
}

WebhookChannelConfig is the generic JSON-POST channel; the body is the raw chstore.Problem so the receiver can route it however it likes.

type WhatsAppChannelConfig

type WhatsAppChannelConfig struct {
	AccountSid string   `json:"accountSid"`
	AuthToken  string   `json:"authToken"`
	From       string   `json:"from"`
	To         []string `json:"to"`
}

WhatsAppChannelConfig wraps Twilio's WhatsApp messaging API.

AccountSid + AuthToken are the standard Twilio API credentials. From is the sender number including the "whatsapp:" prefix and E.164 formatting (e.g. "whatsapp:+14155238886" — the Twilio sandbox number). To is one or more recipient numbers, same format.

Twilio is the de-facto standard for programmatic WhatsApp because it owns the relationship with Meta on the user's behalf. Meta's direct Cloud API works too but requires per-template approval, not viable for ad-hoc alert text.

type ZoomChannel

type ZoomChannel struct {
	ID   string `json:"id"`   // short id (rarely useful — included so the operator can search by it)
	JID  string `json:"jid"`  // the value that goes into `to_channel` on the messages API
	Name string `json:"name"` // human-readable channel name
	Type int    `json:"type"` // 1=DM, 2=Group, 3=Public Channel, 4=Private Channel
}

ZoomChannel is one row of the channel-picker list the Settings UI uses to help operators pick a Channel ID without memorising JIDs. Mirrors Zoom's /chat/users/me/channels API shape — we only surface the fields a human needs to disambiguate one channel from another.

type ZoomChatChannelConfig

type ZoomChatChannelConfig struct {
	AccountID    string `json:"accountId"`
	ClientID     string `json:"clientId"`
	ClientSecret string `json:"clientSecret"`
	ChannelID    string `json:"channelId,omitempty"`
	ToContact    string `json:"toContact,omitempty"`
	// APIBaseURL overrides the default `https://api.zoom.us` for
	// the chat messages endpoint. Optional — empty keeps the
	// public Zoom API. Banks routing outbound traffic through a
	// corporate proxy (api.zoom.us isn't directly reachable from
	// the perimeter) point this at their proxy host. Test
	// environments do the same for mock servers.
	APIBaseURL string `json:"apiBaseUrl,omitempty"`
	// OAuthBaseURL overrides the default `https://zoom.us` for
	// the OAuth token endpoint. Same use case as APIBaseURL but
	// the OAuth host differs from the API host in Zoom's
	// deployment, so it's a separate knob. Most proxies expose
	// both — operators typically fill both fields with the same
	// prefix or leave both empty.
	OAuthBaseURL string `json:"oauthBaseUrl,omitempty"`
	// InsecureSkipVerify disables TLS certificate validation on
	// the OAuth + chat HTTP calls. Use only when the operator
	// has routed Zoom traffic through a corporate proxy that
	// terminates TLS with a private CA the pod doesn't have in
	// its trust store. Setting this is the equivalent of
	// `curl -k` — turns off MITM detection, so reserve it for
	// trusted corp networks. Public Zoom hosts MUST verify.
	InsecureSkipVerify bool `json:"insecureSkipVerify,omitempty"`
	// Legacy webhook fields — kept for graceful migration from
	// pre-v0.4.78 configs. When AccountID is empty AND
	// WebhookURL is set, sendZoomChat returns a clean
	// "reconfigure required" error rather than silently
	// failing. New channels never write these.
	WebhookURL        string `json:"webhookUrl,omitempty"`
	VerificationToken string `json:"verificationToken,omitempty"`
}

ZoomChatChannelConfig — Zoom Chat via Server-to-Server OAuth. Replaces the older incoming-webhook flow because banks want the proper REST API (auditable, account-scoped, no webhook URL that leaks if dumped).

Fields are exactly what Zoom's marketplace app dialog hands the admin:

• AccountID: the Zoom account UUID (Zoom calls it Account ID)
• ClientID: OAuth client_id from the Server-to-Server app
• ClientSecret: OAuth client_secret (write-only after save — the UI never echoes it back)
• ChannelID: the channel JID for the target chat channel (Zoom's "to_channel" field on the messages API)
• ToContact: optional fallback — email of a single contact to DM when ChannelID is empty

On send the notifier exchanges credentials for an access token via /oauth/token and POSTs the chat message to /v2/chat/users/me/messages. Tokens cache for ~1h on a per-(account_id, client_id) key so a burst of N alerts doesn't N-times the OAuth round-trip.