templater

Documentation

Overview

Package templater turns raw log lines into stable templates via the Drain-3 algorithm (online template extraction with a fixed- depth tree). Backend-agnostic: a puller goroutine in this package periodically samples logs from whichever backend is wired (CH or ES), feeds them to Drain, and upserts the resulting templates into chstore.LogTemplate so the operator can see "what shapes of log are firing right now".

Tuned for Java-heavy production traffic where stack traces + MDC contexts dominate, so the masker preserves class names and logger paths while masking IDs and timestamps.

Index

• func IsSensitiveLine(body string) bool
• func LooksLikeOpaqueID(tok string) bool
• func Mask(line string) string
• func NormalizeOperation(name, kind, httpMethod, httpRoute, dbSystem, dbStatement string) string
• func Tokenize(line string) []string
• type Cluster
  • func (c *Cluster) TemplateString() string
• type Drain
  • func NewDrain() *Drain
  • func (d *Drain) Add(line, service string, tsNs int64) *Cluster
  • func (d *Drain) Reset()
  • func (d *Drain) Snapshot() []*Cluster
  • func (d *Drain) Stats() DrainStats
• type DrainStats
• type Puller
  • func New(store *chstore.Store, logs logstore.Store, interval time.Duration, sample int, ...) *Puller
  • func (p *Puller) Start(ctx context.Context)

Functions

func IsSensitiveLine

func IsSensitiveLine(body string) bool

IsSensitiveLine returns true when the body almost certainly carries an auth credential or per-request opaque correlation ID. Such lines are excluded from templating in the puller.

func LooksLikeOpaqueID

func LooksLikeOpaqueID(tok string) bool

LooksLikeOpaqueID returns true when a single token looks like a per-request opaque value rather than a real keyword. Surfaced via /api/logs/patterns post-filter to keep the "Live patterns" panel free of JWT fragments, UUIDs, trace IDs and base64 session ids that statistically score high but mean nothing to a human reader.

Rules (post-filter — token already passed token-worth filters in the backend):

• JWT-shaped (starts with eyJ, base64url chars, ≥16 chars)
• UUID (8-4-4-4-12 hex with dashes; case-insensitive)
• Hex strings ≥16 chars (trace IDs are 32-hex, span IDs 16-hex, MD5 32-hex, SHA-1 40-hex, SHA-256 64-hex)
• All-digit strings ≥4 digits (sequence IDs, request counters, port numbers, epoch timestamps). 3-digit cutoff preserved so HTTP status codes (200, 404, 500) still surface as meaningful patterns.
• Long base64url ≥24 chars with no English-letter run ≥5
• High-digit-ratio tokens (≥60% digits AND length ≥ 10) — catches mixed alphanumeric IDs like "a1b2c3d4e5f6"
• Consonant-only tokens length ≥ 10 — random strings emitted by hash truncation typically have ~no vowels

v0.5.397 — operator-reported: panel still showed UUIDs + trace ids + numeric request ids despite v0.5.336's initial filter. The original rules only caught JWTs and pure-base64 strings; the broader ID shapes leaked through.

func Mask

func Mask(line string) string

Mask returns the input line with every variable substring replaced by "<*>". Order-dependent regex set defined in maskRules — see comments there for ordering rationale.

func NormalizeOperation

func NormalizeOperation(name, kind, httpMethod, httpRoute, dbSystem, dbStatement string) string

NormalizeOperation folds a raw span into a stable, human-readable operation-shape group. Source priority: http.route > DB-statement-stripped > generic name-segment normalization.

name       — raw span name (operation name)
kind       — OTel span kind string (e.g. "server","client"); reserved
             for future shape decisions, currently unused.
httpMethod — http.method attr ("" for client spans without one)
httpRoute  — http.route attr (already templated by instrumentation,
             e.g. "/users/{id}")
dbSystem   — db.system attr ("" when not a DB span)
dbStatement— db.statement attr (raw SQL/command)

Returns "" when nothing applies (no group).

func Tokenize

func Tokenize(line string) []string

Tokenize splits on whitespace AFTER masking, preserving the "<*>" tokens as standalone units. Drain operates on token arrays; whitespace inside a value is already neutralised by the mask pass.

Types

type Cluster

type Cluster struct {
	ID        string
	Template  []string // tokens; "<*>" marks variable positions
	Count     uint64
	FirstSeen int64 // unix ns
	LastSeen  int64 // unix ns
	Services  []string
	Sample    string // representative raw line (for UI hover)
}

Cluster is one extracted log template plus running stats. ID is a stable sha1 over the template tokens — same shape + same order always produces the same id, so duplicate processing across pulls is idempotent.

func (*Cluster) TemplateString

func (c *Cluster) TemplateString() string

TemplateString joins the template tokens with single spaces — the canonical human-readable form for storage + display.

type Drain

type Drain struct {
	Depth        int     // tree depth (default 4)
	MaxChildren  int     // max children per non-leaf (default 100)
	SimThreshold float64 // similarity threshold (default 0.4)
	// contains filtered or unexported fields
}

Drain implements a faithful subset of the Drain-3 online log template extractor (He, Pinjia et al., "Drain: An Online Log Parsing Approach with Fixed Depth Tree"). Each log line walks a fixed-depth tree keyed by:

• Layer 1 — token count (an 11-token "Failed to connect" line never collides with a 4-token "GET /foo 200" line)
• Layer 2 — first token (literal or "<*>")
• Layer 3..Depth — extended literal tokens (only the non-masked positions discriminate)

The leaf node holds a list of clusters; each cluster's template is a token list with "<*>" at variable positions. A new line either:

• hits an existing cluster (similarity ≥ SimThreshold) → refine the template (mark differing positions as "<*>"), bump count.
• hits no cluster → new cluster.

Tradeoffs vs the canonical implementation:

• No LRU eviction of clusters — log_templates is the persistent ledger so memory growth is bounded by the periodic save+compact in the puller.
• MaxChildren is enforced at the tree, not via priority eviction — once a non-leaf node has MaxChildren distinct buckets we fall through to a single "<*>" child so a pathologically variable layer doesn't explode the tree.

func NewDrain

func NewDrain() *Drain

NewDrain builds a tree with the canonical Drain-3 defaults: depth 4, max-children 100, similarity 0.4. Tuned in the paper across multiple log datasets; we've kept these values because they're well-validated and tuning at billion-line scale isn't a quick test.

func (*Drain) Add

func (d *Drain) Add(line, service string, tsNs int64) *Cluster

Add processes one log line: tokenises + masks, walks/extends the tree, returns the matched (or created) cluster. Service + tsNs are recorded on the cluster for the periodic save step. Threadsafe.

func (*Drain) Reset

func (d *Drain) Reset()

Reset clears the tree. Caller persists Snapshot() output first, then calls Reset() to bound in-process memory; the next pull warms a fresh tree against the recent window.

func (*Drain) Snapshot

func (d *Drain) Snapshot() []*Cluster

Snapshot returns every cluster currently in the tree. Used by the puller to flush state into chstore after each batch. The caller treats the slice as read-only; the underlying Cluster pointers continue to mutate on subsequent Add() calls.

func (*Drain) Stats

func (d *Drain) Stats() DrainStats

Stats returns the current health snapshot. Cheap — walks the tree once to count nodes + clusters.

type DrainStats

type DrainStats struct {
	NodeCount    int
	ClusterCount int
	OverflowHits uint64
	ResetCount   uint64
}

DrainStats is the per-instance health snapshot the puller emits at the end of each tick. NodeCount approximates the in-memory tree size; OverflowHits is the cumulative count of childOrCreate calls that fell through to the wildcard child (each one is a hint that the layer's source domain has high cardinality the template tree can't represent well). High values suggest the input stream has more distinct shapes than MaxChildren can hold without over-collapsing.

type Puller

type Puller struct {
	// contains filtered or unexported fields
}

Puller is the background goroutine that drives the Drain templater. Each tick (default 5 min) it samples N=1000 recent logs from whichever backend is wired, feeds them to Drain, then flushes the resulting clusters into chstore.LogTemplate.

Sample-based on purpose — at billion-log/day a full pass per tick would dominate the cluster's load. 1000 samples / 5 min is enough to surface every template emitting at ≥10/min without significant under-sampling (Drain merges on similarity so a slightly truncated input still produces the same template).

Lock-gated for HA: only one replica per tick writes; the others skip cleanly. The lease TTL is generous (2× tick) so a slow tick doesn't get a second writer fighting it.

func New

func New(store *chstore.Store, logs logstore.Store, interval time.Duration, sample int, lock cache.Lock) *Puller

New returns a puller with sane defaults. interval defaults to 5min; sample defaults to 1000 docs.

func (*Puller) Start

func (p *Puller) Start(ctx context.Context)

Start runs until ctx is cancelled. Each tick:

1. Try the HA lock; skip if another replica got it.
2. Pull ~sample docs from the last `interval` window.
3. Feed every doc to the Drain templater.
4. Snapshot the resulting clusters; upsert each into CH.
5. Reset Drain — next tick starts cold against the next window. Persistent state lives in log_templates; the in-memory tree is just a per-batch scratchpad.

The pre-cancel select gives ctx a fair shake on shutdown so a long-running pull doesn't block ProcessExit.