Documentation
¶
Overview ¶
Package rules loads and validates the YAML rule configuration used to match and structure log lines.
Index ¶
Constants ¶
const ( FieldMetaSourceFile = "source_file" FieldMetaSourceLine = "source_line" )
FieldMetaSourceFile and FieldMetaSourceLine are the only two values Field.Meta accepts. See parse.SourceMeta for how they're resolved.
Variables ¶
This section is empty.
Functions ¶
func Collapse ¶
Collapse rewrites every rule whose pattern/fields exactly match a registered preset (after normalization, see normalizePattern) into `preset: <name>`, leaving everything else untouched. Returns the rewritten YAML and the number of rules it collapsed.
Types ¶
type Config ¶
type Config struct {
Rules []Rule `yaml:"rules"`
// Mask declares global, rule-independent redaction/hashing applied to
// every rule's structured/converted data the same way - see MaskRule.
Mask []MaskRule `yaml:"mask"`
// Ignore declares rules.yaml-wide conditions for skipping raw lines
// before pattern matching - see IgnoreConfig.
Ignore IgnoreConfig `yaml:"ignore"`
// Compression optionally sets the output Parquet compression codec and
// level; unset fields fall back to the CLI flags, then to the default
// (see internal/compression).
Compression compression.Settings `yaml:"compression"`
// RowGroup optionally caps the number of rows per Parquet row group on
// every output file; unset falls back to the CLI flag, then to
// unlimited (see internal/rowgroup).
RowGroup rowgroup.Settings `yaml:"row_group"`
}
Config is the top-level rules.yaml document.
type Field ¶
type Field struct {
Name string `yaml:"-"`
Type string `yaml:"type"`
Format string `yaml:"format"`
Replace []ReplaceRule `yaml:"replace"`
Normalize []NormalizeRule `yaml:"normalize"`
// Key, if set, takes this field's raw value from the rule's parsed
// structured data (see Rule.Structured) under this key name, instead
// of from a same-named pattern capture group.
Key string `yaml:"key"`
// Extra, if true, collects every structured-data key not consumed by
// another field's Key into this field as a JSON string. At most one
// field per rule may set Extra.
Extra bool `yaml:"extra"`
// Meta, if set to FieldMetaSourceFile or FieldMetaSourceLine, takes
// this field's raw value from the current input line's source
// metadata (see parse.SourceMeta) instead of a pattern capture group
// or structured data. Unlike Key/Extra, a Meta field never reads
// structured data, so it does not require the rule to declare
// Structured. Empty for every field that isn't opted in.
Meta string `yaml:"meta"`
// Optional, if true, lets this field's structured data key be absent
// (or present with an empty value, for any type other than string) -
// see parse.Convert - instead of that being a conversion error that
// falls back to the next candidate rule. Only valid combined with Key;
// Validate rejects it on a timestamp field, since a nil timestamp
// could otherwise silently drop a row picked as a multi-file merge key
// (see internal/convert.mergeKeyField).
Optional bool `yaml:"optional"`
// ResolvedFormat is Format resolved once by ResolveFormat, at Load
// time - see TimeFormat. Only meaningful when Type == "timestamp".
ResolvedFormat TimeFormat `yaml:"-"`
}
Field describes how a named capture group should be typed and normalized. Name is set by Rule's custom UnmarshalYAML, the only place in this package that knows the field's declaration order in the source YAML - see Rule.
type IgnoreConfig ¶
type IgnoreConfig struct {
// Patterns is a list of regexps; a line matching ANY of them (partial
// match, via regexp.MatchString) is ignored.
Patterns []string `yaml:"patterns"`
PatternsRe []*regexp.Regexp `yaml:"-"`
// MaxLength ignores a line whose byte length exceeds this value. <= 0
// means unlimited (the zero value, so an empty ignore: block ignores
// nothing).
MaxLength int `yaml:"max_length"`
// InvalidUTF8 ignores a line that isn't valid UTF-8 (utf8.ValidString).
InvalidUTF8 bool `yaml:"invalid_utf8"`
// Empty ignores a line that's empty after strings.TrimSpace.
Empty bool `yaml:"empty"`
}
IgnoreConfig declares rules.yaml-wide conditions for skipping raw input lines before pattern matching even begins. Declared globally under Config.Ignore (not nested under any rule), matching mask:/compression:/ row_group: - but unlike MaskRule, its conditions don't chain: Reason (see ignore.go) evaluates them as an independent OR and returns the first one that matches, in a fixed priority order.
func (*IgnoreConfig) Reason ¶
func (ic *IgnoreConfig) Reason(line string) string
Reason returns "" when line should not be ignored, otherwise the name of the first matching condition, checked in this fixed priority order regardless of declaration order in rules.yaml: "empty", "invalid_utf8", "max_length", "pattern". The order is an implementation choice (cheapest checks first) - see the ignore: design doc.
type MaskRule ¶
type MaskRule struct {
// Type is "key" (Pattern matches a structured-data key name; the whole
// matched key's value is masked) or "pattern" (Pattern matches inside a
// type: string field's value; only the matched substring is masked).
Type string `yaml:"type"`
Pattern string `yaml:"pattern"`
Regexp *regexp.Regexp `yaml:"-"`
// Action is "redact" (replace with Value) or "hash" (replace with a
// truncated, unkeyed SHA-256 digest - deliberately not HMAC, since the
// goal is "same input, same output", not dictionary-attack resistance).
Action string `yaml:"action"`
// Value is the literal replacement for action: redact. Empty string is
// valid (deletes the matched content), matching replace:'s value: ”.
Value string `yaml:"value,omitempty"`
// Length is the SHA-256 hex digest's truncated length (1-64) for
// action: hash.
Length int `yaml:"length,omitempty"`
}
MaskRule redacts or deterministically hashes sensitive data at import time. Declared globally under Config.Mask (not per-rule), and applied to every rule's structured/converted data the same way; declared entries chain in order when more than one matches the same key or value - see parse.SplitMaskRules, applyKeyMaskJSON/applyKeyMaskFlat, and parse.ApplyPatternMask.
type NormalizeRule ¶
type NormalizeRule struct {
Pattern string `yaml:"pattern"`
Value string `yaml:"value"`
Regexp *regexp.Regexp `yaml:"-"`
}
NormalizeRule maps a captured raw string to a canonical value when Pattern matches.
type ReplaceRule ¶
type ReplaceRule struct {
Pattern string `yaml:"pattern"`
Replacement string `yaml:"value"`
Regexp *regexp.Regexp `yaml:"-"`
}
ReplaceRule replaces every regexp match of Pattern within a field's raw value with Replacement (Go's regexp.ReplaceAllString - $1-style capture group backreferences work without any extra code). Declared rules chain: each rule's output becomes the next rule's input.
type Rule ¶
type Rule struct {
Name string `yaml:"name"`
Pattern string `yaml:"pattern"`
Fields []Field `yaml:"-"`
Regexp *regexp.Regexp `yaml:"-"`
// Preset, if set, names a fixed pattern/fields definition from
// presetRegistry (see presets.go) that Load expands into Pattern/
// Fields before compiling. Mutually exclusive with declaring pattern/
// fields directly - Validate checks this using
// declaredPatternOrFields, captured here at YAML-decode time, before
// Load's expansion overwrites Pattern/Fields with the preset's
// values.
Preset string `yaml:"preset"`
// Structured optionally parses one of this rule's captured fields
// (Structured.Source) as JSON/LTSV/logfmt, letting other fields pull
// values out of it by key (see Field.Key/Field.Extra) instead of by
// capture group position. Populated by Rule's UnmarshalYAML, the only
// place that reads the `structured:` key.
Structured *StructuredConfig `yaml:"-"`
// Continuation is an optional regexp pattern. A line matching it while
// this rule has an in-progress multi-line entry open (see
// internal/convert.fileCursor) is folded into that entry instead of
// starting a new one; the pattern's named capture groups say which
// field(s) receive the matched content. Unset means this rule is
// always single-line, matching pre-existing behavior.
Continuation string `yaml:"continuation"`
ContinuationRegexp *regexp.Regexp `yaml:"-"`
// contains filtered or unexported fields
}
Rule is a single pattern-match rule: a name (output type), the regexp pattern used to match lines, and the fields extracted from named capture groups. Fields is ordered the way they were declared in rules.yaml, which becomes the output Parquet file's column order (see internal/schema.Build) - this is why Rule needs its own UnmarshalYAML: decoding `fields:` into a Go map (the obvious approach) would silently lose that declaration order, since map iteration order is unspecified.
func (*Rule) UnmarshalYAML ¶
UnmarshalYAML decodes name and pattern normally, but walks the fields mapping node directly (instead of decoding it into a Go map) so field declaration order is preserved. The YAML syntax for `fields:` is unchanged - still a mapping of name to type/definition - only the in-memory representation differs from what plain struct decoding would produce.
type StructuredConfig ¶
type StructuredConfig struct {
Source string `yaml:"source"`
Format string `yaml:"format"`
// PresetRegexp is set by Load, once, when Format names a registered
// preset instead of json/ltsv/logfmt: it's presetRegistry[Format]'s
// Pattern, compiled (same "compile once at Load time" approach as
// Rule.Regexp). nil when Format is json/ltsv/logfmt. parse.Convert
// branches on this to pick ParsePreset over ParseStructured.
PresetRegexp *regexp.Regexp `yaml:"-"`
}
StructuredConfig configures parsing embedded structured data (JSON, LTSV, logfmt, or a preset's fixed pattern) out of one of a rule's pattern-captured fields, named by Source. Format selects the parser: "json", "ltsv", "logfmt", or the name of an entry in presetRegistry (see presets.go).
type TimeFormat ¶
type TimeFormat struct {
Layout string
EpochUnit time.Duration
// Candidates holds the ordered layout strings to try when this
// TimeFormat came from format: "auto". Empty for every other format.
Candidates []string
// LastGood indexes into Candidates: the position that parsed
// successfully last time, tried first on the next call. Shared via
// pointer across every copy of this TimeFormat (Field is copied by
// value on each call), since input processing is single-threaded (no
// goroutines) - see internal/convert. If that ever changes, updates to
// *LastGood need to become atomic or field-scoped locking needs to be
// added. nil for every non-auto format. Exported (like Layout/
// EpochUnit) so internal/parse, a different package, can read and
// update it.
LastGood *int
}
TimeFormat is a timestamp Field's Format string, resolved once (at rules.Load time, via ResolveFormat) into an efficient parsing strategy - mirroring how Rule.Pattern is compiled once into Rule.Regexp rather than re-interpreted on every log line.
EpochUnit == 0 means "not an epoch format": parse with Layout via time.ParseInLocation. EpochUnit != 0 means "epoch format": parse as a number of EpochUnit ticks since the Unix epoch (Layout is unused).
Candidates != nil means "format: auto": Layout and EpochUnit are unused, and internal/parse tries each layout in Candidates (see LastGood) instead.
func ResolveFormat ¶
func ResolveFormat(format string) (TimeFormat, error)
ResolveFormat interprets a timestamp Field's Format string, auto-detecting which of four styles it is:
- The literal string "auto": returns a TimeFormat whose Candidates holds the fixed, ordered auto-detection layout list (see autoCandidateLayouts) and whose LastGood is a freshly allocated pointer, independent from every other ResolveFormat call.
- A known preset name (presetLayouts/presetEpochUnits), matched case-sensitively and exactly.
- A strptime pattern, if it starts with '%' (see strptimeToLayout).
- Otherwise, a raw Go reference-time layout string, used as-is - this is what every Format string meant before presets/strptime existed, so existing rules.yaml files keep working unchanged.
format == "" resolves successfully to a zero-value TimeFormat; whether a timestamp field requires a non-empty Format is rules.Validate's concern, not ResolveFormat's.
Source Files