parser

README

![gopherbadger-tag-do-not-edit]

Parser

Parser is in charge of turning raw log lines into objects that can be manipulated by heuristics. Parsing has several stages represented by directories on config/stage. The alphabetical order dictates the order in which the stages/parsers are processed.

The runtime representation of a line being parsed (or an overflow) is an Event, and has fields that can be manipulated by user :

• Parsed : a string dict containing parser outputs
• Meta : a string dict containing meta informations about the event
• Line : a raw line representation
• Overflow : a representation of the overflow if applicable

The Event structure goes trough the stages, being altered with each parsing step. It's the same object that will be later poured into buckets.

Parser configuration

A parser configuration is a Node object, that can contain grok patterns, enrichement instructions.

For example :

filter: "evt.Line.Labels.type == 'testlog'"
debug: true
onsuccess: next_stage
name: tests/base-grok
pattern_syntax:
  MYCAP: ".*"
nodes:
  - grok:
      pattern: ^xxheader %{MYCAP:extracted_value} trailing stuff$
      apply_on: Line.Raw
statics:
  - meta: log_type
    value: parsed_testlog

Name

optional if present and prometheus or profiling are activated, stats will be generated for this node.

Filter

filter: "Line.Src endsWith '/foobar'"

• optional filter : an expression that will be evaluated against the runtime of a line (Event)
  • if the filter is present and returns false, node is not evaluated
  • if filter is absent or present and returns true, node is evaluated

Debug flag

debug: true

• optional debug : a bool that sets debug of the node to true (applies at runtime and configuration parsing)

OnSuccess flag

onsuccess: next_stage|continue

• mandatory indicates the behaviour to follow if node succeeds. next_stage make line go to next stage, while continue will continue processing of current stage.

Statics

statics:
    - meta: service
      value: tcp
    - meta: source_ip
      expression: "Event['source_ip']"
    - parsed: "new_connection"
      expression: "Event['tcpflags'] contains 'S' ? 'true' : 'false'"
    - target: Parsed.this_is_a_test
      value: foobar

Statics apply when a node is considered successful, and are used to alter the Event structure. An empty node, a node with a grok pattern that succeeded or an enrichment directive that worked are successful nodes. Statics can :

• meta: add/alter an entry in the Meta dict
• parsed: add/alter an entry in the Parsed dict
• target: indicate a destination field by name, such as Meta.my_key The source of data can be :
• value: a static value
• expr_result : the result of an expression

Grok patterns

Grok patterns are used to parse one field of Event into one or several others :

grok:
  name: "TCPDUMP_OUTPUT"
  apply_on: message

name is the name of a pattern loaded from patterns/. Base patterns can be seen on the repo : https://github.com/logrusorgru/grokky/blob/master/base.go

---

grok:
  pattern: "^%{GREEDYDATA:request}\\?%{GREEDYDATA:http_args}$"
  apply_on: request

pattern which is a valid pattern, optionally with a apply_on that indicates to which field it should be applied

Patterns syntax

Present at the Event level, the pattern_syntax is a list of subgroks to be declared.

pattern_syntax:
  DIR: "^.*/"
  FILE: "[^/].*$"

Enrichment

Enrichment mecanism is exposed via statics :

statics:
  - method: GeoIpCity
    expression: Meta.source_ip
  - meta: IsoCode
    expression: Enriched.IsoCode
  - meta: IsInEU
    expression: Enriched.IsInEU

The GeoIpCity method is called with the value of Meta.source_ip. Enrichment plugins can output one or more key:values in the Enriched map, and it's up to the user to copy the relevant values to Meta or such.

Trees

The Node object allows as well a nodes entry, which is a list of Node entries, allowing you to build trees.

filter: "Event['program'] == 'nginx'" #A
nodes: #A'
  - grok: #B
      name: "NGINXACCESS"
      # this statics will apply only if the above grok pattern matched
      statics: #B'
        - meta: log_type
          value: "http_access-log"
  - grok: #C
      name: "NGINXERROR"
      statics:
        - meta: log_type
          value: "http_error-log"
statics: #D
  - meta: service
    value: http

The evaluation process of a node is as follow :

• apply the filter (A), if it doesn't match, exit
• iterate over the list of nodes (A') and apply the node process to each.
• if a grok entry is present, process it
  • if the grok entry returned data, apply the local statics of the node (if the grok 'B' was successful, apply B' statics)
• if any of the nodes or the grok was successful, apply the statics (D)

Code Organisation

Main structs :

• Node (config.go) : the runtime representation of parser configuration
• Event (runtime.go) : the runtime representation of the line being parsed

Main funcs :

• CompileNode : turns YAML into runtime-ready tree (Node)
• ProcessNode : process the raw line against the parser tree, and produces ready-for-buckets data

Documentation

Index

• Variables
• func GenDateParse(date string) (string, time.Time)
• func GeoIpASN(field string, p *types.Event, ctx interface{}) (map[string]string, error)
• func GeoIpCity(field string, p *types.Event, ctx interface{}) (map[string]string, error)
• func GeoIpInit(cfg map[string]string) (interface{}, error)
• func IpToRange(field string, p *types.Event, ctx interface{}) (map[string]string, error)
• func Parse(ctx UnixParserCtx, xp types.Event, nodes []Node) (types.Event, error)
• func ParseDate(in string, p *types.Event, x interface{}) (map[string]string, error)
• func SetTargetByName(target string, value string, evt *types.Event) bool
• type EnrichFunc
• type EnricherCtx
  • func Loadplugin(path string) ([]EnricherCtx, error)
• type GeoIpEnricherCtx
• type InitFunc
• type Node
  • func LoadStages(stageFiles []Stagefile, pctx *UnixParserCtx, ectx []EnricherCtx) ([]Node, error)
  • func (n *Node) ProcessStatics(statics []types.ExtraField, event *types.Event) error
• type Parsers
  • func LoadParsers(cConfig *csconfig.GlobalConfig, parsers *Parsers) (*Parsers, error)
• type Stagefile
• type UnixParserCtx
  • func Init(c map[string]interface{}) (*UnixParserCtx, error)

Variables

var ExportedFuncs = []string{"GeoIpASN", "GeoIpCity"}

All plugins must export a list of function pointers for exported symbols

var NodesHits = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Name: "cs_node_hits_total",
		Help: "Total events entered node.",
	},
	[]string{"source", "name"},
)

var NodesHitsKo = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Name: "cs_node_hits_ko_total",
		Help: "Total events unsuccessfuly exited node.",
	},
	[]string{"source", "name"},
)

var NodesHitsOk = prometheus.NewCounterVec(
	prometheus.CounterOpts{
		Name: "cs_node_hits_ok_total",
		Help: "Total events successfuly exited node.",
	},
	[]string{"source", "name"},
)

var ParseDump bool

var StageParseCache map[string]map[string]types.Event

Functions

func GenDateParse

func GenDateParse(date string) (string, time.Time)

func GeoIpASN

func GeoIpASN(field string, p *types.Event, ctx interface{}) (map[string]string, error)

func GeoIpCity

func GeoIpCity(field string, p *types.Event, ctx interface{}) (map[string]string, error)

func GeoIpInit

func GeoIpInit(cfg map[string]string) (interface{}, error)

All plugins must export an Init function

func IpToRange

func IpToRange(field string, p *types.Event, ctx interface{}) (map[string]string, error)

func Parse

func Parse(ctx UnixParserCtx, xp types.Event, nodes []Node) (types.Event, error)

func ParseDate

func ParseDate(in string, p *types.Event, x interface{}) (map[string]string, error)

func SetTargetByName

func SetTargetByName(target string, value string, evt *types.Event) bool

ok, this is kinda experimental, I don't know how bad of an idea it is ..

Types

type EnrichFunc

type EnrichFunc func(string, *types.Event, interface{}) (map[string]string, error)

should be part of a packaged shared with enrich/geoip.go

type EnricherCtx

type EnricherCtx struct {
	Funcs      map[string]EnrichFunc
	Init       InitFunc
	Plugin     *plugin.Plugin //pointer to the actual plugin
	Name       string
	Path       string      //path to .so ?
	RuntimeCtx interface{} //the internal context of plugin, given back over every call
	// contains filtered or unexported fields
}

func Loadplugin

func Loadplugin(path string) ([]EnricherCtx, error)

mimic plugin loading

TODO fix this shit with real plugin loading

type GeoIpEnricherCtx

type GeoIpEnricherCtx struct {
	// contains filtered or unexported fields
}

type InitFunc

type InitFunc func(map[string]string) (interface{}, error)

type Node

type Node struct {
	FormatVersion string `yaml:"format"`
	//Enable config + runtime debug of node via config o/
	Debug bool `yaml:"debug,omitempty"`
	//If enabled, the node (and its child) will report their own statistics
	Profiling bool `yaml:"profiling,omitempty"`
	//Name, author, description and reference(s) for parser pattern
	Name        string   `yaml:"name,omitempty"`
	Author      string   `yaml:"author,omitempty"`
	Description string   `yaml:"description,omitempty"`
	Rerferences []string `yaml:"references,omitempty"`
	//if debug is present in the node, keep its specific Logger in runtime structure
	Logger *log.Entry `yaml:"-"`
	//This is mostly a hack to make writting less repetive.
	//relying on stage, we know which field to parse, and we
	//can as well promote log to next stage on success
	Stage string `yaml:"stage,omitempty"`
	//OnSuccess allows to tag a node to be able to move log to next stage on success
	OnSuccess string `yaml:"onsuccess,omitempty"`

	//Filter is executed at runtime (with current log line as context)
	//and must succeed or node is exited
	Filter        string                    `yaml:"filter,omitempty"`
	RunTimeFilter *vm.Program               `yaml:"-" json:"-"` //the actual compiled filter
	ExprDebugger  *exprhelpers.ExprDebugger `yaml:"-" json:"-"` //used to debug expression by printing the content of each variable of the expression
	//If node has leafs, execute all of them until one asks for a 'break'
	LeavesNodes []Node `yaml:"nodes,omitempty"`
	//Flag used to describe when to 'break' or return an 'error'
	EnrichFunctions []EnricherCtx

	/* If the node is actually a leaf, it can have : grok, enrich, statics */
	//pattern_syntax are named grok patterns that are re-utilised over several grok patterns
	SubGroks map[string]string `yaml:"pattern_syntax,omitempty"`
	//Holds a grok pattern
	Grok types.GrokPattern `yaml:"grok,omitempty"`
	//Statics can be present in any type of node and is executed last
	Statics []types.ExtraField `yaml:"statics,omitempty"`
	//Whitelists
	Whitelist types.Whitelist     `yaml:"whitelist,omitempty"`
	Data      []*types.DataSource `yaml:"data,omitempty"`
	// contains filtered or unexported fields
}

func LoadStages

func LoadStages(stageFiles []Stagefile, pctx *UnixParserCtx, ectx []EnricherCtx) ([]Node, error)

func (*Node) ProcessStatics

func (n *Node) ProcessStatics(statics []types.ExtraField, event *types.Event) error

type Parsers

type Parsers struct {
	Ctx             *UnixParserCtx
	Povfwctx        *UnixParserCtx
	StageFiles      []Stagefile
	PovfwStageFiles []Stagefile
	Nodes           []Node
	Povfwnodes      []Node
	EnricherCtx     []EnricherCtx
}

func LoadParsers

func LoadParsers(cConfig *csconfig.GlobalConfig, parsers *Parsers) (*Parsers, error)

type Stagefile

type Stagefile struct {
	Filename string `yaml:"filename"`
	Stage    string `yaml:"stage"`
}

type UnixParserCtx

type UnixParserCtx struct {
	Grok       grokky.Host
	Stages     []string
	Profiling  bool
	DataFolder string
}

func Init

func Init(c map[string]interface{}) (*UnixParserCtx, error)