eql

package module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Jul 9, 2026 License: MIT Imports: 7 Imported by: 0

README

EQL Parser

Go Reference Go Report Card

A production-ready Go parser for EQL (Event Query Language), the correlation language used by Elastic Security detection rules. It extracts conditions, fields, correlation structure, and pipes from EQL queries used in Elasticsearch, Kibana, and the legacy Endgame ecosystem.

Hand-written lexer and recursive-descent parser, zero dependencies, and tested against a large corpus of real-world detection rules plus millions of fuzzed and mutated inputs.

Features

  • Full EQL grammar: event queries, sequence, join, and sample correlations, by join keys, with maxspan, with runs, until, and ![...] missing-event clauses.
  • Every operator: == != < <= > >=, the case-insensitive :, like / like~, regex / regex~, in / in~ / not in, arithmetic, and boolean and / or / not.
  • Condition extraction: field, operator, value, negation (De Morgan-aware), case-sensitivity, alternatives for list operators, event category, sequence step, and pipe stage.
  • Field discovery: optional ?fields, backtick-escaped `field names`, dotted paths, and array indices (process.args[0]).
  • Functions: wildcard, cidrMatch, endsWith, startsWith, stringContains, length, match, and the ~ case-insensitive variants.
  • Pipes: head, tail, plus the legacy count, unique, unique_count, sort, and filter.
  • Permissive legacy acceptance: single-quoted strings, = equality, child of / descendant of / event of lineage, and ?"raw" strings from Endgame-era rules.
  • Robust by construction: never panics, never hangs (MaxParseTime watchdog), bounded input and nesting, and graceful error recovery that still extracts what it can.
  • Input normalization: smart quotes, zero-width characters, markdown code fences, and JSON-escaped newlines from copy-pasted or extracted rules.

Installation

go get github.com/craftedsignal/eql-parser

Usage

Condition extraction
package main

import (
	"fmt"

	eql "github.com/craftedsignal/eql-parser"
)

func main() {
	result := eql.ExtractConditions(
		`process where process.name : "cmd.exe" and process.parent.name : "explorer.exe"`,
	)

	for _, c := range result.Conditions {
		fmt.Printf("%s %s %q (category=%s)\n", c.Field, c.Operator, c.Value, c.EventCategory)
	}
	// process.name : "cmd.exe" (category=process)
	// process.parent.name : "explorer.exe" (category=process)
}
Correlation (sequence / join / sample)
result := eql.ExtractConditions(`
sequence by process.entity_id with maxspan=5m
  [process where event.type == "start" and process.name : "msxsl.exe"]
  [network where event.type == "connection" and network.direction : "egress"]`)

seq := result.Sequence
fmt.Println(seq.Kind)        // "sequence"
fmt.Println(seq.MaxSpanMS)   // 300000
fmt.Println(seq.ByFields)    // [process.entity_id]
fmt.Println(len(seq.Steps))  // 2
AST access
q, err := eql.Parse(`process where process.name in ("a.exe", "b.exe")`)
if err != nil {
	// err is non-nil for malformed input, but q still holds the partial AST.
}
fmt.Println(q.String()) // canonical re-rendering (round-trips)

API

Function Description
ExtractConditions(query) *ParseResult Full extraction: conditions, sequence info, pipes, fields, errors. Never panics.
Parse(query) (*Query, error) Parse to an AST with round-trippable String().
ParseExpression(input) (Expr, error) Parse a standalone boolean expression.
NormalizeQuery(query) string Clean up pasted/extracted query text.
ClassifyFieldUsage(result, field) FieldUsage Structured field role: join key, sequence steps, pipes, until.
ClassifyFieldProvenance(result, field) FieldProvenance Field role as a main/joined/join_key/ambiguous string (sibling-parser vocabulary).
DeduplicateConditions(conditions) []Condition Remove exact-duplicate conditions, preserving context.
IsStatisticalQuery(result) bool Query aggregates/correlates (count/unique pipe, or sequence/sample).
HasComplexWhereConditions(result) bool Any pattern/list/function operator is used.
GetEventTypeFromConditions(result) string Canonical event type (e.g. windows_4688) from an event-code condition.
MaxSpanDuration(span) (int64, bool) Convert a maxspan value to milliseconds.

ParseResult carries Conditions, EventCategories, Sequence, Pipes, Commands, Fields, JoinKeys (also via GroupByFields()), and Errors. Each Condition records the Field, Operator, Value, Negated, CaseInsensitive, LogicalOp, Alternatives, EventCategory, SequenceStep, FromUntil, FromMissing, PipeStage, IsOptional, Function, ValueIsField, CoFields, and Lineage. Sequence/join/sample steps additionally expose a recursive Subquery *ParseResult — a self-contained extraction of that correlation term, the EQL analog of a join subsearch.

Sigma interoperability

A bidirectional Sigma ↔ EQL translator ships in cmd/generated-corpus (kept out of the dependency-free core, matching the sibling parsers). Correctness is enforced by a round-trip corpus: translating a rule to the other dialect and re-parsing it must reproduce the same set of conditions under a shared canonical vocabulary. Against the SigmaHQ corpus (3,100+ rules) and the real EQL corpus, every field-translatable rule round-trips with zero condition loss in both directions — constructs with no faithful cross-dialect form (EQL sequences, Sigma free-text keywords, escaped wildcards, unicode-obfuscation values) are explicitly classified, never silently mistranslated. Regenerate with make sigma-corpus.

At larger scale, a 100,000-rule corpus — ~46k real Sigma rules scraped from 68 GitHub repositories plus the CraftedSignal library, deduped, combined with generated complex rules — runs through parse → translate → round-trip with zero panics and 100% round-trip fidelity on every field-translatable rule. The translated EQL (~94k queries) is committed and re-parsed by the core test suite. Build with make rules-corpus.

Testing

make test        # go test -race ./...
make fuzz        # native Go fuzzing (FuzzExtractConditions, FuzzParse)
make corpus      # regenerate the differential test corpus
make benchmark   # lexer / parser / extractor benchmarks

The suite includes unit tests for the lexer, parser, and extractor; a round-trip property test (parse → render → parse is a fixpoint); the full 1,209 real detection rules (elastic/detection-rules + endgameinc/eqllib), all parsing cleanly; a curated valid/adversarial corpus; bounded-AST differential testing with an independent oracle; a 500,000-query generated corpus (0 panics, 100% parse rate at scale); a Sigma ↔ EQL round-trip corpus; native fuzzing (millions of executions); and a 100k-iteration structured stress test plus ~200k adversarial mutation runs — all with zero panics, at 100% statement coverage.

Dialect notes

The canonical dialect is modern Elasticsearch EQL. Legacy Endgame syntax is accepted permissively (and flagged where it is invalid in modern EQL) because real-world rule corpora mix both. The parser reports semantic problems (sequence needing two events, missing-event clauses needing maxspan, comparison chaining, unknown pipes) in ParseResult.Errors while still extracting everything it can — extraction is best-effort by design.

Limitations

  • EQL functions are recognized structurally; their return types are not evaluated. length(x) > 2 extracts x with the length function recorded, not a computed value.
  • Regex and wildcard values are extracted as literal strings; patterns are not compiled or matched.
  • The parser targets extraction and structural analysis, not execution. It does not query Elasticsearch.

License

MIT — see LICENSE.

Documentation

Index

Constants

View Source
const MaxInputSize = 1 << 20 // 1 MiB

MaxInputSize is the largest query Parse and ExtractConditions accept.

Variables

View Source
var ErrEmptyQuery = errors.New("empty query")

ErrEmptyQuery is returned when the input contains no tokens.

View Source
var ErrInputTooLarge = errors.New("input exceeds maximum size")

ErrInputTooLarge is returned when the input exceeds MaxInputSize.

View Source
var MaxParseTime = 5 * time.Second

MaxParseTime is the maximum time allowed for parsing a single query. Queries that exceed it return an error result instead of hanging.

Functions

func ExprString

func ExprString(e Expr) string

ExprString renders any expression node as canonical EQL text.

func GetEventTypeFromConditions

func GetEventTypeFromConditions(r *ParseResult) string

GetEventTypeFromConditions derives a canonical event-type string (e.g. "windows_4688", "sysmon_1") from an event-code condition, so EQL rules classify identically to the Sigma/SPL/KQL rules for the same telemetry. Returns "" when the query carries no recognizable event code.

func HasComplexWhereConditions

func HasComplexWhereConditions(r *ParseResult) bool

HasComplexWhereConditions reports whether any extracted condition uses a pattern/list/function operator rather than plain comparison. Mirrors the sibling helper.

func HasUnmappedComputedFields

func HasUnmappedComputedFields(r *ParseResult) bool

HasUnmappedComputedFields reports whether the query defines computed fields that are not backed by a source field. EQL has no computed-field construct (no eval/extend/project), so this is always false; it exists for uniform API parity with the sibling parsers.

func IsStatisticalQuery

func IsStatisticalQuery(r *ParseResult) bool

IsStatisticalQuery reports whether the query aggregates or correlates rather than simply filtering: it uses a counting/uniquing pipe, or is a sequence/sample (stateful correlation). Mirrors the sibling helper.

func MaxSpanDuration

func MaxSpanDuration(span string) (int64, bool)

MaxSpanDuration converts a maxspan value like "30s" or "2h" to milliseconds. Returns false when the text is not a recognized duration.

func NormalizeQuery

func NormalizeQuery(query string) string

NormalizeQuery cleans up EQL text that was pasted from documents, extracted from JSON/YAML/TOML rule files, or otherwise mangled in transit, without changing the meaning of well-formed queries. ExtractConditions applies it automatically; Parse does not.

Types

type BadExpr

type BadExpr struct {
	Near string
}

BadExpr is a placeholder emitted during error recovery.

type Binary

type Binary struct {
	Op   string
	L, R Expr
}

Binary covers logical (and/or), comparison (== != < <= > >= : treated separately), and arithmetic (+ - * / %) operators. Op is the canonical lowercase operator text.

type Call

type Call struct {
	Name        string
	Insensitive bool
	Args        []Expr
}

Call is a function call. Insensitive marks the ~ suffix (endsWith~).

type Condition

type Condition struct {
	Field           string   `json:"field"`
	Operator        string   `json:"operator"` // "==", "!=", "<", "<=", ">", ">=", ":", "like", "regex", "in", or a function name ("wildcard", "cidrmatch", ...)
	Value           string   `json:"value"`
	Negated         bool     `json:"negated"`
	CaseInsensitive bool     `json:"case_insensitive,omitempty"` // : operator, ~ variants
	LogicalOp       string   `json:"logical_op"`                 // "AND" or "OR" connecting to the previous condition
	Alternatives    []string `json:"alternatives,omitempty"`     // all values for list operators and merged same-field ORs
	EventCategory   string   `json:"event_category,omitempty"`   // category of the event filter this came from ("any" or "" for bare)
	SequenceStep    int      `json:"sequence_step"`              // 0-based step index; 0 for non-sequence queries
	FromUntil       bool     `json:"from_until,omitempty"`       // condition came from an until clause
	FromMissing     bool     `json:"from_missing,omitempty"`     // condition came from a ![ missing-event clause
	PipeStage       int      `json:"pipe_stage"`                 // 0 = main query, N = Nth pipe (1-based, filter pipes)
	IsOptional      bool     `json:"is_optional,omitempty"`      // field carried the ? optional marker
	Function        string   `json:"function,omitempty"`         // function wrapping the field, e.g. "length" in length(f) > 5
	ValueIsField    bool     `json:"value_is_field,omitempty"`   // Value names another field (field-to-field comparison)
	CoFields        []string `json:"co_fields,omitempty"`        // other fields in the same comparison (e.g. b in "a + b > 10")
	Lineage         string   `json:"lineage,omitempty"`          // "child", "descendant", "event" for legacy lineage subqueries
}

Condition is one field predicate extracted from an EQL query.

func DeduplicateConditions

func DeduplicateConditions(conds []Condition) []Condition

DeduplicateConditions removes duplicate conditions, keeping the first occurrence of each. Two conditions are duplicates when they agree on field, operator, value, negation, case-sensitivity, alternatives, and correlation context (step, pipe stage, until, lineage). Exported for parity with the sibling parsers' adapters.

type EventQuery

type EventQuery struct {
	Category    string // literal category text ("" when Any or Bare)
	CategoryAny bool   // `any where ...`
	Bare        bool   // no `category where` prefix, just an expression
	Where       Expr
}

EventQuery is `category where condition`, `any where condition`, or — as a tolerated extension for extraction of stored rule fragments — a bare boolean expression without a category (Bare=true).

type ExpectedCondition

type ExpectedCondition struct {
	Field        string   `json:"field"`
	Operator     string   `json:"operator"`
	Value        string   `json:"value,omitempty"`
	Alternatives []string `json:"alternatives,omitempty"`
	Negated      bool     `json:"negated,omitempty"`
	SequenceStep int      `json:"sequence_step,omitempty"`
}

ExpectedCondition is a condition signature used for differential testing. It captures the fields the generator can predict deterministically.

type Expr

type Expr interface {
	// contains filtered or unexported methods
}

Expr is the interface implemented by all expression nodes.

func ParseExpression

func ParseExpression(input string) (Expr, error)

ParseExpression parses a standalone boolean expression (no event category, sequences, or pipes).

type Field

type Field struct {
	Optional bool
	Path     []PathSeg
}

Field is a (possibly optional, possibly indexed) field reference such as `?process.args[0]` or “ `host-name`.ip “.

func (*Field) Name

func (f *Field) Name() string

Name returns the canonical dotted path without the optional marker, e.g. "process.args[0]".

type FieldProvenance

type FieldProvenance string

FieldProvenance indicates where a field originates relative to a query's correlation structure. The string values match the sibling parsers' provenance vocabulary (main/joined/join_key/ambiguous) so a platform adapter can treat EQL uniformly with KQL/SPL; ProvenanceUnknown is an EQL addition for a field absent from the result.

const (
	// ProvenanceMain is a field filtering the primary event stream (a
	// non-sequence query, the first sequence step, or a filter pipe).
	ProvenanceMain FieldProvenance = "main"
	// ProvenanceJoined is a field from a correlated event — a later sequence
	// step or an until clause.
	ProvenanceJoined FieldProvenance = "joined"
	// ProvenanceJoinKey is a field used as a correlation key (a `by` clause).
	ProvenanceJoinKey FieldProvenance = "join_key"
	// ProvenanceAmbiguous is a field used in more than one distinct role.
	ProvenanceAmbiguous FieldProvenance = "ambiguous"
	// ProvenanceUnknown is returned for a field not present in the result.
	ProvenanceUnknown FieldProvenance = "unknown"
)

func ClassifyFieldProvenance

func ClassifyFieldProvenance(r *ParseResult, field string) FieldProvenance

ClassifyFieldProvenance reports the role a field plays in the parsed query, using the same vocabulary as the sibling parsers: a correlation key (join_key), a primary-stream filter (main), a correlated-event filter (joined), or several roles at once (ambiguous). A field the query never references is ProvenanceUnknown.

type FieldUsage

type FieldUsage struct {
	IsJoinKey bool  `json:"is_join_key"`
	Steps     []int `json:"steps,omitempty"` // sequence steps referencing the field
	InPipes   bool  `json:"in_pipes"`
	InUntil   bool  `json:"in_until"`
}

FieldUsage classifies how a field participates in a query.

func ClassifyFieldUsage

func ClassifyFieldUsage(r *ParseResult, field string) FieldUsage

ClassifyFieldUsage reports how a field participates in the parsed query.

type GeneratedCase

type GeneratedCase struct {
	ID       int64               `json:"id"`
	Seed     int64               `json:"seed"`
	Query    string              `json:"query"`
	Expected []ExpectedCondition `json:"expected"`
}

GeneratedCase is one entry in the generated differential corpus: a rendered query plus the conditions a correct extractor must produce.

type InExpr

type InExpr struct {
	X           Expr
	List        []Expr
	Negated     bool
	Insensitive bool
}

InExpr is `x [not] in[~] (a, b, c)`.

type Lineage

type Lineage struct {
	Kind string // "child", "descendant", "event"
	Sub  *EventQuery
}

Lineage is the legacy Endgame relationship predicate: `child of [process where ...]`, `descendant of [...]`, `event of [...]`.

type Literal

type Literal struct {
	Kind  LiteralKind
	Value string // decoded string value
	Text  string // raw number text
	Bool  bool
	Raw   bool // originated from a raw string form
}

Literal is a constant value. For strings, Value holds the decoded text; for numbers, Text holds the original notation.

type LiteralKind

type LiteralKind int

LiteralKind discriminates literal values.

const (
	LitString LiteralKind = iota
	LitNumber
	LitBool
	LitNull
)

type Not

type Not struct {
	X Expr
}

Not is prefix `not expr`.

type Paren

type Paren struct {
	X Expr
}

Paren preserves explicit grouping from the source.

type ParseResult

type ParseResult struct {
	Conditions      []Condition   `json:"conditions"`
	EventCategories []string      `json:"event_categories,omitempty"` // unique categories in source order
	Sequence        *SequenceInfo `json:"sequence,omitempty"`         // set for sequence/join/sample queries
	Pipes           []PipeInfo    `json:"pipes,omitempty"`
	Commands        []string      `json:"commands,omitempty"`  // construct kind + pipe names, in order
	Fields          []string      `json:"fields,omitempty"`    // every field referenced anywhere (source order, deduped)
	JoinKeys        []string      `json:"join_keys,omitempty"` // every by-clause field (global + per-step)
	Errors          []string      `json:"errors,omitempty"`
}

ParseResult contains everything extracted from a query.

func ExtractConditions

func ExtractConditions(query string) *ParseResult

ExtractConditions parses an EQL query and extracts conditions, correlation structure, pipes, and referenced fields. It never panics and never hangs: hostile input yields a result whose Errors explains what went wrong. Input is normalized with NormalizeQuery first.

func (*ParseResult) GroupByFields

func (r *ParseResult) GroupByFields() []string

GroupByFields returns the query's correlation keys (the `by` clause fields), under the name the sibling parsers use for their grouping axis. It is an alias for the JoinKeys field, provided for uniform adapter code.

type PathSeg

type PathSeg struct {
	Name    string
	Index   int
	IsIndex bool
}

PathSeg is one segment of a dotted field path: a name or an array index.

type PatternExpr

type PatternExpr struct {
	Kind          PatternKind
	X             Expr
	Patterns      []Expr
	Parenthesized bool // list form was written with parentheses
	Insensitive   bool
}

PatternExpr is `x : v`, `x like v`, `x regex v`, or their list forms `x like ("a*", "b*")`. Insensitive marks the ~ variants (":" is inherently case-insensitive and keeps Insensitive=false).

type PatternKind

type PatternKind string

PatternKind distinguishes the string-matching predicates.

const (
	PatternSeq   PatternKind = ":"     // case-insensitive equality with wildcards
	PatternLike  PatternKind = "like"  // wildcard match
	PatternRegex PatternKind = "regex" // regular expression match
)

type Pipe

type Pipe struct {
	Name string
	Args []Expr
}

Pipe is `| name arg1, arg2`.

type PipeInfo

type PipeInfo struct {
	Name string   `json:"name"`
	Args []string `json:"args,omitempty"`
}

PipeInfo is one post-processing pipe with rendered arguments.

type Query

type Query struct {
	Body  QueryBody
	Pipes []*Pipe
}

Query is a parsed EQL statement: one body (event query, sequence, join, or sample) followed by zero or more pipes.

func Parse

func Parse(query string) (*Query, error)

Parse parses a single EQL statement and returns its AST. The parser is tolerant: it recovers from many errors and still produces a partial AST, but any problem encountered is reported in the returned error (the AST is still valid to inspect). Input is parsed as-is; use NormalizeQuery first for content pasted from documents.

func (*Query) String

func (q *Query) String() string

String renders the query as canonical single-line EQL.

type QueryBody

type QueryBody interface {
	// contains filtered or unexported methods
}

QueryBody is implemented by *EventQuery and *Sequence.

type Sequence

type Sequence struct {
	Kind        SequenceKind
	By          []Expr // global join keys (sequence/join/sample `by`)
	MaxSpan     string // raw duration text, e.g. "30s" ("" when absent)
	WithByOrder bool   // header was written `with maxspan=... by ...` (parsed-then-tolerated order)
	Steps       []*SequenceStep
	Until       *SequenceStep
}

Sequence is `sequence ...`, `join ...`, or `sample ...`.

type SequenceInfo

type SequenceInfo struct {
	Kind      string             `json:"kind"` // "sequence", "join", "sample"
	MaxSpan   string             `json:"max_span,omitempty"`
	MaxSpanMS int64              `json:"max_span_ms,omitempty"`
	ByFields  []string           `json:"by_fields,omitempty"` // global join keys
	Steps     []SequenceStepInfo `json:"steps"`
	Until     *SequenceStepInfo  `json:"until,omitempty"`
}

SequenceInfo describes the correlation structure of a sequence, join, or sample query — the EQL analog of join metadata in the other parsers.

type SequenceKind

type SequenceKind string

SequenceKind discriminates the three correlation constructs that share the Sequence node shape.

const (
	KindSequence SequenceKind = "sequence"
	KindJoin     SequenceKind = "join"
	KindSample   SequenceKind = "sample"
)

type SequenceStep

type SequenceStep struct {
	Missing bool // opened with ![ — matches the absence of the event
	Query   *EventQuery
	By      []Expr // per-step join keys
	Runs    int    // `with runs=N`, 0 when absent
	WithKey string // raw key of a `with key=value` modifier ("runs" normally)
}

SequenceStep is one bracketed term of a sequence/join/sample.

type SequenceStepInfo

type SequenceStepInfo struct {
	EventCategory  string       `json:"event_category"`
	ByFields       []string     `json:"by_fields,omitempty"`
	Runs           int          `json:"runs,omitempty"`
	Missing        bool         `json:"missing,omitempty"`
	ConditionCount int          `json:"condition_count"`
	Subquery       *ParseResult `json:"subquery,omitempty"`
}

SequenceStepInfo describes one bracketed event of a sequence/join/sample. Subquery holds a self-contained recursive extraction of the step's event filter (`[category where ...]`), the EQL analog of a join subsearch — its conditions are step-relative (SequenceStep 0) and independently inspectable.

type Token

type Token struct {
	Type  TokenType
	Text  string // raw source text (original spelling)
	Value string // decoded value for strings and backtick identifiers, lowercase-insensitive-normalized for keywords
	Pos   int    // byte offset in input
	Line  int    // 1-based line
	Col   int    // 1-based column (bytes)

	Tilde        bool // identifier carried a ~ suffix (case-insensitive function)
	Raw          bool // string was raw (triple-quoted or ?"..." form)
	TripleQuoted bool // string used """..."""
	SingleQuoted bool // string used '...' (legacy Endgame form)
}

Token is a single lexical unit with position information.

type TokenType

type TokenType int

TokenType identifies the lexical class of a token.

const (
	TokenEOF TokenType = iota
	TokenError

	// Literals and names
	TokenIdent         // process, endsWith, endsWith~ (Tilde=true)
	TokenBacktickIdent // `my-field` (Value holds the unescaped name)
	TokenString        // "...", """...""", '...', ?"..." (Value holds the decoded value)
	TokenNumber        // 42, 3.14, 1e-5, .5

	// Keywords
	TokenAnd
	TokenOr
	TokenNot
	TokenIn         // in
	TokenInTilde    // in~
	TokenLike       // like
	TokenLikeTilde  // like~
	TokenRegex      // regex
	TokenRegexTilde // regex~
	TokenWhere
	TokenSequence
	TokenJoin
	TokenSample
	TokenUntil
	TokenBy
	TokenWith
	TokenMaxspan
	TokenOf
	TokenAny
	TokenTrue
	TokenFalse
	TokenNull

	// Operators and punctuation
	TokenEQ       // ==
	TokenNEQ      // !=
	TokenLT       // <
	TokenLTE      // <=
	TokenGT       // >
	TokenGTE      // >=
	TokenColon    // :
	TokenAssign   // =
	TokenPlus     // +
	TokenMinus    // -
	TokenStar     // *
	TokenSlash    // /
	TokenPercent  // %
	TokenPipe     // |
	TokenComma    // ,
	TokenLParen   // (
	TokenRParen   // )
	TokenLBrack   // [
	TokenRBrack   // ]
	TokenMissing  // ![ (missing-event subquery opener)
	TokenDot      // .
	TokenOptional // ? (optional field marker)
)

func (TokenType) String

func (t TokenType) String() string

type Unary

type Unary struct {
	Op string
	X  Expr
}

Unary is prefix arithmetic minus/plus.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL