fireeye

package module
v0.0.0-...-b749d33 Latest Latest
Warning

This package is not in the latest version of its module.

 Go to latest Published: Jul 29, 2026 License: MIT Imports: 22 Imported by: 0

README

FireEye Rule Engine

FireEye is a detection-only Go package for rule matching over arbitrary map[string]string events. It parses, validates, loads, and evaluates reusable rules; callers own alerting, blocking, routing, response generation, and every other post-detection action.

Install

go get github.com/phil-fly/FireEye@latest

Import the package as fireeye:

import fireeye "github.com/phil-fly/FireEye"

Basic Usage

engine := fireeye.NewEngine()
engine.SetPrimaryFields([]string{"Method", "Path", "URI", "RawQuery", "RequestBody"})

err := engine.UpsertRule(fireeye.ThreatRule{
    ID:         "admin-path-probe",
    Enabled:    true,
    RequireAll: false,
    Priority:   fireeye.PriorityHigh,
    Meta: fireeye.Meta{
        Title:       "Admin Path Probe",
        Description: "Detects requests probing admin paths.",
    },
    Rules: map[string]fireeye.Rule{
        "Path": {
            Type: fireeye.MatchContains,
            Values: []any{"/admin"},
        },
    },
}, fireeye.LoadOptions{})
if err != nil {
    panic(err)
}

result := engine.Detect(map[string]string{
	"Method": "GET",
	"Path":   "/admin/login",
})

Rule Sources

Rules can be loaded from:

  • In-memory YAML or JSON via DecodeRuleBundle and ReplaceRules
  • A local rule directory via rulefile.DirectoryProvider
  • A caller-managed RuleProvider via LoadFromProvider
  • Runtime updates via UpsertRule and RemoveRule

YAML rules may use global pattern variables and rule or extraction templates from _global_*.yml / _global_*.yaml files.

Metadata Input

The core engine accepts arbitrary K-V metadata:

event := map[string]string{
    "src_ip": "10.0.0.1",
    "path":   "/login",
    "agent":  "curl/8.0",
}

For web traffic, httpfields.Parse, httpfields.Read, and httpfields.FromRequest convert HTTP requests into the same K-V shape using fields such as Method, URI, Path, RawQuery, headers, form values, and RequestBody.

Directory-backed loading is explicit and keeps filesystem state out of Engine:

provider := rulefile.NewDirectoryProvider("./rules")
if err := engine.LoadFromProvider(ctx, provider, fireeye.LoadOptions{}); err != nil {
    return err
}

Use rulewatch.Run(ctx, engine, provider, opts) when the directory should be reloaded. It blocks until cancellation, debounces changes, and retains the last good engine snapshot when a reload fails.

One logical rule may contain multiple groups. Groups are OR alternatives to each other. Each group must declare logic: AND or logic: OR, and every field matcher in groups[].rules uses the same matcher, extraction, template, index, cache, and sensitivity behavior as top-level rules:

groups:
  - logic: AND
    rules:
      Method:
        type: string
        data: [GET]
      Path:
        type: string
        data: [/first]
  - logic: AND
    rules:
      Method:
        type: string
        data: [POST]
      Path:
        type: string
        data: [/second]

Use type: exists without data to match field presence. With notIs: true, the matcher succeeds only when every candidate field in a compound key such as Header:A|Header:B is absent.

DecodeRuleBundle accepts either one rule or a rules: bundle. Decoding is strict: legacy candidates, deception_response, and other unknown fields are errors rather than silently ignored configuration.

Detect returns a neutral DetectionResult containing the parent rule ID, metadata, tags, extracted fields, and detection state. DetectDetailed also returns the matched parent rule. DetectAllDetailed returns a caller-limited, deterministically ordered set of every fully matched rule when post-detection policy needs more than the single engine winner. None of these APIs selects or executes an action.

Public API Boundary

The stable business-facing API includes:

  • ThreatRule.Validate for complete rule validation and Rule.Validate for a single field rule;
  • ThreatRule.ExtractFields for running configured extraction flows without constructing a detection result;
  • Engine.Detect and Engine.DetectDetailed for the deterministic single winner, and Engine.DetectAllDetailed for detailed multi-match detection;
  • Engine.DetectWithOptions and Engine.DetectDetailedWithOptions when sensitive-field behavior should be explicit at the call site, plus Engine.DetectAllDetailedWithOptions for the same behavior across all matches;
  • Engine.IndexStats for a typed index statistics snapshot and Engine.RebuildIndex for an explicit rebuild when operational tooling needs one;
  • ThreatRule.HasExplicitPriority for checking whether decoded YAML or JSON supplied priority.

Indexes are maintained automatically by NewEngine, UpsertRule, ReplaceRules, RemoveRule, and primary-field updates. The rule tree, capability validators, extraction handler, detection-result builder, and the stored explicit-priority flag are implementation details.

DetectionOptions.SensitiveFieldsMatchByPresence makes a sensitive field predicate match when that field exists without comparing configured values. Detect(event) uses normal value comparison. Call DetectWithOptions only when presence matching is required.

Multi-match results are ordered by priority descending and RuleID ascending. A positive MaxMatches limits retained and returned matches; TotalMatches still reports the complete count and Truncated reports overflow. Values less than one return every match. Source-file order and final policy selection remain caller responsibilities.

Legacy Rule Migration

The repository-local Go CLI converts older FireEye files without adding legacy aliases to DecodeRuleBundle. Always run a validation-only pass first:

go run ./tools/fireeye_rule_migrate \
  -input ../truth-rule/rule \
  -output ../truth-rule/rule-v0.1.0 \
  -dry-run

Remove -dry-run to write the separately validated output. Directory mode reads only top-level .yml, .yaml, and .json rule files, copies top-level _global_*.yml and _global_*.yaml files unchanged, and ignores subdirectories. Existing destination files require -force; input and output paths must differ, and an output directory cannot be nested under the input directory.

The tool converts top-level rules + and and legacy candidates to explicit groups while preserving metadata, matchers, extraction, priority, and document boundaries. It reports and removes only these recognized post-detection fields: deception_response, response, route, action, alert, and block. Every other unknown field is an error. See docs/usage.md for the full migration contract.

Validate migrated exp samples separately from schema migration:

go run ./tools/fireeye_rule_exp_verify \
  -input ../truth-rule/rule-v0.1.0 \
  -report ../truth-rule/rule-v0.1.0-exp-report.json

The verifier reports transport parsing, isolated parent-rule matching, and full-snapshot rule selection independently. This prevents a malformed sample or an overlapping higher-priority rule from being mislabeled as the same failure. The external dataset test is opt-in:

FIREEYE_RULE_EXP_DIR="$(cd ../truth-rule/rule-v0.1.0 && pwd)" \
  go test ./tools/fireeye_rule_exp_verify -run TestExternalRuleExpressions -v

See the 2026-07-20 reference report for the current 1,514-rule baseline.

Development

go test ./...
go test -race ./...
go test ./... -cover
go test -bench=. ./...

See docs/usage.md for API details, performance notes, and best practices.

Stability

FireEye currently uses v0.x semantic versions. The documented core contract centers on Engine, ThreatRule, RuleGroup, Rule, RuleBundle, and DetectionResult plus the business-facing methods listed above.

v0.2.0 intentionally removes the v0.1 compatibility wrappers and moves HTTP, filesystem, and watcher behavior into focused subpackages. It also replaces mutable tree/cache exposure with typed snapshots. These breaking v0.x changes are recorded in the changelog.

Security

Report suspected vulnerabilities through the repository's private GitHub security advisory flow. See SECURITY.md for scope and reporting details.

License

FireEye is available under the MIT License.

Documentation

Overview

Package fireeye provides a detection-only rule engine for matching arbitrary string-valued event fields.

Callers own all behavior after detection, including alerting, blocking, routing, and response generation. The root package owns rule preparation, indexing, matching, and the RuleProvider contract. Filesystem, HTTP, and watcher adapters live in rulefile, httpfields, and rulewatch respectively.

Index

Examples

Constants

View Source 
const RegexMatchTimeout = 100 * time.Millisecond

RegexMatchTimeout limits regexp2 matching to avoid pathological rule matches.

Variables

This section is empty.

Functions

This section is empty.

Types

type DetectionMatch 

type DetectionMatch struct {
	Detection DetectionResult `json:"detection"`
	Rule      ThreatRule      `json:"rule"`
}

DetectionMatch is one fully matched rule and its isolated detection result.

type DetectionOptions 

type DetectionOptions struct {
	// SensitiveFieldsMatchByPresence makes a sensitive field predicate match
	// when the field exists, without comparing its configured values.
	SensitiveFieldsMatchByPresence bool
}

DetectionOptions controls one detection operation.

type DetectionResult 

type DetectionResult struct {
	Matched              bool              `json:"matched"`
	Title                string            `json:"title"`
	Description          string            `json:"description,omitempty"`
	References           []string          `json:"references,omitempty"`
	Level                string            `json:"level,omitempty"`
	Remediation          string            `json:"remediation,omitempty"`
	EventType            string            `json:"event_type,omitempty"`
	Tags                 []string          `json:"tags,omitempty"`
	RuleID               string            `json:"rule_id,omitempty"`
	ExtractedFields      map[string]string `json:"extracted_fields,omitempty"`
	ProcessingTimeMicros int64             `json:"processing_time_us"`
	AttemptedRules       int               `json:"attempted_rules"`
}

DetectionResult is the structured result of one detection run.

func (DetectionResult) JSON 

func (r DetectionResult) JSON() string

JSON returns an indented JSON representation of the result.

type Engine 

type Engine struct {
	// contains filtered or unexported fields
}

Engine owns rule loading, indexing, and detection state.

func NewEngine 

func NewEngine() *Engine

NewEngine creates an empty rule engine.

func (*Engine) Detect 

func (t *Engine) Detect(eventData map[string]string) DetectionResult

Detect detects one event using value comparison for sensitive fields.

Example 
package main

import (
	"fmt"

	fireeye "github.com/phil-fly/FireEye"
)

func main() {
	engine := fireeye.NewEngine()
	err := engine.UpsertRule(fireeye.ThreatRule{
		ID:       "admin-path-probe",
		Enabled:  true,
		Priority: fireeye.PriorityHigh,
		Meta: fireeye.Meta{
			Title: "Admin path probe",
		},
		Rules: map[string]fireeye.Rule{
			"Path": {
				Type:   fireeye.MatchContains,
				Values: []any{"/admin"},
			},
		},
	}, fireeye.LoadOptions{})
	if err != nil {
		panic(err)
	}

	result := engine.Detect(map[string]string{"Path": "/admin/login"})
	fmt.Println(result.Matched, result.RuleID)
}

Output:
true admin-path-probe

func (*Engine) DetectAllDetailed 

func (t *Engine) DetectAllDetailed(eventData map[string]string, maxMatches int) MultiDetectionResult

DetectAllDetailed returns a deterministic list of fully matched rules. A positive maxMatches limits the returned list while TotalMatches remains complete. Returned rules are clones and can be safely mutated.

func (*Engine) DetectAllDetailedWithOptions 

func (t *Engine) DetectAllDetailedWithOptions(eventData map[string]string, opts MultiDetectionOptions) MultiDetectionResult

DetectAllDetailedWithOptions is the configurable detailed multi-match API.

func (*Engine) DetectDetailed 

func (t *Engine) DetectDetailed(eventData map[string]string) (DetectionResult, ThreatRule)

DetectDetailed returns the detection result and the matched rule.

func (*Engine) DetectDetailedWithOptions 

func (t *Engine) DetectDetailedWithOptions(eventData map[string]string, opts DetectionOptions) (DetectionResult, ThreatRule)

DetectDetailedWithOptions detects one event and returns a cloned matched rule.

func (*Engine) DetectWithOptions 

func (t *Engine) DetectWithOptions(eventData map[string]string, opts DetectionOptions) DetectionResult

DetectWithOptions detects one event using explicit operation options.

func (*Engine) IndexStats 

func (tm *Engine) IndexStats() IndexStats

IndexStats returns an inspection snapshot without exposing index internals.

func (*Engine) LoadFromProvider 

func (t *Engine) LoadFromProvider(ctx context.Context, provider RuleProvider, opts LoadOptions) error

LoadFromProvider loads rules from a caller-managed source.

func (*Engine) PrimaryFields 

func (tm *Engine) PrimaryFields() []string

PrimaryFields returns a snapshot of the ordered index fields.

func (*Engine) RebuildIndex 

func (tm *Engine) RebuildIndex()

RebuildIndex rebuilds the match index from the current rule snapshot.

func (*Engine) RemoveRule 

func (t *Engine) RemoveRule(ruleID string) bool

RemoveRule deletes one rule and rebuilds the match index.

func (*Engine) ReplaceRules 

func (t *Engine) ReplaceRules(bundle RuleBundle, opts LoadOptions) error

ReplaceRules atomically replaces the in-memory rule snapshot.

func (*Engine) SetPrimaryFields 

func (tm *Engine) SetPrimaryFields(fields []string)

SetPrimaryFields replaces the ordered fields used to build the rule index.

func (*Engine) UpsertRule 

func (t *Engine) UpsertRule(rule ThreatRule, opts LoadOptions) error

UpsertRule prepares and stores one rule with the current global config.

type ExtractionNode 

type ExtractionNode struct {
	Ref   string           `json:"$ref,omitempty" yaml:"$ref,omitempty"` // template reference, for example extraction_templates.name
	Field string           `json:"field" yaml:"field"`
	Rules []ExtractionRule `json:"rules" yaml:"rules"`
}

ExtractionNode defines a named extraction flow.

type ExtractionRule 

type ExtractionRule struct {
	Type       ExtractionType `json:"type" yaml:"type"` // regexp base64
	Expression string         `json:"data,omitempty" yaml:"data,omitempty"`
}

ExtractionRule defines one extraction step.

type ExtractionType 

type ExtractionType string

ExtractionType identifies a supported extraction step. 

const (
	ExtractionBase64Decode ExtractionType = "base64"    // Base64 decode
	ExtractionRegexp       ExtractionType = "regexp"    // regular expression extraction
	ExtractionURLDecode    ExtractionType = "urldecode" // URL decode
	ExtractionGJSON        ExtractionType = "gjson"     // JSON path extraction
)

type FieldIndexStats 

type FieldIndexStats struct {
	ExactMatchRules int `json:"exact_match_rules"`
	RegexpRules     int `json:"regexp_rules"`
	ScanRules       int `json:"scan_rules"`
}

FieldIndexStats describes the index entries for one event field.

type GlobalConfig 

type GlobalConfig struct {
	Version             string                    `yaml:"version"`              // config version
	Patterns            map[string]string         `yaml:"patterns"`             // pattern variables
	RuleTemplates       map[string]Rule           `yaml:"rule_templates"`       // rule templates
	ExtractionTemplates map[string]ExtractionNode `yaml:"extraction_templates"` // extraction templates
}

GlobalConfig contains reusable patterns and templates.

func NewGlobalConfig 

func NewGlobalConfig() *GlobalConfig

NewGlobalConfig creates an empty global config.

type GroupLogic 

type GroupLogic string

GroupLogic identifies how rules inside one RuleGroup are combined. 

const (
	GroupLogicAND GroupLogic = "AND"
	GroupLogicOR  GroupLogic = "OR"
)

type IndexStats 

type IndexStats struct {
	RuleCount        int                        `json:"rule_count"`
	DefaultRuleCount int                        `json:"default_rule_count"`
	Fields           map[string]FieldIndexStats `json:"fields"`
	HotRules         []RuleHitStats             `json:"hot_rules"`
}

IndexStats is a read-only snapshot of the engine's match index.

type LoadOptions 

type LoadOptions struct {
	// SkipSensitiveRules excludes rules marked sensitive from the new snapshot.
	SkipSensitiveRules bool
}

LoadOptions controls rule loading behavior.

type MatchType 

type MatchType string

MatchType identifies how a rule compares field values. 

const (
	MatchString      MatchType = "string"       // exact string match
	MatchRegexp      MatchType = "regexp"       // regular expression match
	MatchStartsWith  MatchType = "startswith"   // prefix match
	MatchEndsWith    MatchType = "endswith"     // suffix match
	MatchContains    MatchType = "contains"     // substring match
	MatchContainsAll MatchType = "contains_all" // all configured substrings must be present
	MatchExists      MatchType = "exists"       // field-presence match
)

type Meta 

type Meta struct {
	Title       string   `json:"title" yaml:"title"`                                                // display name
	Author      string   `json:"author,omitempty" yaml:"author,omitempty"`                          // rule author
	Description string   `json:"description,omitempty" default:"nil." yaml:"description,omitempty"` // description
	Remediation string   `json:"remediation,omitempty" default:"nil." yaml:"remediation,omitempty"` // remediation guidance
	References  []string `json:"references,omitempty" yaml:"references,omitempty"`                  // references
}

Meta contains rule metadata.

type MultiDetectionOptions 

type MultiDetectionOptions struct {
	DetectionOptions

	// MaxMatches limits retained and returned matches. Values less than one
	// return every match. TotalMatches always reports the complete match count.
	MaxMatches int
}

MultiDetectionOptions controls one detailed multi-match detection operation.

type MultiDetectionResult 

type MultiDetectionResult struct {
	Matches              []DetectionMatch `json:"matches"`
	TotalMatches         int              `json:"total_matches"`
	Truncated            bool             `json:"truncated"`
	AttemptedRules       int              `json:"attempted_rules"`
	ProcessingTimeMicros int64            `json:"processing_time_us"`
}

MultiDetectionResult reports deterministic matches and overflow state.

type Priority 

type Priority int

Priority is the rule priority range, from 1 to 100. 

const (
	PriorityLowest  Priority = 1   // lowest priority
	PriorityLow     Priority = 25  // low priority
	PriorityMedium  Priority = 50  // medium priority
	PriorityHigh    Priority = 75  // high priority
	PriorityHighest Priority = 100 // highest priority

	// Priority band boundaries.
	PriorityLowMax    Priority = 24  // maximum low-priority value
	PriorityMediumMax Priority = 74  // maximum medium-priority value
	PriorityHighMax   Priority = 100 // maximum high-priority value
)

Canonical priority values.

type Rule 

type Rule struct {
	Ref             string           `json:"$ref,omitempty" yaml:"$ref,omitempty"` // template reference, for example rule_templates.name
	Sensitivity     bool             `json:"sensitivity,omitempty" default:"false" yaml:"sensitivity,omitempty"`
	ValueExtraction bool             `json:"value_extraction,omitempty" default:"false" yaml:"value_extraction,omitempty"`
	Extraction      []ExtractionRule `json:"extractionflow,omitempty" yaml:"extractionflow,omitempty"`
	Type            MatchType        `json:"type" yaml:"type"`
	Values          []any            `json:"data,omitempty" yaml:"data,omitempty"`
	Negate          bool             `json:"notIs,omitempty" yaml:"notIs,omitempty"`
}

Rule defines one field matching rule.

func (Rule) Validate 

func (r Rule) Validate() error

Validate checks one field rule, including supported capabilities and regexp syntax.

type RuleBundle 

type RuleBundle struct {
	GlobalConfig *GlobalConfig
	Rules        []ThreatRule
}

RuleBundle contains a caller-provided rule snapshot.

func DecodeRuleBundle 

func DecodeRuleBundle(data []byte) (RuleBundle, error)

DecodeRuleBundle strictly decodes one rule or a rules bundle from YAML or JSON bytes.

type RuleGroup 

type RuleGroup struct {
	Logic GroupLogic      `json:"logic" yaml:"logic"`
	Rules map[string]Rule `json:"rules" yaml:"rules"`
}

RuleGroup is one alternative field-rule set within a logical threat rule.

type RuleHitStats 

type RuleHitStats struct {
	RuleID string `json:"rule_id"`
	Count  int64  `json:"count"`
}

RuleHitStats reports one rule's in-process match count.

type RuleProvider 

type RuleProvider interface {
	Load(ctx context.Context) (RuleBundle, error)
}

RuleProvider lets callers supply rules from any backing store.

type ThreatRule 

type ThreatRule struct {
	ID string `json:"id,omitempty" yaml:"id,omitempty"`

	Meta `json:"meta" yaml:"meta"` // rule metadata

	Enabled     bool                      `json:"enabled" yaml:"enabled"`                                             // enabled state
	RequireAll  bool                      `json:"and,omitempty" yaml:"and,omitempty"`                                 // AND semantics when true
	Priority    Priority                  `json:"priority,omitempty" default:"middle" yaml:"priority,omitempty"`      // priority
	Sensitivity bool                      `json:"sensitivity,omitempty" default:"false" yaml:"sensitivity,omitempty"` // sensitivity flag
	Rules       map[string]Rule           `json:"rules,omitempty" yaml:"rules,omitempty"`                             // field matching rules
	Groups      []RuleGroup               `json:"groups,omitempty" yaml:"groups,omitempty"`                           // alternative field-rule sets
	Extraction  map[string]ExtractionNode `json:"extraction,omitempty" yaml:"extraction,omitempty"`                   // extraction flows

	Source      string   `json:"source,omitempty" yaml:"source,omitempty"`         // rule source
	System      string   `json:"system,omitempty" yaml:"system,omitempty"`         // matching system
	Level       string   `json:"level,omitempty" yaml:"level,omitempty"`           // threat level
	Tags        []string `json:"tags,omitempty" yaml:"tags,omitempty"`             // tags
	EventType   string   `json:"event_type,omitempty" yaml:"event_type,omitempty"` // event type
	Expressions []string `json:"exp,omitempty" yaml:"exp,omitempty"`               // expressions
	// contains filtered or unexported fields
}

ThreatRule is the complete rule definition.

func (*ThreatRule) ExtractFields 

func (t *ThreatRule) ExtractFields(event map[string]string) map[string]string

ExtractFields runs this rule's configured extraction flows against an event.

func (*ThreatRule) HasExplicitPriority 

func (t *ThreatRule) HasExplicitPriority() bool

HasExplicitPriority reports whether priority was present in decoded YAML or JSON.

func (*ThreatRule) Validate 

func (t *ThreatRule) Validate() error

Validate checks the complete public rule contract.

Source Files

View all Source files

Directories

Path Synopsis
Package httpfields converts HTTP requests into FireEye event fields.
 Package httpfields converts HTTP requests into FireEye event fields.
Package rulefile provides strict filesystem adapters for FireEye rules.
 Package rulefile provides strict filesystem adapters for FireEye rules.
Package rulewatch reloads a FireEye engine when a rule directory changes.
 Package rulewatch reloads a FireEye engine when a rule directory changes.
tools
fireeye_rule_exp_verify command
fireeye_rule_migrate command

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL
go.dev uses cookies from Google to deliver and enhance the quality of its services and to analyze traffic. Learn more.