alerts

README

mohawk/alerts

Mohawk is a metric data storage engine that uses a plugin architecture for data storage and a simple REST API as the primary interface.

Alerting rules

Alerting rules in Mohawk servers send alerts to an Alertbuffer if a metric value is outside valid range.

Usage

Running with some alerts rules in the config file:

./mohawk -c example.config.yaml
2017/12/01 17:40:58 Start server, listen on http://0.0.0.0:8080
...

Running the alert buffer:

./examples/alert-buffer.py
Starting httpd...
{"ID":"free_memory is low","Metric":"free_memory","Tenant":"_ops","State":true,"AlertIfLowerThan":2000,"AlertIfHigherThen":8000,"Type":0,"TrigerValue":40,"TrigerTimestamp":1512142870000}
127.0.0.1 - - [01/Dec/2017 17:41:18] "POST /append HTTP/1.1" 200 -
{"ID":"free_memory is low","Metric":"free_memory","Tenant":"_ops","State":false,"AlertIfLowerThan":2000,"AlertIfHigherThen":8000,"Type":0,"TrigerValue":4000,"TrigerTimestamp":1512142893000}
127.0.0.1 - - [01/Dec/2017 17:41:38] "POST /append HTTP/1.1" 200 -
{"ID":"free_memory is low","Metric":"free_memory","Tenant":"_ops","State":true,"AlertIfLowerThan":2000,"AlertIfHigherThen":8000,"Type":0,"TrigerValue":40,"TrigerTimestamp":1512142901000}
127.0.0.1 - - [01/Dec/2017 17:41:48] "POST /append HTTP/1.1" 200 -
...

Trigering the alerts using curl command:

$ curl http://localhost:8080/hawkular/metrics/gauges/raw -d "[{\"id\":\"free_memory\", \"data\":[{\"timestamp\": $(date +%s)000, \"value\": 4000}]}]"
$ curl http://localhost:8080/hawkular/metrics/gauges/raw -d "[{\"id\":\"free_memory\", \"data\":[{\"timestamp\": $(date +%s)000, \"value\": 40}]}]"

Configuring Alerts

Alerting Configuration is done using the config.yaml file. Using the alerts key, we set a list of alerts, each alert has a unique name, list of metric ids ( or tags search string ) and a valid range for that metrics. If a metric value change from being valid to not valid or from not valid to valid, an alert status change is triggered and sent to allert buffer.

For example:

alerts:
- id: "free_memory or cpu_usage are lower then 1000 or higher then 8000"
  metrics:
  - "free_memory"
  - "cpu_usage"
  alert-if-lower-than: 1000
  alert-if-higher-than: 8000
- id: "free_memory is lower then 500"
  metrics:
  - "free_memory"
  alert-if-lower-than: 500
- id: "cpu_usage is above 95%"
  metrics:
  - "cpu_usage"
  alert-if-higher-than: 95
- id: "this alert use tags"
  tags: "hostname:.*.com"
  alert-if-higher-than: 95

Documentation

Overview

Package alerts for alert rules

Index

• type Alert
• type AlertRules
  • func (a *AlertRules) FilterAlerts(tenant string, id string, state string) []Alert
  • func (a *AlertRules) Init()
• type AlertType

Types

type Alert

type Alert struct {
	ID                string            `mapstructure:"id"`
	Annotations       map[string]string `mapstructure:"annotations"`
	Metrics           []string          `mapstructure:"metrics"`
	Tags              string            `mapstructure:"tags"`
	Tenant            string            `mapstructure:"tenant"`
	AlertIfHigherThan *float64          `mapstructure:"alert-if-higher-than"`
	AlertIfLowerThan  *float64          `mapstructure:"alert-if-lower-than"`
	Type              AlertType
	State             bool
	TrigerMetric      string
	TrigerValue       float64
	TrigerTimestamp   int64
}

Alert defines one alert

type AlertRules

type AlertRules struct {
	Storage        storage.Storage
	ServerURL      string
	ServerMethod   string
	ServerInsecure bool
	Verbose        bool
	Alerts         []*Alert
	AlertsInterval int
	Heartbeat      int64
}

AlertRules defines prameters for the alerts run rutine

func (*AlertRules) FilterAlerts

func (a *AlertRules) FilterAlerts(tenant string, id string, state string) []Alert

FilterAlerts return a list of alerts, filter by tenant, id and state

func (*AlertRules) Init

func (a *AlertRules) Init()

Init fill in missing configuration data, and start the alert checking loop

type AlertType

type AlertType int

AlertType describe alert type