banshee

command module
v0.2.0 Latest Latest
Warning

This package is not in the latest version of its module.

Go to latest
Published: Apr 29, 2016 License: MIT Imports: 14 Imported by: 0

README

Banshee

Banshee is a real-time anomalies(outliers) detection system for periodic metrics.

Build Status GoDoc

snap-01 snap-02 snap-03

Requirements

  1. Go >= 1.5.
  2. Node and gulp.

Build

  1. Clone the repo.
  2. Build binary via make.
  3. Build static files via make static.

Deployment

https://godoc.org/github.com/eleme/banshee#hdr-Deployment

Upgrade

https://godoc.org/github.com/eleme/banshee#hdr-Upgrade

Philosophy

3-sigma:

>>> import numpy as np
>>> x = np.array([40, 52, 63, 44, 54, 43, 67, 54, 49, 45, 48, 54, 57, 43, 58])
>>> mean = np.mean(x)
>>> std = np.std(x)
>>> (80 - mean) / (3 * std)
1.2608052883472445 # anomaly, too big
>>> (20 - mean) / (3 * std)
-1.3842407711224991 # anomaly, too small

Documentation

https://godoc.org/github.com/eleme/banshee

Statsd Backend

https://www.npmjs.com/package/statsd-banshee

Migrate from bell

https://godoc.org/github.com/eleme/banshee#hdr-Migrate_from_bell

Authors

License

MIT Copyright (c) 2015 - 2016 Eleme, Inc.

Documentation

Overview

Banshee is a real-time anomalies(outliers) detection system for periodic metrics.

Use Case

We are using it to monitor our website and rpc services intefaces, including called frequency, response time and exception calls. Our services send statistics to statsd, statsd aggregates them every 10 seconds and broadcasts the results to its backends including banshee, banshee analyzes current metrics with history data, calculates the trending and alerts us if the trending behaves anomalous.

For example, we have an api named get_user, this api's response time (in milliseconds) is reported to banshee from statsd every 10 seconds:

20, 21, 21, 22, 23, 19, 18, 21, 22, 20, ..., 300

Banshee will catch the latest metric 300 and report it as an anomaly.

Why don't we just set a fixed threshold instead (i.e. 200ms)? This may also works but it is boring and hard to maintain a lot of thresholds. Banshee will analyze metric trendings automatically, it will find the "thresholds" automatically.

Features

1. Designed for periodic metrics. Reality metrics are always with periodicity, banshee only peeks metrics with the same "phase" to detect.

2. Multiple alerting rule configuration options, to alert via fixed-thresholds or via anomalous trendings.

3. Coming with anomalies visualization webapp and alerting rules admin panels.

4. Require no extra storage services, banshee handles storage on disk by itself.

Requirements

1. Go >= 1.5.

2. Node and gulp.

Build

1. Clone the repo.

2. Build binary via `make`.

3. Build static files via `make static`.

Command Line

Usage:

banshee [-c config] [-d] [-v]

Flags:

-c config
	Load config from file.
-d
	Turn on debug mode.
-v
	Show version.

Configuration

See package config.

Statsd Integration

In order to forward metrics to banshee from statsd, we need to add the npm module statsd-banshee to statsd's banckends:

1. Install statsd-banshee on your statsd servers:

$ cd path/to/statsd
$ npm install statsd-banshee

2. Add module statsd-banshee to statsd's backends in config.js:

{
, backends: ['statsd-banshee']
, bansheeHost: 'localhost'
, bansheePort: 2015
}

Migrate from bell

Require bell.js v2.0+ and banshee v0.0.7+:

./migrate -from bell.db -to banshee.db -with-projs -with-users
mv banshee.db path/to/storage/admin

Compontents

Banshee have 4 compontents and they are running in the same process:

1. Detector is to detect incoming metrics with history data and store the results.

2. Webapp is to visualize the detection results and provides panels to manage alerting rules, projects and users.

3. Alerter is to send sms and emails once anomalies are found.

4. Cleaner is to clean outdated metrics from storage.

Alerting Sender

See package alerter and alerter/exampleCommand.

Deployment

Via fabric(http://www.fabfile.org/):

python deploy.py -u hit9 -H remote-host:22 --remote-path "/service/banshee"

See deploy.py docs for more.

Upgrade

Just pull the latest code:

git remote add upstream https://github.com/eleme/banshee.git
git pull upstream master

Note that the admin storage sqlite3 schema will be auto-migrated.

Implementation Details

1. Detection algorithms, see package detector.

2. Detector input net protocol, see package detector.

3. Storage, see package storage.

4. Filter, see package filter.

Reference: https://github.com/eleme/banshee/blob/master/intro.md

License

MIT (c) eleme, inc.

Directories

Path Synopsis
Package alerter implements an alerter to send sms/email messages on anomalies found.
Package alerter implements an alerter to send sms/email messages on anomalies found.
exampleCommand
Example alerter command to echo message to console.
Example alerter command to echo message to console.
Package config handles configuration parsing.
Package config handles configuration parsing.
Package detector is a tcp server to detect anomalies.
Package detector is a tcp server to detect anomalies.
Package filter implements fast wildcard like filtering based on suffix tree.
Package filter implements fast wildcard like filtering based on suffix tree.
Package health implements the health statistic aggregation.
Package health implements the health statistic aggregation.
Package models implements metric, index, rule, project, user modeling.
Package models implements metric, index, rule, project, user modeling.
Package storage implements banshee's persistence storage.
Package storage implements banshee's persistence storage.
admindb
Package admindb handles the admin storage on SQLite3.
Package admindb handles the admin storage on SQLite3.
indexdb
Package indexdb handles the indexes storage.
Package indexdb handles the indexes storage.
metricdb
Package metricdb handles the metrics storage.
Package metricdb handles the metrics storage.
Package util provides util functions.
Package util provides util functions.
idpool
Package idpool implements a reusable integer id pool.
Package idpool implements a reusable integer id pool.
log
Package log implements leveled logging.
Package log implements leveled logging.
mathutil
Package mathutil provides math util functions.
Package mathutil provides math util functions.
safemap
Package safemap implements a map like container with rw lock to keep groutine safety.
Package safemap implements a map like container with rw lock to keep groutine safety.
trie
Package trie implements a in-memory trie tree.
Package trie implements a in-memory trie tree.
Package webapp implements a simple http web server to visualize detection results and to manage alerting rules.
Package webapp implements a simple http web server to visualize detection results and to manage alerting rules.

Jump to

Keyboard shortcuts

? : This menu
/ : Search site
f or F : Jump to
y or Y : Canonical URL